… The World Will Never Be the Same
Big Data has already changed the world – and it’s just getting started. As the technology matures, the range of applications is growing ever broader.
The drumbeat coming out of the venture and technology communities about big data probably seems like hype to many. After all, computers have been handling large amounts of data for decades. What’s changed? The answer is that while the pot may have been on the stove for years, it is now reaching a boil.
Readers are probably familiar with Moore’s Law. In the late 1960s Intel founder Gordon Moore predicted that transistor densities would double every two years, a prediction that has held true for over forty years. Moore made no comparable prediction regarding data storage, but the history is that the cost of storing data has improved at an even faster rate, halving every eighteen months since 1980. That’s a factor of 16 million over 33 years.
The chart below puts this progression into context by tracking the declining cost of storing information on hard disks. To “make it real”, we have converted the costs per gigabyte to a more approachable metric: each data point represents the cost of the hard disks of that vintage that would be required to store all the information in the 17 million books contained in the US Library of Congress.
Between 1980 and 2013 the “cost to store the Library of Congress” fell from roughly $27 billion to a rather more manageable $6,000. The implications of this vanishing cost of data storage are profound. Increasingly, there is no piece of data too small or too worthless to store. And once stored, that data can be mined, analyzed and manipulated to produce unprecedented value. Over the past decade a host of companies have developed tools to do just that, as well as other tools to compile disparate databases into coherent and usable wholes. The technology and the tools are interesting in their own right, and have produced some lucrative outcomes for their venture investors.
Those tools will continue to improve, but they are already sufficient to empower other entrepreneurs to apply them in novel and creative ways, and indeed venture capital portfolios contain an array of investments in companies applying Big Data techniques to advertising, marketing, healthcare and even education. A dramatic recent example, however, demonstrated that it’s not just applications with commercial underpinnings or objectives that can change the world.
———-
In the weeks following last year’s US presidential election various stories began to emerge relating to the Obama campaign’s very sophisticated application of Big Data techniques to all aspects of the campaign.
In an article posted in MIT’s Technology Review last month,
long-time political operative Joe Trippi summarized the 2012 campaign as follows:
“…big data gave Obama 2012 the names of all 69 million people who voted for the candidate in 2008 and allowed the campaign to rebuild that winning coalition, vote by vote. Big data told the campaign which voters were undecided, and even which voters with otherwise Republican attitudes could be swayed to vote for the president. The campaign spent over $100 million developing the biggest network of people in political history. Millions of Americans heard from other Americans about issues that mattered to them. Those conversations were more powerful than the billions of dollars spent on TV ads. It requires no hyperbole to say that Obama 2012 changed everything.”
Alexis C. Madrigal described a number of specific technical accomplishments in The Atlantic magazine.
We have excerpted two of the more notable examples here.
“They created the most sophisticated email fundraising program ever. The digital team, under Rospars’ leadership, took their data-driven strategy to a new level. Any time you received an email from the Obama campaign, it had been tested on 18 smaller groups and the response rates had been gauged. The campaign thought all the letters had a good chance of succeeding, but the worst-performing letters did only 15 to 20 percent of what the best-performing emails could deliver. So, if a good performer could do $2.5 million, a poor performer might only net $500,000. The genius of the campaign was that it learned to stop sending poor performers.
…the Analytics team built a tool they called The Optimizer, which allowed the campaign to buy eyeballs on television more cheaply. They took set-top box (that is to say, your cable or satellite box or DVR) data from Davidsen’s old startup, Navik Networks, and correlated it with the campaign’s own data. This occurred through a third party called Epsilon: The campaign sent its voter file and the television provider sent their billing file and boom, a list came back of people who had done certain things like, for example, watched the first presidential debate. Having that data allowed the campaign to buy ads that they knew would get in front of the most people at the least cost. They didn’t have to buy the traditional stuff like the local news, either. Instead, they could run ads targeted to specific types of voters during reruns or off-peak hours. “
The most comprehensive work we have seen on this subject comes from author Sasha Issenberg in her new book The Victory Lab: The Secret Science of Winning Campaigns. In it she relates how Big Data proved its worth to the Obama campaign team after the 2008 elections, and how the team’s enthusiastic and entrepreneurial embrace of Big Data driven techniques came to pervade the 2012 campaign – with spectacular results.
“His (Dan Wagner, the team leader) approach amounted to a decisive break with 20th-century tools for tracking public opinion, which revolved around quarantining small samples that could be treated as representative of the whole…After the voters returned Obama to office for a second term, his campaign became celebrated for its use of technology—much of it developed by an unusual team of coders and engineers—that redefined how individuals could use the Web, social media, and smartphones to participate in the political process…underneath all that were scores describing particular voters: a new political currency that predicted the behavior of individual humans. The campaign didn’t just know who you were; it knew exactly how it could turn you into the type of person it wanted you to be…”
In her concluding paragraph she writes:
“Few events in American life other than a presidential election touch 126 million adults, or even a significant fraction that many, on a single day. Certainly no corporation, no civic institution, and very few government agencies ever do. Obama did so by reducing every American to a series of numbers…”
Further excerpts from the book are posted on the Technology Review website.
http://www.technologyreview.com/contributor/sasha-issenberg/
The 2012 election demonstrated that the winning teams and the winning strategies will increasingly be those that can best harness the power of Big Data. Our world is business, not politics, but the lesson holds true. Whether you manage an established business or are starting a new one, make sure the forces of Big Data are on your side.
Big Data changed the world. And it’s just getting started.
Our next installment will be about Big Data approaches to education.
