Thermonuclear blast, 1954. Public Domain. Source: Wikimedia, Department of Energy
The 20th century began, conceptually, when Albert Einstein published his seminal paper on relativity. The intellectual revolution from this led to E = mc^2, nuclear energy, and the atomic bomb. Nuclear weapons revolutionized war, diplomacy, geopolitics, and – to a lesser extent – electric power, although a nuclear reaction is a pretty inefficient way to boil water and run a turbine.
The 21st century began with the dot-com bust and the proliferation of data networks, data mining, data analysis, and data storage. “Big Data” is the term we use for data sets too large and complex for traditional data management approaches. There are now more than 1 trillion websites with more than 10 trillion individual web pages and more than 500 exabytes of data. (An exabyte is a billion gigabytes.) And the data just keep growing.
The complexity boggles the mind, in the same way that the implications of relativity did a hundred years ago: time doesn’t move forward uniformly, light bends around large mass objects, and the universe’s mass-energy is gradually running down. But while the 20th century gave us immensely powerful nuclear weapons, it was still ruled by everyday politics: the military is subordinate to civilian authority, war is an extension of diplomacy, talking beats fighting, and so on. Nuclear equations didn’t do anything to change human nature.
“Relativity” by M.C. Escher. Source: Wikipedia
Likewise, the rules of big data must still follow the everyday rules of “little data”: correlation does not equal causation, bad data leads to bad conclusions, formulating the right question is harder than finding the answer, there’s always more data to collect, and many others. Just because we have more information doesn’t mean our thinking has gotten any better. Indeed, one of the most important axioms of data modelling is that accuracy is far more important than precision.
Still, there’s a lot of new material out there. We can now use satellite images to gauge industrial activity in remote locations, we can screen-scrape a billion prices per day to estimate inflation around the world, we can see lending and borrowing activity in real-time. This has the potential to make our economic analysis and investment decisions far more efficient.
Just don’t expect a revolution. “Big Data” may make us more informed, but it won’t make us any smarter.
Douglas R. Tengdin, CFA
Charter Trust Company
“The Best Trust Company in New England”