It seems that everyone is talking about Big Data. Big Data refers to data sets so large and complex that normal database management tools aren’t powerful enough to handle them. Because so many people are using the internet in so many diverse ways there are billions and trillions of gigabytes out there to be studied. The hope is that all this information can be crunched in such a way so that researchers can spot trends, prevent diseases, fight crime, and avoid potholes.
Wikipedia is an example of Big Data. So is all the GPS data generated by smartphones on the road, so when you’re driving somewhere you can avoid the traffic-jam five miles ahead. Because computing power is improving, scientific processes that once took 10 years can be done in less than a week.
This new technology will likely fall short of its over-inflated promises. All their sophisticated big-data credit models didn’t help the big banks to avoid the Financial Crisis. In fact, over-reliance of managers on their models probably made it worse. Google Flu is another example of a Big Data flop.
People who work with data know the perils of confusing correlation with causation, trend extrapolation, data-mining, confirmation bias, and data quality. Quantity is no substitute for quality. Too often, Big Data leads to big hype and hubris,
Data sets will continue to grow as the world becomes more complex. Computers may be able to turn information into knowledge, but it takes insight to make that knowledge useful. Wisdom just doesn’t grow on database-trees.
Douglas R. Tengdin, CFA
Chief Investment Officer