Know your limits, big data

comments 0






Nicolas Zahn's picture

Big data, i.e. the application of sophisticated algorithms to vast amounts of data to generate new insights, has yielded interesting results and opened up a world of new possibilities.[1] However, the drivers behind big data are subject to change in the near future and businesses need to understand how changing attitudes and regulatory frameworks favor certain types of big data implementation and limit the growth potential in other areas.

Growth potential…

Big data has left its initial niche and become a buzzword in several industries. Businesses around the world are thrilled by the ways big data could help them generate value and gain a competitive edge.[2] Several studies have tried to distill the drivers that turn big data from fascinating science into business value. They can be summarized as follows:[3] i) availability of data, i.e. quantity, quality and variety; ii) ability to process data, through algorithms and technologies as well as well-trained personnel; iii) ability to use processed data in a profitable way. Our society generates more and more data as technology penetrates even the most private environments: the same amount of data that was created “from the beginning of recorded time until 2003” was created “every two days” in 2011.[4] And not only has technology helped to create massive datasets, the technologies used to analyze does data sets is also rapidly evolving.[5] Thus, one is easily convinced that while ‘only’ 28% of respondents to the Big Data @ Work 2012 study have implemented big data, the “trend is far from over.”[6]

…with limitations

However, all three of the mentioned drivers face challenges in the near future. The shortage of personnel equipped to harness the full potential of big data and open questions regarding data policies have already been identified as challenges (process).[7] In addition, the availability of certain kinds of data will be negatively influenced (availability). Also, businesses jumping on the big data wagon might realize that big data is not the right solution for their problems[8] and regulators around the world might feel compelled to address growing public unease about big data by limiting its use (use).


Data can come from inside a company, e.g. manufacturing, R&D and sales but also from outside a company, e.g. from actual or potential customers, so-called “people analytics.”[9] New technologies and a willingness to share information make this latter type of data source a promising field for big data that has already “fueled the growth of a multi-billion dollar industry.”[10] However, concerns are growing that data gathered from real persons, especially without their explicit knowledge about the use of said data, is highly problematic. While dystopian scenarios - where free will is an illusion, everything determined by algorithms, and everyone a victim of the data he produced - mainly surfaced in the feuilleton pages,[11] the NSA revelations and other scandals in the IT industry, especially data brokers that are often acting against commonly held principles,[12] have led to a change in attitudes.[13] People realize that they do not fully understand how ‘their’ data is being used for big data and what consequences it could have.[14] As a reaction, they become less willing to ‘give away’ their data.[15] Over time, companies committed to transparency and accountability might regain the trust of their customers,[16] but it is clear that the availability of data gathered from persons, especially outside a company, will be limited. Companies focusing their big data efforts on operational optimization, a minority at the moment,[17] will have an easier time.


The changing public views are also reflected in the regulatory sphere, adding to the inherent wish to regulate any new technology as well as the growing grip state authorities are getting on regulation of the internet and transborder flows of data. Regulators, especially in the US, state that current legal frameworks do not adequately protect individuals and voices for restrictions of the use of big data are getting louder.[18] Again, the focus lies not on the use of big data for internal operational optimization, e.g. enhancing logistics, but on the use and potential abuse of personal data. Hence, companies thinking about implementing big data, especially if big data is customer-centric or even the business model, should keep the changing regulatory environment under close watch and try to understand how their use of big data is seen from other actors, be it competitors, customers or regulators.


With this in mind, one can still agree that big data is here to stay and to grow, however, its growth might not be as quick as expected but, keeping long-term impact in mind, and this might be a good thing.


[1] (Cukier und Mayer-Schönberger 2013); (Dobbs, et al. 2011)

[2] (Dobbs, et al. 2011); (Peck 2013); (Schroek, et al. 2012)

[3] For further reference see: (Balboni, et al. 2013); (Ciesielski 2014); (Dobbs, et al. 2011); (Schroek, et al. 2012)

[4] (Hoffer 2014)

[5] (Dobbs, et al. 2011); (Schroek, et al. 2012)

[6] (Ciesielski 2014); (Schroek, et al. 2012)

[7] (Dobbs, et al. 2011).

[8] This essay will not elaborate on this point but for further reference see: (Ciesielski 2014); (Cukier und Mayer-Schönberger 2013); (Davenport 2013); (Mayer-Schönberger und Cukier 2013); (Ross, Beath und Quaadgras 2013); (Schneider 2013)

[9] (Peck 2013)

[10] (CCST 2013)

[11] (Cukier und Mayer-Schönberger 2013); (ICT4Peace 2013); (Peck 2013); (Schneider 2013)

[12] Such as the OECD Guidelines Governing the Protection of Privacy and Transborder Flows of Personal Data (OECD 2013)

[13] The term „Snowden effect“ is already being used (Dooley 2013); (ICT4Peace 2013); (Markoff 2013). Also, the techniques used by intelligence services and big data are practically identical (Ciesielski 2014)

[14] (Peck 2013) offers us the scenario of a potential employee who is rejected because an algorithm determined that he would not fit the position, however, the applicant never knows the reason.

[15] (W.I.R.E 2013)

[16] (Schroek, et al. 2012)

[17] (Schroek, et al. 2012)

[18] (CCST 2013); (GAO 2013); (Markoff 2013); (Peck 2013); (Schneider 2013)


Balboni, Fred, Glenn Finch, Cathy Rodenbeck Reese, and Rebecca Shockley. 2013. "Analytics: A blueprint for value - Converting big data and analytics insights into results." IBM Institute for Business Value.
CCST. 2013. "A Review of the Data Broker Industry: Collection, Use, and Sale of Consumer Data for Marketing Purposes." Committee on Commerce, Science, and Transportation Office of Oversight and Investigations.
Ciesielski, Rebecca. 2014. "Müssen wir mit allem rechnen? Ein Lagebericht aus der Welt des DataiData Nr. 1." February.
Cukier, Kenneth, and Viktor Mayer-Schönberger. 2013. "The Rise of Big Data: How It's Changing the Way We Think About the World." Foreign Affairs May/June: 28-40.
Davenport, Tom. 2013. "Big Data and the Role of Intuiton." December.
Dobbs, Richard, James Manyika, Charles Roxburgh, and Susan Lund. 2011. "Big data: The next frontier for innovation, competition, and productivity." McKinsey Global Institute.
Dooley, Roger. 2013. "Is Google Really Evil? Or Just Smart?" October.
GAO. 2013. "Information Resellers - Consumer Privacy Framework Needs to Reflect Changes in Technology and the Marketplace." Report to the Chairman, Committee on Commerce, Science, and Transportation, U.S. Senate, United States Government Accountability Office.
Hoffer, David. 2014. "What Does Big Data Look Like? Visulaization Is Key for Humans." January.
ICT4Peace. 2013. "What's so Big about Big Data? Panel at ICCM 2013, Nairobi, Kenya." November.
Markoff, John. 2013. "Viewing Where the Internet Goes." December.
Mayer-Schönberger, Viktor, and Kenneth Cukier. 2013. Big Data: A Revolution that Will Transform how We Live, Work, and Think. Houghton Mifflin Harcourt.
OECD. 2013. "OECD Guidelines Governing the Protection of Privacy and Transborder Flows of Personal Data." OECD.
Peck, Don. 2013. "They're Watching You at Work." They're Watching You at Work. December.
Ross, Jeanna W., Cynthia M. Beath, and Anne Quaadgras. 2013. You May Not Need Big Data After All. December.
Schneider, Manfred. 2013. "Die Erkenntnis von nichts." December.
Schroek, Michael, Rebecca Shockley, Janet Smart, Dolores Romero-Morales, and Peter Tufano. 2012. "Analytics: The real-world use of big data." IBM Institute for Business Value.
W.I.R.E. 2013. "Das Grosse Rauschen - Warum die Datengesellschaft mehr Menschenverstand braucht." no. 12.