3. Big data – Big deal? What are the trends in big data analysis? Is Big data a key tool for increasing competitiveness and productivity?

comments 0






Samuel Chua's picture

If we were to seek the origins of big data, we will not have to look very far back into the past. Big data’s roots can be traced back to the rise of the Dotcom bubble, and its subsequent implications on modern science. Back then, science was the pioneer of utilising huge amounts of data for research. Why then is big data only becoming a big deal now? According to Gordon Moore, computing hardware will double in power approximately every 2 years[1].By extension, this improvement in computing power has made it more affordable and efficient to collect and store massive amounts of data today, which would not have been feasible maybe a decade or two ago.

Today, big data is used to predict stock price movements, airfare ticket prices and even – although controversially – foresee potential terrorist and flu crises. No doubt, the notion of big data accelerating workplace processes and efficiency is a given. Bearing this in mind, we should be cautious of allowing the data to lead users to the false conclusions, and also consider the ethical consequences of collecting and utilizing such knowledge.

Correlation replaces Causation

Authors of the book Big Data[2], Viktor Mayer-Schönberger and Kenneth Cukier, argue that the main paradigm shift in how big data has changed how we analyse issues is to accept that correlation rather than causation is the appropriate method in deciphering big data. We learn in statistics class that even strongly correlated instances do not naturally pan out to have a direct causal link. However, what the authors are suggesting is that sometimes, certain data trends are not possible or easy to be rationalized by our human brains, but we should acknowledge what it is telling us nonetheless.

This argument is logical because at a macro level, with millions and billions of data sets, algorithms can draw conclusions faster and more accurately than our human brains can conceive. More inputs, even with the downside of more outliers and imprecisions, can surpass small samples of exact data in terms of practicality.

Will Big Data result in decrease of Employment Opportunities in the Current Firms?

Many opponents of big data contend that the revolution would lead to the obsolescence of the professional or skilled worker. Just as how Fordism streamlined the factory workforce. Will big data result in a decline in manpower due to its ability to formulate smoother and more efficient business processes? Ultimately, the answer is both yes and no.

Firstly, restructuring manpower is a necessary evil. With the data of everything and everyone in a firm compiled and analysed, companies can sieve out the underperformers. The data can then provide and recommend alternative solutions, such as a job rotation, change in incentives, or in the unfortunate case: retrenchment, in order to improve employee productivity.

Secondly, big data may see a shift in employment opportunities. Firms would require (more) teams of computer scientists to program algorithms and interpret the raw data gathered. According to research done by consulting firm Bain and Company, only 4% of Large Cap firms polled have decent analytics talent.[3] Most likely because, these skills come at a price, which only the Large Caps are able to afford.  However, the smaller firms need not see this as a disadvantage, but a chance to innovate. Rather than investing on mapower and tools to collate and source for more data, if these less capitalized companies can focus their resources on understanding collected data faster, and more effectively without the human experts, this will allow them to surge ahead in the big data arms race.

Employee productivity is not the only aspect of the firm that has benefited from big data. Formulation of managerial action plans also have been made more efficient through its implementation. Citing the same report by Bain & Co., firms with the talent to interpret and utilize big data have proven competitive advantages; double the odds of being in the top quartile in their respective industry, triple the assurance in carrying out decisions and 5 times the likelihood of quicker decision making processes.

Another form of elimination of the human worker is by replacing them with Artificial Intelligence (AI), and their huge data sets. Companies of the future may no longer turn to the subject experts to seek advice. Consider IBM’s supercomputer, Watson, who is able to understand natural language and concoct logical and accurate replies, the AI experts of the future do not seem as far-fetch as a work of science fiction.

The Value of Data

The growth of online and digital transactions has “datafied” monetary dealings, making it easier for compilation, computation and subsequent analysis by the AI’s algorithms. This reduces fraud, generates near real-time information, and can possibly provide recommendations to managers and executives regarding segmentation, targeting and positioning strategies. From a marketer’s perspective, collected data can be used in facilitating bundling and cross-selling efforts, accurately identifying the needs, taste and even purpose from customers’ purchasing and search patterns.

Potentially, big data could become the new currency of trade. Companies like big data leader Google have already begun recognizing the real value of data. They have been providing useful services like search engines and mobile software, without charging users a cent. However, there is no free lunch in this world, and what we consumers are actually paying for is far more valuable than money: information. All the data gathered through inputs made by users can be monetized by passing them on to companies for business purposes – and in the case of Google via Google AdWords. Other than monetizing data directly for marketing, the data can be recycled into other forms of products and services, venturing into areas such as linguistics, sciences and psychology.

Limits to Big Data

Despite big data being touted as a competitive advantage when it comes to running businesses, we should avoid becoming blindsided by it completely. Data itself can be rigid and as the algorithms’ parameters and measurements are set by people. The computer scientists who concoct them are human, thus they (big data and supercomputers) can be prone to unforeseen human error which we are all susceptible to. For example when comparing aggregate data and disaggregate data, lurking variables could be covered up due to the aggregation. This can be explained by the Yule-Simpson’s Paradox (refer to table 1.1 and 1.2), where aggregated data shows a reverse trend when compared to the disaggregate data. Unless big data firms can invent a more flexible and self-improving algorithm or machine, we should be wary in trusting the data wholeheartedly.

Table 1.1 Aggregated data of Credit Card Promotion response rate. CC1 seems to have a more favourable response.

  Credit Card Promotion 1(CC1) Credit Card Promotion 2(CC2) Total
Responded 4,900 2,800 7,700
Didn't Respond 310,000 262,500 572,500
Total 314,900 265,300 580,000
Response Rate 1.56% 1.06% 1.33%
Table 1.2 Disaggregated data of Credit Card Promotion response rate. The trend is reversed and the lurking variable “gender” reveals an opposite trend – in favour of CC2. favour.
  Male Female
  CC1 CC2 Total CC1 CC2 Total
Responded 4,800 1,800 6,600 100 1,000 1,100
Didn't Respond 220,000 62,500 282,500 90,000 200,000 290,000
Total 224,800 64,300 289,100 90,100 201,000 291,100
Response Rate 2.14% 2.80% 2.28% 0.11% 0.50% 0.38%


Big Data’s Ethical Issues

Critiques have condemned the power of big data. Citing “Minority Report” futuristic possibilities, where criminals are apprehended before they even planned or carried out the act, or spying of personal lives by external agencies.  Admittedly, these claims are not unfounded, as we recall that slightly more than a year ago, Instagram decided to change their privacy policy to sell user photos without expressly notifying them, causing an uproar among users.

However, the issue is two-sided and can be resolved with 3 proposed strategies. Firstly, users have the ability to choose to remove their personal data whenever they disagree with a website’s privacy guidelines. Users then can move their information to alternative websites which share a similar purpose.

Still, the aforementioned can only be achieved when users are clearly notified of the change in guidelines, which then ties in with the role big data companies and social media sites have to play. In order to achieve clear and well-informed decisions of their users, companies need to put their policies in plain and simple language and make their fine prints more apparent. This would ensure that sufficient notice is given to consumers.

Finally, governments are key in ensuring enforcement of the privacy laws between businesses and society. Ensuring a fair agreement is made between all stakeholders. Though, in the meantime, this would seem unlikely as nations have disparate jurisdictions and to form a common entity to enforce it universally would take time and effort.

Conclusively, upper management of firms would seriously need to reconsider the importance of Big Data and their company’s position on it. No doubt, it will be changing the employment and competitive landscape of the future, making clearer decisions for the firm. Yet simultaneously, maintaining stakeholders’ privacy and ethical interests at heart.  




[1] Moore, G. E. (1965). Cramming More Components onto Integrated Circuits. Reprint IEEE retrieved from http://www.cs.utexas.edu/~fussell/courses/cs352h/papers/moore.pdf

[2] Mayer-Schönberger, V. and Cukier, K. N. (2013) Big Data: A Revolution That Will Transform How We Live, Work, and Think. Great Britain: John Murray.

[3] Wegener, R. and Sinha, V. (2013) The value of Big Data: How analytics differentiates winners. Bain & Co. Retrieved from http://www.bain.com/publications/articles/the-value-of-big-data.aspx