Millions of bytes to Millions of dollars

comments 0






Sidharth Ganesh's picture

At the World Economic Forum in January 2012 at Switzerland, Big data was declared as a new class of economic asset, like currency or gold. Millions of interconnected sensors are being embedded in physical devices that sense, create and communicate data in this age of the “Internet of Things” or “Industrial Internet”. Big data is now a term that has been consociated with over 150,000 employment opportunities, intelligent business decisions and organizational transformations worth millions of dollars.

Evolution of technology and Information explosion

The past few years has played host to advances in various technologies that have directly fueled the adoption of big data. Recent trends in Cloud computing have made immense computing power and data storage options available at much affordable prices than before. Also, the availability of powerful open-source software and technology as well as the outburst of available data have been powerful motivators in this direction.

It is estimated that the amount of data that is captured is growing at 50% a year. By 2009, nearly all sectors in the US economy had at least an average of 200TB of stored data per company, and many sectors had more than 1 petabyte in mean stored data per company [1]. Enterprises are collecting data with greater granularity and frequency, capturing every customer transaction, attaching personal information and also collecting more information about consumer behavior in many different environments. The increasing use of multimedia in sectors such as healthcare and other consumer-facing industries have also contributed significantly. The high dependence on social networks for communication and a boom in the adoption of smartphones has driven social media data volumes through the roof. Increasing use of wearable technology such as Google Glass, smart-watches and bands also produce vast amounts of data.

The competitive advantage

A study by MIT Sloan Management review and IBM Institute for Business Value found that improvement of information and analytics was a top priority in their organizations [2]. However, the data centered economy is in its nascent stages. While the forerunners have made the impact of traditional analytics very clear and inviting, research at McKinsey Global Institute shows that the scale and scope of changes that big data are bringing are at an inflection point, set to expand greatly, as a series of technology trends accelerate and converge. The use of big data is becoming a key way for leading companies to outperform their peers. Competitors that employ big data analytics are in the top of their game – consumer products, finance, retail or travel and entertainment.

Several organizations have been known to mine customer data to improve their products and services. “Product kaizen” is termed as the ability of products to leverage data to improve with use. Ranging from tailored marketing campaigns to customer segmentation, big data is changing the way organizations listen to their customers. Advances in Natural Language Processing (NLP) has enabled organizations to analyze consumer postings about them on social media sites such as Facebook and Twitter, and gauge the immediate impact of their marketing campaigns and understand how consumer sentiment about their brands is changing.

In the retail industry, retailers analyze sales, pricing and customer interests to fine tune supply, advertisement efforts and timings of price markdowns – which can lead to an estimated 60% increase in operating margin [1]. By mining tremendous amounts of customer data, retailers ‘learn’ customer’s preferences, estimated buying power and observe trends in overall buying patterns. The vast amount of customer data available now translates into personalized promotions for every customer, in contrast to common offers across customer segments earlier before. By making supply and demand signals between retail stores and vendors, and thereby optimizing the supply chain, ‘stock-out’ is now a thing of the past.

Various organizations have leveraged big data to drive their R&D processes. Several of Google’s services from its primary search to Translate are heavily dependent on large amounts of data from its users. Credit card companies monitor every purchase and can identify fraudulent ones with a high degree of accuracy, using rules derived by crunching through billions of transactions. The value from the healthcare sector in US adopting big data techniques is estimated at $300 billion every year [1]. A few other functions that have employed big data have been shown in Table 1 [4].




Supply chain

Simulate and optimize supply chain flows; reduce inventory and stock-outs

Dell, Wal-Mart, Amazon

Customer selection, loyalty and service

Identify customers with the greatest profit potential; increase likelihood that they will want the product or service offering; retain loyalty

Harrah’s, Capital One, Barclays


Identify the price that will maximize yield, or profit

Progressive, Mariott

Human Capital

Select the best employees for particular tasks or jobs, at particular compensation levels

New England Patriots, Oakland A’s, Boston Red Sox, Google

Product and service quality

Detect quality problems early and minimize them

Honda, Ford Motor, PepsiCo, Intel, Southwest Airlines

Financial performance

Better understand the drivers of financial performance and the effects of nonfinancial factors

MCI, Verizon

Research and development

Improve quality, efficacy and where applicable, safety of products and services

Novartis, Amazon, Yahoo!, Google, Disney

Employee and customer engagement

Motivate better performance from employees, increase participation and contributions from customers

Salesforce, VMware, Coca Cola, Electronic Arts

Table 1.

Another concept that can be applied across multiple industrial sectors, called Gamification involves using the big data that your employees are generating as they interact with your system to motivate better performance and drive a hard ROI. Applying the same concept to the Internet of Customers, Gamification is now being employed to motivate customers to engage, contribute and be more loyal to their product/service. Research forecasts the market for gamification solutions will increase by more than 11-fold over the next four years [3]. 

“Torture the data long enough and they will confess to anything”

The existence of big data intensifies the search for interesting correlations. However, correlations do not establish causality. Usually, these correlations have a provoking interpretive story that usually gives the correlation in some sense. For example, American Express found that people who run up large bills on their Amex card and then register a new forwarding address in Florida have a greater likelihood to declare bankruptcy; reason attributing to the fact that Florida has one of the most liberal bankruptcy laws. This brings us to point of identifying patterns that create answers to questions you didn’t even know to ask.

Other innovative uses of Big Data commonly called “now-casting”, refers to the use of real-time data to describe contemporaneous activities before official data sources are available. By tracking the incidence of flu-related search terms, Google Flu trends can identify possible flu outbreaks one or two weeks earlier than official health reports. Global Pulse is a new initiative by the UN to leverage big data for global development – to help predict job losses, spending reductions or disease outbreaks in a given region.


Most CIOs (Chief Information officers) admit that their data are of poor quality, and do not trust the information they had to base their decisions on [4]. Many say that technology meant to make sense of it often just produces more data. Instead of finding the needle in the hay stack, they are making more hay.

Organizational leaders often lack the understanding of the value in big data as well as how to unlock this value. A central idea to the adoption of big data analytics is that you need to consider data as a central asset of your service, and not as an ‘exhaust’ that comes from it. A common feature that was noticed in over 63% of companies that have successfully implemented a big data approach to business intelligence involved a centralized enterprise unit as the primary source of analytics [5]. A centralized analytics unit can provide a home for more advanced skills to come together within the organization, providing both advanced models and enterprise governance through establishing priorities and standards.

However, there is a shortage of talent of people with expertise in statistics and machine learning, and of managers who can handle the big data approach. Some organizations cope by contracting work to countries such as India, home to many statistical experts.

Furthermore, data privacy is an issue whose importance, particularly to consumers is growing as the value of big data becomes more apparent. Another closely related concern is data security – how to protect competitively sensitive data or other private data. In December 2013, Target saw one of their nightmares coming true when criminals forced their way into Target’s systems, gaining access to personal as well as credit card information of over 110 million customers [6].

The huge loads of data we have now are a resource, and the tools available to mine are growing exponentially. However monumental the benefits it can bring to an organization or business, we must not be blinded in its glitter. Rather, we must embrace the technology for the impact it can make as well as the limitations and imperfections. 


[1] McKinsey Global Institute, "Big Data: The next frontier for innovation, competition, and productivity," McKinsey&Company, 2011.
[2] E. L. R. S. M. S. H. N. K. Steve LaValle, "Big Data, Analytics and the Path from Insights to Value," MIT Sloan Management Review, 2011.
[3] R. Paharia, "How Gamification and Big Data are driving business today," 8 January 2014. [Online]. Available:
[4] The Economist, "Data, data everywhere," The Economist, 2010.
[5] T. H. Davenport, "Competing on Analytics," Harvard Business Review, January 2006.
[6] B. Marr, "When Big Data turns into a big nightmare!," 21 1 2014. [Online]. Available:
[7] B. Marr, "9 Amazing Ways Big Data is used today to change the world," 5 11 2013. [Online]. Available:
[8] T. Eliott, "Interview: The need for Big Data governance," 27 January 2014. [Online]. Available:
[9] D. Amerland, "3 Ways Big Data changed Google's hiring process," 2014 January 2014. [Online]. Available:
[10] D. Bollier, "The Promise and Peril of Big Data," The Aspen Institute, 2010.
[11] M. C. J. M. Jacques Bughin, "Clouds, big data and smart assets: Ten tech-enables business trends to watch," McKinsey Quarterly, 2010.
[12] The Economist, "Economist special report: The data deluge," Economist, 2010.
[13] C. Duhigg, "How companies learn your secrets," The NY Times, 16 February 2012.
[14] S. Lohr, "The Age of Big Data," Sunday Review, NY Times, 11 February 2012.