Big Data: Time to jump onto the bandwagon

comments 0






Prabhat Singh's picture

Before I begin, let me humbly confess to my inability to pinpoint the definition of Big Data. While Big Data is clearly the next disruptive innovation for businesses-and humanity in general-the term still spells enigma for most. Thus, before delving deeper,I make an attempt to at least elucidate the core differences between Big Data and extant data technologies.

“Big,” like beauty, lies in the eyes of the beholder, which implies that Big Data is definitely not contingent upon just the size of data. Suffice to say that Big Data is a new epistemological paradigm-beyond capacity of legacy IT systems-which reduces dependence on the classic method of “sampling,” helping derive innumerable hidden correlations between hitherto unknown parameters,from humongous data sets.

Now that a workable definition of Big Data has been established,this piece will attempt to critically analyze its possible impact on businesses’ bottom line,and the upcoming trends in this technology.

1.Business impact

Research by EIU1 and MIT Professors2 suggests that Big Data has huge tangible impact on every conceivable part of business cycle,because of its sheer computation capacity that allows companies to “invent” information by combining and understanding disparate datasets that were impossible to analyze before. This is exemplified3 by how 100x more data-coupled with sentiment analysis4-helped a travel giant redefine its “best” customers as those who were not necessarily using its services the most-which was the erstwhile metric-but were most influential in evangelizing it online. The company privileged these customers,who helped it gain several million dollars in surplus revenue. Such information throws up a world of possibilities,and companies capitalizing on them are turning out to be industry leaders.

Essentially, there are two major novelties that Big Data has facilitated,which make it indispensable to business growth- a)predictive analysis, b)customer micro-segmentation.  While the former provides a decision-making tool to organizations-helping them determine much faster than before their subsequent strategy-the latter helps organizations pinpoint the target consumers for their next campaign.

1.1 Enter Adam

The arguments in favour of Big Data essentially point to how it helps businesses enhance profits by providing unprecedented insight into consumer choice and behaviour. To explain their crux,I look to my favourite hypothetical friend Adam. Let’s assume Adam is an upper-middle class guy with access to a smartphone and other services that qualify him as a digital native. Of course,he’s addicted to social networking. The scenarios mentioned below will demonstrate vicariously how companies-across varied industries-are leveraging Big Data for predictive analysis and customer micro-segmentation in the field of marketing, inventory management etc. to leave traditional competitors behind.

In the days of traditional data,when Adam had to order a product online,he would laboriously dig through piles of Amazon pages to find what he liked. Sometimes he would receive mass-promotional advertisements,riddled with products he’d never bothered about. Though now,he ends up buying a bunch of products from Amazon while being on a single page! All this because Amazon now couples Adam’s historical purchases with sentiment analysis to gauge his likings. Customer micro-segmentation helped Amazon turn Adam-an individual-into a targetable market in himself,while predictive analytics helped it recommend products in real-time,making the combination too tempting for him to resist. Adam’s gushing that his favourite retailer sweated exclusively for him; his loyalty to Amazon deepens. For the record, 30% of Amazon’s sales5 may come from such strategies.

Lately, Adam’s grown increasingly discontented with the services provided by his bank-UBS. Gone are the good old days of sulking,for Adam now has a smartphone to tweet his sentiments right away. The run-of-the-mill bank might be oblivious to the churning of one of their customers, but what does UBS do? Well, its Big Data algorithms for sentiment analysis track Adam’s tweet(again displaying customer micro-segmentation) and help it ameliorate his suffering before he churns. In another instance, a transaction is made in Spain from Adam’s credit card. But the bank,always keeping track of Adam’s location through his smartphone,comes to know instantly that he’s in Norway. Bingo! They nail the fraud and prevent any losses to Adam.  Adam’s delighted-he retweets his new found love for the bank. His army of Twitter followers is amazed at such service. They think about banking with UBS too. Again,predictive analysis and customer micro-segmentation together have worked wonders.

Recently, Adam was taken by surprise by a Walmart ad for products he’d never bought.But something felt eerily familiar. Oh! Adam had given those products a good long look the last few times he was there. Does this mean Walmart now gleans information from tracking eye movements? Precisely. And Walmart does so with assistance from a Big Data company called Realeyes6. Big Data’s advantages to retailers are not limited to promotions. Walmart now combines daily data on weather patterns,expected number and choices of customers in each store, inter alia,to optimize its inventory,thus using Big Data to redefine the concept of “JIT.”

Not surprisingly,instances of Adam’s benefiting from Big Data don’t form an exhaustive set. Big Data is perhaps the only way to track “Black Swan events, such as network intrusion,which occur very frugally even in gigantic datasets,often escaping the scrutiny of sampling methods. Big Data also helps firms cut costs massively. McKinsey predicts7 that proper application of Big Data could enable US healthcare industry reduce $200bn in annual expenditure. Besides,Big Data carries innumerable non-economic benefits8.

1.2 All good, then?

Not really. Despite the boons,the banes of Big Data are not too far to seek. Used loosely,it could result in massive capital expenditure, with little ROI. Reaping gains from Big Data requires organizational changes where decisions are based on data-driven evidence rather than on HiPPO9. Even so, an HBR blog10 warns against “Big Data’s Blind Spots”-phantom correlations with little real value that gullible marketers could get trapped in. Such “Blind Spots” also inflate the possibility of the classic folly of conflating causation with correlation. An example11 comes from the customer-churn econometric model of an energy company, where Google searches for another energy supplier were 65% responsible for the churn(this was inspired by close correlation between search for an energy supplier and customer attrition). However,sharper insight revealed that other factors were responsible for attrition,and search only began once customers had decided to switch.

Big Data’s detractors rightly decry “dirty data”-the kind that’s redundant to the analysis sought. An example12 of dirty data is analysis of tweets in NYC a night after Sandy,which shows a huge spike in nightlife. Does this mean New Yorkers were partying to allay their suffering? No-an overwhelming majority of tweets came from parts of the city untouched by the hurricane. Further,solely depending on Big Data to accurately predict the highly capricious  human behaviour is a recipe for disaster. Naturally, the data and algorithms need consistent updating. Even so, “deification13” of data remains a dangerous possibility. To ensure reliable results,Big Data must never override human judgement,which can only be done by having sagacious data scientists governed by foresighted managers.

2.Upcoming Trends

2.1 Technology

Essentially, the world of Big Data technology is see-sawing between two ends-both extremely desirable and crucial-one being real time analysis and the other being quantity of data stored. While the latter is done best by “data-intensive” tools such as Hadoop, the former is a forte of “compute-intensive” tools like SAP HANA. Technology is clearly headed towards integrating the two such that new tools can perform real-time analysis on humongous datasets,while being cost effective-Cloudera Impala14 is an honest effort in this direction. Going further,the increasing penetration of high-speed internet and smartphones will generate data that will make today’s Big Data seem puny. Technology will have to keep evolving to minimize human interface to allow quicker analysis and do away with complex algorithms needed to be written currently. Advanced visualization techniques to make sense of enormous datasets will have to develop.

2.2 Privacy

As Big Data becomes ever more ubiquitous,grave questions on consumer privacy and security, and proprietorship of data itself,are bound to arise. Could Big Data become Big Brother? After all,NSA relies entirely upon Big Data-pulling data from obscure digital traces-for its evil designs. Governments will continue concentrating data in the name of ensuring security,risking a dystopian society15. Not just governments,Big Data provides disproportionate heft to any corporation that holds troves of data.

As intrusiveness grows,there are bound to be trade-offs between consumers’ privacy and service, and the likelihood is that the latter will prevail16. As such trends grow,striking the right balance between risks and rewards from Big Data might well be the greatest public policy challenge of our time. Strict laws governing data-which keep consumer interest sacrosanct-will have to evolve.

3.Going Forward

Big Data’s here to stay. In times to come,it will be almost binding upon firms to invest in it to keep up with competition. Besides investments, deriving value from Big Data would require embedding it in company culture, coupled with sincere efforts to train employees to critically evaluate Big Data. Big Data carries the potential to metamorphose humanity, and it’s contingent upon humans to utilize it far beyond traditional avenues to transform education17, healthcare etc., or misuse it to usher in dystopia. Let’s hope good sense prevails.



1. Economist Intelligence Unit (2012). ‘Big Data: Lessons from the Leaders’ URL:

2. Erik Brynjolfsson. URL:

3. Harvard Business Review. Michael Schrage (November 25, 2013). URL:

4. Wikipedia-Sentiment Analysis. URL:

5. CNN Money (July 30, 2012). URL:

6. Realeyes. URL:

7. McKinsey (2011). ‘Big Data Full Report’ URL:

8. National Institutes of Health (March 29, 2012). URL:

9. Harvard Business Review (October, 2012). ‘Big Data: The Management Revolution’ URL:

10. Harvard Business Review. Jesko Perrey, Dennis Spillecke and Andris Umblijs (December 23, 2013). URL:

11. Harvard Business Review. Jesko Perrey, Dennis Spillecke and Andris Umblijs (December 23, 2013). URL:

12. Strata 2013: Kate Crawford, ‘Algorithmic Illusions: Hidden Biases of Big Data.’ URL:

13. The Scotsman. Tiffany Jenkins: ’Don’t count on big data for answers.’ URL:

14. Cloudera (October 24, 2012). URL:

15. PredPol. URL:

16. Stanford Law Review (September 3, 2013). ‘Privacy and Big Data.’ URL:

17. Christopher Rollyson. ‘Big Data in Healthcare and Education: Two Examples.’ URL:

Articles you may like