Big Data – What’s the big deal?

comments 0






Pavneet Singh Kochhar's picture

While I am writing this essay, millions of people around the world are posting status, photos and videos on Facebook, tweets on Twitter, tagging photos on Instagram, updating their profiles on LinkedIn. We create 2.5 quintillion bytes of data everyday which comes from several sources such as sensors used to gather climate information, posts to social media sites, digital pictures and videos, transaction records, cell phone GPS signals, telemetry from automobiles and the list goes on [1]. All this is big data. Boyd and Crawford define big data as a phenomenon which is interplay of technology, analysis and mythology [2]. The use of the term “Big Data” on the internet has increased significantly over the past few years [3].


Why big data is “BIG”?

Big data is defined in terms of three Vs: volume, velocity and variety.

Volume – Data of all types are getting generated in unprecedented volume surpassing petabytes of information.

Velocity – The pace at which is data is generated is increasing with increased interactions. This calls for reacting quickly to deal with data velocity in order to maximize its value for business.

Variety- Not only big data is huge and fast, it is being generated from diverse sources which make it much more interesting and challenging.


Why should you care?

Big data will become a key basis of competition by improving productivity, creating breakthrough innovations, augmenting decision making and generating consumer surplus [4]. Big data will impact each and every one of us. Analysis of big data will provide hoards of opportunities and will bring significant advantages over those organizations that do not use it. Big data will make information transparent and available at a faster speed which can be leveraged by companies to make crucial strategic decisions. Data available to stakeholders in a timely fashion can create tremendous business opportunities. Most of the big data is in digital form which makes it more accurate and detailed than the non-digital data. Big data can be leveraged by companies to understand customer behavior and segregate customers into different categories, which can help them provide customized products based on customer’s past actions. Big data can be used to analyze the response of customers towards products and services which can help organizations improve their offerings in the future. This is particularly useful in executing high value marketing campaigns and advertise products which are much more likely to appeal to customers.


Big data will significantly impact the strategic decision making and data-driven organizations will stay ahead of the organizations lacking this capability. Earlier the senior management used to make decisions based on their knowledge and experience. However, as the customers are becoming smarter, tech-savvy and their preferences are changing, there is a need to better understand customers and their choices. Big data analysis can provide valuable insights that would otherwise remain hidden. There is an increase in the organizations using data-driven methodology to make business decisions. Research suggests that big data is used 58 percent of the time and 29 percent of the time for decision making and decision automation, respectively [5]. Furthermore, lack of data is not the issue, lack of strategy is. 85 percent of the respondents accept that ability to analyze the data and use it in real time is more important than the volume of data [5].


Usage of big data will increase competitiveness and generate value for the firms. Companies whether established or start-ups will leverage big data to innovate and produce next generation of products and services. One such example is LinkedIn, which has developed offerings based on data such as giving suggestions like People You May Know, Who’s Viewed Your Profile, People Also Viewed, People You May Know, Ads You May Be Interested In and various other features. This strategy has attracted numerous customers to LinkedIn, with the current community strength of over 200 million people spread in more than 200 countries [6].


As we collect and crunch more data, big data would be highly useful in daily activities such as helping doctors find diagnoses, identify patients which are at high risk, fight diseases, understand climate change, determine root causes of failures, generate coupons based on customer’s purchase behavior, identify privilege customers, customized recommendation to customers depending on the location, detect, prevent and arbitrate financial fraud cases and the list goes on. Big data can be used not only for new products and services but also to understand the real needs of population and to create a better society. Big data can be utilized for ‘nowcasting’ i.e., to make predictions such as predict the output of next quarter, predict GDP,  predict weather and chances of natural calamities, future business opportunities etc. Not only data that is collected primarily is useful, but also data that is generated as a by-product of our daily lives which is also called as “Data Exhaust” such as products browsed on e-commerce websites, profile viewed on social networking sites, business transactions etc. Furthermore, the secondary value of data would be much bigger than that created at the time of collection [7].


Processing large amount of data is a challenge, however, not only the data is growing, but also technologies to handle this data are rendering more power to the customers and businesses [8]. Companies are using high performance analytics to remove the constraints on variables that it can process. As companies are recognizing the power of big data to increase productivity & better understand customers, there is a strong incentive for them to invest and overcome barriers to open new avenues of opportunities, bring higher efficiency, improve services and to be more productive.


Big Data in use:

United Parcel Service Inc. (UPS), which is the largest shipment company in the world, is using On-Road Integrated Optimization and Navigation (ORION), which uses data from customers, drivers and the delivery vehicles to find the shortest route between points of origin to destination [9]. The system collects information from more than 100,000 delivery vehicles and devices carried by over 55,000 drivers to find optimal routes to reduce distance, time and fuel. Further, the data is used to perform a major redesign of the UPS drivers’ route structures. The company now tracks data on 16.3 million packages per day for 8.8 million customers, with an average of 39.5 million tracking requests from customers per day and stores over 16 petabytes of data [10]. UPS has already experienced substantial savings of more than 8.4 million gallons of fuel by cutting 85 million miles off of daily routes in 2011. Furthermore, it expects to save $30 million by cutting down one daily mile per driver.


New York City Fire Department is mining data to analyze which buildings have the characteristics of catching fire [11]. Each year several hundred buildings are gutted due to fire due to age of the buildings or electrical issues. The city officials have compiled a list of 60 such factors that are used by an algorithm to assign a risk score to each building, which can be used to inspect high risk buildings.


Global Pulse[1] (An initiative by UN) is a series of research projects in collaboration with private sector and academia to use digital data and real-time analytics to analyse the impact of socio-economic stress on the populations such as monitoring women’s and children’s health, women’s role and equal opportunities in the workplace, analyzing barriers to the access of loans to small businesses in Kenya [12].


Healthcare is one of the largest sectors of US economy which accounts for nearly 17 percent of GDP and employs 11 percent of the workforce. The increasing cost of healthcare not only makes it difficult for citizens to bear medical expenses, but also creates burden on the country’s finances by increasing the debt. The US government is investing in several research initiatives which leverage big data and analytics to find solutions for vexing healthcare questions which have been challenging scientists and researchers for a long time. One such initiative is BRAIN, which aims to study the human brain and find ways to prevent and cure disorders such as Alzheimer’s disease, Parkinson’s disease, schizophrenia etc. [13]. AstraZeneca has partnered with HealthCore, to conduct studies on chronic diseases and to find the most effective and economical treatments for them [14]. AstraZeneca will combine HealthCore data along with its own clinical trial data in an electronic medical record system to find ways to improve health outcomes, to guide R&D investment decisions and to manage the total cost of care.


As companies are becoming smart and customers’ even smarter, big data has created the raucous and with big data already making inroads into almost every sector, it is sure to become the next BIG thing.







[1] Bringing big data to the enterprise, retrieved from {}

[2] Critical Questions for Big Data - Provocations for a cultural, technological, and scholarly

phenomenon, retrieved from {}

[3] Improving Decision making in the World of Big Data, retrieved from {}

[4] Big data: The next frontier for innovation, competition, and productivity, retrieved from {}

[5] The Deciding Factor: Big Data and Decision Making, retrieved from {}

[6] 200 Million Members!, retrieved from {}

[7] Big Data: A Revolution That Will Transform How We Live, Work, and Think, Book authors {Viktor Mayer-Schönberger & Kenneth Cukier}

[8] At Last! Users can Regain Control of their Big Data, retrieved from {}

[9] Meet ORION, Software That Will Save UPS Millions by Improving Drivers' Routes, retrieved from


[10] Big Data in Big Companies, retrieved from {}

[11] How NYC is using data to fight fires, retrieved from {}

[12] United Nations Global Pulse, retrieved from {}

[13] The Brain Initiative, retrieved from {

[14] Seizing the potential of ‘big data’, retrieved from {}