“Big data”– a term that has become the phrase de-jure of the corporate world is no longer confined to the sphere of technology. It is an integral part of business and is providing solutions to long-standing business challenges for banking and financial markets companies around the world. But is it really a big leap from the conventional structured data analytics and is transforming the business? In this article we explore the various ways in which this technology is being used by businesses across the globe to answer the big question- is big data just a passing phenomena or is it here to stay?
Big data refers to data available in various forms – not just structured but also semi-structured like XML and EDI Documents and unstructured like Text, multimedia etc. Big Data analytics is the strategy of using this huge amount of data which is now accessible through internet, mobile messages and various other platforms, to extract useful information , that can be further analyzed to help in the decision making process
Evolution of Data Analytics in business
The relational database management systems (RDBMS) can be considered as the first step to modern data storage and analysis. The goal of this system is characterized by ACID properties (Atomicity, Consistency, Isolation and Durability) of data stored by RDBMS.
The 1980’s witnessed the emergence of one of the most widespread technology as well as management strategy- Enterprise Resource Planning (ERP). ERP software necessitated the transition from existing operational databases to the data warehouses where historical and archived data were transformed and analyzed using data mining tools.
While the enterprise data warehouses are used by majority of the organizations till today for data mining to achieve business needs, the huge penetration of the Internet technology and the social media revolution has generated phenomenal quantity of unstructured data which cannot be processed by the traditional ERP systems. This brings us to the edge of a new technological revolution that corporate world is witnessing- the Big Data Analytics.
Trends in Big data Analytics
It is believed that 85% of the data comes in unstructured form. Facebook operates on around 500 terabytes of text data and similar amount of image data every day, more than 28000 multimedia messages are sent every second, while Boeing Jet engines generate 10 terabytes of information for every 30 minutes of operation. The organizations today realize the importance of the information that can be garnered through this amount of data readily available to them. Following are the key trends observed in the use of big data from a business rather than technology perspective:
- Most organisations engaged in developing big data roadmap: Just like various approaches to ERP implementation, analysts are now working on the effective strategies of implementing big data, most common of which is the 4 phase approach conceptualized by IBM- Educate, Explore, Engage, and Execute. The figure below explains the different stages and showcases the results of a recent survey of IBM on the status (stage) of big data implementation by global organizations across all industries.
Figure1: Most organisations are developing big data strategies; very few have actually embedded it into their operations
Source: Analytics: The real world use of Big Data, IBM 2012
A similar survey by Gartner in 2013 shows 64% organizations worldwide have invested or were planning to invest in Big Data, but less than 8% have actually deployed Big Data.
- Internal Data currently the primary source of Big Data: Big Data analytics is still at its nascent stage where most companies are using it on sourcing and analyzing internal data to explore the huge untapped value locked in the existing internal system. This consists of the machine generated transactional and log data, which though available in structured and semi-structured form are too large in volume and velocity to be stored and analysed by traditional systems. The most widely used unstructured form of data is text available in news media, social media, emails and other free form text sources. Various text mining tools like system T, twitter and tm package in R are being used extensively by organizations to extract valuable insights from these sources.
Figure2: Sources of Big data used by financial organisations v/s organisations across all sectors
Source: BigData @ Work survey, IBM 2012
- Major areas of application of big data: Most organisations are investing in big data to gain better consumer insights through data obtained by surveys, social media forums, recorded calls in customer service centers and other sources. Another area where big data finds extensive application is in risk analytics and fraud management, especially in high risk businesses like the Banking and insurance industries. It is also being used to achieve operational optimization to increase efficiency.
- Data Visualisation: It is method of presenting data in a form that is understandable to all and help derive business insights easily even by non-technical staff. Outburst of infographics in news and social media to explain every event be it election results or prediction of a match’s results shows that organisations widely acknowledge the effectiveness of data visualization and this is expected to be one of the key trends in business this year according to Forbes. R, SAS visual analytics and tableau provide great scope of data visualisation.
Cost Benefit analysis of Big Data
The tremendous value that big data analytics generate is expressed by tech-pundits in tens and hundreds billions of Euros per year, however, any big investment decision is based on the economic value of the investment, i.e. the potential benefits minus the costs incurred.
In a recent WinterCorp report, a Total Cost of Data (TCOD) framework has been proposed to estimate the total cost of a big data solution.
The components of cost are as follows:
- System cost includes cost of acquisition, maintenance and up-gradation of the big data system architecture (Hadoop, Teradata etc), plus the cost of power, space and cooling;
- System and data administration cost includes cost of experts for administering the system;
- Software development and maintenance cost include cost of data integration which requires developing or buying and ETL (extract, transform, and load) solution to prepare the data for analytics, cost of developing queries and procedural programs for advanced analytics and cost of developing or acquiring analytic applications for specific business needs like credit risk analysis for banks.
While system cost (cost of technology) signifies CAPEX which decreases with time, the remaining cost heads are mainly management related costs signifying OPEX that continues to increase with usage over time, posing a major challenge for organisations pursuing big data.
The benefits of big data analytics in achieving productivity and achieving competitive edge:
- Big data can increase operational efficiency by making information transparent. The biggest advantage comes from combining different pools of data each of which is managed separately in a business, and generating insights that help in optimization of resources and increasing productivity.
- With data becoming cheaper and more readily accessible than ever before, organisations relying only on proprietary data as a source of competitiveness faces severe threat from those that are torturing data from all possible sources to make better forecasts through predictive analytics. Big data thus helps in formulating business strategy and even changing the business model in response to changing macroeconomic conditions.
- It leads to remarkable increase in supply chain surplus in advanced manufacturing industries, like automobiles or aircrafts which require information from a wide range of suppliers across the world.
- Nothing can be more efficient than big data analytics in uncovering hidden trends and suspicious activities by straining massive volumes of unstructured data to detect and prevent frauds
- Defining pricing and marketing strategy has never been easier than through social media analytics, text mining and other forms of market engineering activities through big data. Marketers leverage this technology hugely to improve customer service, identify new needs and develop new products, identify new markets and enhance customer experience.
- It has a huge potential in revolutionizing risk management in banks and financial institutions, help in managing large diverse portfolios and achieving regulatory compliance.
From the above analysis it can be safely concluded that big data is here to stay and with its tremendous potential in providing business insights it can lead to a transition from the decision support systems to decision making systems in large organisations. However, the increasing operational expenses related to big data solutions is a major concern for organisations which intend to use it as a competitive asset in the long run. So, organisations have to define specific goals they want to achieve through big data, and find ways to filter the useful knowledge relevant to the goal from the large flow of data and discard the rest. Thus success of organisations using this technology will thus depend on their ability to identify the information to be extracted and the ones that they can let go of, keeping potential future benefits in mind, given an estimated cost structure.
Big Data: What does it really cost?- Special report, Winter Corporation, Cambridge MA, 2013
Trends in big data analytics- Karthik Kambatla, Giorgos Kollias, Vipin Kumar, Ananth Grama, 2014
Analytics: The real world use of big data in financial services, IBM and Saiid Business School
Why big data is the new competitive advantage by Tim McGuire, James Manyika, and Michael Chui
Top Four Big Data Trends For Businesses In 2014- Forbes
Submitted by Wei JiaMarch 16, 2012 8:57 pm
Submitted by John Paul De GuzmanMay 22, 2012 7:24 pm
Submitted by Benjamin MuellerJuly 26, 2012 3:06 pm
Submitted by Benjamin MuellerDecember 8, 2014 6:30 pm
Submitted by Joel LamJanuary 18, 2015 6:14 pm
Submitted by Anonymous (not verified)June 2, 2012 4:20 pm