Organisations that invest in big data analytics typically do so to answer one of four questions: what happened, why it happened, what will happen, or what should be done about it. Those four questions correspond to descriptive, diagnostic, predictive, and prescriptive analytics. Each requires different data, different tools, and different levels of organisational maturity. The practical value of big data lies not in collecting it but in moving up that chain.
The term big data was popularised in a 2001 Gartner report by Doug Laney that described three Vs: volume, velocity, and variety. Since then the global datasphere has grown from 33 zettabytes in 2018 to a projected 175 zettabytes by 2025, according to International Data Corporation forecasts. That growth has made the distinction between analytics types more important, because the cost of storing and crunching data falls only if the organisation knows which questions it is trying to answer.
The four analytics types and how they are applied
Descriptive: what happened
Descriptive analytics summarises what has already happened. A retailer reports monthly sales by region. A hospital displays average emergency department wait times. The outputs are dashboards and standard reports. This is the most common form of analytics in most organisations because the data is already structured and the tools are mature.
Diagnostic: why it happened
Diagnostic analytics asks why something happened. If sales dropped in a region, diagnostic analytics might correlate the drop with a price change, a competitor promotion, or a weather event. The technique relies on drill-down queries and hypothesis testing. It is harder than descriptive analytics because it requires clean data and the willingness to test multiple explanations.
Predictive: what will happen
Predictive analytics uses models and machine learning to estimate future outcomes. A manufacturer forecasts when a machine will fail based on vibration and temperature sensor data. A bank estimates the probability that a loan applicant will default. Accuracy depends on the quality and volume of historical data and on the model being retrained as new data arrives.
Prescriptive: what to do about it
Prescriptive analytics recommends a course of action. It combines forecasting with optimisation or simulation. A logistics company might use prescriptive analytics to reroute trucks in real time when a delivery hub is congested. This is the least deployed type because it requires integration with operational systems and a culture that trusts algorithmic recommendations.
Data sources organisations combine
Internal and external inputs
Big data analytics rarely relies on a single feed. Organisations combine internal transactional records with external data to get a fuller picture. Common internal inputs include point-of-sale systems, customer relationship management databases, server logs, and sensor readings from Internet of Things devices. External inputs include social media feeds, government datasets, credit bureau reports, and third-party demographic or weather data.
The integration challenge
The challenge is joining these streams at scale. Transactional records are often structured and stored in relational databases. Social media feeds are unstructured text. Sensor data is time-series with high velocity. The ISO/IEC 20547 series of standards addresses big data reference architecture and security, providing a framework for organisations building systems that must handle all three types.
Start small
In practice, most organisations start with two or three inputs and add more as they gain confidence in the analytics pipeline. A common mistake is ingesting too many feeds before the governance and data quality processes are in place, which produces inconsistent results and erodes trust in the analytics function.
Industry use cases: retail, healthcare, manufacturing, financial services
Retail
In retail, big data analytics powers demand forecasting, personalised recommendations, and dynamic pricing. A retailer might combine point-of-sale data with weather feeds to predict demand for seasonal items and adjust inventory across stores. The measurable outcome is reduced stockouts and less markdown waste.
Healthcare
In healthcare, a 2011 McKinsey Global Institute report estimated that the US healthcare sector could create more than $300 billion in value annually through big data analytics. Applications include predicting patient readmissions, identifying cohorts for clinical trials, and optimising hospital staffing. The sensitivity of health data means that analytics programmes in this sector are heavily shaped by privacy regulation and ethical review boards.
Manufacturing
In manufacturing, predictive maintenance is the flagship use case. Vibration sensors on motors, temperature sensors on bearings, and current draw data from programmable logic controllers feed models that flag equipment likely to fail. The payoff is fewer unplanned outages and longer asset life. Manufacturers also use descriptive analytics for yield analysis and prescriptive analytics to adjust production parameters in real time.
Financial services
In financial services, fraud detection and credit risk assessment have used quantitative models for decades. The shift to big data has allowed banks to incorporate alternative data such as mobile phone usage patterns and utility payment histories to score thin-file borrowers. Regulatory requirements around model explainability and fairness constrain how these models are built and deployed.
The role of machine learning and statistical modelling
Choosing the right approach
Machine learning and statistical modelling are the engines that extract patterns from large datasets. The choice between them depends on the question and the data. Traditional methods such as linear regression and logistic regression are interpretable and work well when the relationships in the data are relatively simple and the sample size is moderate. Machine learning methods such as random forests, gradient boosting, and neural networks can capture complex non-linear interactions and scale to very large datasets, but they are harder to explain.
From MapReduce to the cloud
MapReduce, patented by Google in 2004, provided the programming model that underpinned early large-scale data work and influenced the development of Hadoop. Apache Hadoop, created by Doug Cutting and Mike Cafarella in 2005, became a foundational technology for big data analytics by allowing organisations to distribute computation across clusters of commodity servers. Today, many organisations use cloud-based services that abstract away the infrastructure layer, but the core concepts of distributed storage and parallel computation remain the same.
Production reality
The practical choice of algorithm is often determined less by theoretical performance and more by the organisation's ability to serve the model in production. A model that achieves 99% accuracy but takes 10 seconds to score each record may be useless for a real-time fraud detection system that needs a decision in milliseconds.
Infrastructure, governance, and privacy
Storage and compute
Building the infrastructure to ingest, store, and process big data involves decisions about storage format, compute engine, and data pipeline orchestration. Many organisations use a data lake architecture in which raw data is stored in its native format and schema is applied at read time. Others use a data warehouse for structured data and a data lake for semi-structured or unstructured data. The trend is toward unified platforms that can handle both.
Governance
Governance covers data quality, metadata management, and access controls. Without governance, a data lake becomes a data swamp where no one trusts the numbers. The European Union's General Data Protection Regulation (GDPR), which came into force on 25 May 2018, imposed strict rules on the handling of personal data. It affects analytics programmes that involve EU residents, requiring organisations to document the lawful basis for computation, implement data minimisation, and allow individuals to request deletion of their data.
Privacy and ethics
Privacy and ethical considerations go beyond legal compliance. Models trained on historical data can perpetuate or amplify biases present in that data. A hiring model trained on past successful hires may discriminate against candidates from underrepresented groups if the historical hiring pool was not diverse. Organisations that deploy analytics at scale need processes for auditing models for fairness and for explaining decisions to regulators and affected individuals.
Operationalising analytics outputs
From report to decision
The final step is moving analytics from a report to a decision. Descriptive analytics outputs typically feed dashboards that managers review weekly or monthly. Predictive and prescriptive analytics need to be embedded in operational workflows. A predictive maintenance model is only useful if it triggers a work order in the maintenance management system. A fraud detection model is only useful if it blocks a transaction or flags it for review in real time.
Process change, not just technology
Operationalising analytics requires changes to business processes, not just technology. The team that builds the model is rarely the team that acts on its output. Organisations that succeed at this stage invest in change management, train frontline staff to interpret model outputs, and create feedback loops so that the model improves when its predictions are wrong.
The integration difference
The organisations that capture the most value from big data are not the ones with the most data or the most sophisticated algorithms. They are the ones that integrate analytics into the rhythm of the business: a retailer that adjusts prices every morning based on yesterday's sales and today's forecast, a manufacturer that schedules maintenance based on predicted failure probability rather than a fixed calendar, a bank that re-evaluates credit limits monthly rather than annually. That integration is the difference between having analytics and using analytics.
Key facts
- Origin of 'big data': 2001 Gartner report by Doug Laney defining volume, velocity, variety
- Hadoop creation: 2005 by Doug Cutting and Mike Cafarella, open-source distributed processing framework
- MapReduce patent: 2004 by Google, programming model for large-scale data processing
- GDPR effective date: 25 May 2018, strict rules on personal data processing
- McKinsey healthcare estimate: $300 billion annual value potential for US healthcare through big data (2011)
- Global datasphere growth: 33 zettabytes (2018) to 175 zettabytes (2025) per IDC forecast
- Big data architecture standard: ISO/IEC 20547 series for reference architecture and security
Frequently asked questions
What is the difference between descriptive and predictive analytics?
Descriptive analytics summarises historical data to answer what happened. Predictive analytics uses models or machine learning to estimate what will happen, based on patterns in the historical data.
Which industry gets the most value from big data analytics?
A 2011 McKinsey Global Institute report identified healthcare as a sector with more than $300 billion in annual value potential. Financial services, retail, and manufacturing also have well-documented returns from fraud detection, demand forecasting, and predictive maintenance respectively.
What infrastructure do organisations need to do big data analytics?
Most organisations use a combination of data lakes for raw storage, compute engines such as Apache Hadoop or cloud-based services for processing, and data pipeline tools for ingestion. The ISO/IEC 20547 series provides a reference architecture for such systems.
