With the rise of big data, organisations are increasingly investing more into big data technologies. According to SNS Research in the report “Big Data Market: 2017 – 2030 – Opportunities, Challenges, Strategies, Industry Verticals & Forecasts”, global spending on big data technology is forecasted to be more than $57 billion this year.
With this increased focus on big data, we tend to get too engrossed in the data, and completely miss out on the real issues of the problem.
Last year, I was involved in a project to try to find out why customers were still using cash cheques for payments. From many data points that we have, it led us to an assumption that these customers were using cheques because they were free, instead of using Internet banking where they were charged for every transaction. We decided to validate this hypothesis by interviewing a few of our customers.
To my surprise, one of the customers revealed that the reason for using cash cheques was because their employees did not have any bank accounts. In fact, some of them even prefer the payments in cash, so that they can use it for their daily necessities.
If we did not talk to these customers, we may be led into thinking that reducing the cost of online transactions could be the solution to get companies to go digital.
What is Small Data?
Small data is data which is small enough for us to interpret. It connects people to the data. Put simply, small data is all about the customer (or end-user). Small data describes the customer, the actions they take, and what they truly need.
Thick data is another concept similar to small data. Both are advocating for qualitative research to complement big data.
#1 Quantitative vs Qualitative
Big data focuses on the quantitative aspect while small data focuses on the qualitative aspect. In big data, large volumes of data, and statistically normalised and standardised data are highly valued. In small data, qualitative research and customer interviews are often employed to find out more about the voice of the customer.
#2 Correlation vs Causation
Big data explains correlation while small data explains causation.
An example of how correlation is not causation can be seen in the case of Google Flu Trends (GFT) launched by Google in 2008 to predict flu activity. GFT aggregates millions of searches and tries to predict the likelihoods of a flu outbreak based on the number of flu related queries. Instead of predicting flue outbreaks, the model was overestimating flu activity, and more likely to be predicting winter. This could be due people searching for flu-like symptoms as temperatures drop, or simply people mis-diagnosing themselves.
#3 Engineering vs Design thinking
In engineering problems, the problem to be solved has already been defined, whereas in design thinking problems, the problems are usually open-ended, and deal with people who are irrational and unpredictable. Engineering problems usually has a lower tolerance for risk, for example, a new aircraft engine malfunctioning versus a personal wealth management app not being user friendly.
Big data is more suitable for the engineering problem as it requires more certainty and larger sample sizes. On the other hand, small data is more relevant for the design thinking problem. Through the use of prototypes and customer interviews, the insights acquired can be used to streamline the process, improve the customer experience, or develop a new product.
The Future of Big and Small Data
Big data is not an excuse for not being present. Sometimes, directly speaking with customers might yield more insights than relying on data alone.
Big data is only data, and only tells us the “what” and not the “why”. Together with small data (or thick data), we are better able to complement big data findings with the social context to better understand our customers.
Chan Jing Wei 