The truth about big data

A bit of a rant

Much of the discussion I am witnessing around big data seems to be missing the point. We understand that something significant is happening in the technology arena. Mobile, cloud, the internet of things, data, analytics are all converging on something. But what? In order to see the true potential of big data and analytics requires you to be able to predict what disruptive changes lie ahead for your industry.  Did the publishing industry see Amazon coming?   Did the advertising industry see Facebook coming?   Did the gaming industry predict the surge in online gaming?  Did the travel industry foresee the success of online booking portals?

Big Data

Envisioning the future as part of strategy formulation is a vital capability for any company.  Some companies do invest a lot of money creating a mock up version of the future that can be seen, touched and experienced in order to motivate and direct their research efforts.

I remember visiting the house of the future at the Redmond Campus 10 years ago before social big data as we know it was dreamed of. This house depicted a lifestyle where the living space reacted to your needs, the time of day, your mood, who you were with and created the living environment most suited for that moment through manipulating lighting, sunlight, images on displays, energy usage and so on. Put together this served to make the living space experience optimal. Visiting this house was a valuable opportunity in order to see exactly where technology investments would be needed in future to realise this vision. In Microsofts case an integrated entertainment and information system; home automation hardware; energy optimisation software and systems all were necessary to realise the vision. (Interestingly many of the innovations that I witnessed in Redmond are now mainstream and taken as part of the normal specification of (upmarket) homes).

So when it comes to big data; in order to understand our investment needed now are we able to see the possibilities clearly enough to create a compelling vision?

Prediction is an inexact science. Anyone can do it. In the area of human behaviour most get predictions wrong most of the time. A few people do get it right however, and those that do are in demand for their  analytical capabilities, or intuition or technique; or all three.

I am at a conference at the moment and I loved the quote in Andries Botha’s presentation on Big Data this morning:

“Big data is like teenage sex. Everyone talks about it. Everyone thinks everyone else is doing it. But only a few actually know what its really about.”

Without a unified vision of what big data holds in future; may I venture an opinion:

The future of big data is the ability to represent and validate models of behaviour that are context related and which can be used to predict useful recommendations to people in similar contextual situations.

In a more practical sense for example; can we collect enough information about a persons profile and behaviour to predict what their needs will be next – and offer advice or suggestions to them on how best to fulfil those needs in a way that ultimately favours your (or your clients) product or service?

Its not a perfect vision for sure, but for the moment let’s work with this. Amazon does something like this when it suggests what books you might like.

So lets get started and build your big data model.

  1. Open Excel.
  2. Stare at the screen a while.
  3. Close Excel.

Some companies are better positioned to start the big data journey.  They already have lots of data about individuals:  financial, health, insurance, banking, social, education.  Most companies don’t have any big data yet.

The reality of big data is that it is only “big” for some. If you don’t own the data or have legitimate access to it you cannot possibly analyse it. And if you can’t analyse it how can you model the underlying systems in order to make predictions? So in reality unless a company has a deliberate strategy to collect sufficient relevant information big data is as useless as no data at all.

We can probably identify the biggest owners of personal data on the planet. Facebook is certainly one, and there are a handful of others. This is why software companies want you to connect to Facebook in order to log in. When you connect to Facebook and without further thought give the “app” permission to read your likes and posts and those of your friends; in effect you are giving the company that owns the app licence to access  data that otherwise they could not get hold of.

Smart. I am not sure how many people actually think about this when wiring up their apps to Facebook. Not only are you forfeiting your right to privacy, but you actually allow this app to intrude on the privacy of your friends.

It is only when you sit down and try to think of real human behaviour models that you might be able to develop that the reality of the data limitations sets in. There are too many dimensions to behaviour to model meaningfully unless you have access to really large and diverse and relevant “big” data. And without a model that is reliable you cannot predict much with any confidence. And without prediction how can you relate context data to predicted outcomes that can be utilised for business/education or any other discipline.

I have already mentioned that without being able to predict something useful that results in decisions or actions that would otherwise not been possible, the true potential of big data is lost. Many people talk about this analysis as if it is quite trivial. Not so. The ability to understand the relationships between entities in a statistically uncertain environment and to extract patterns and models from these relationships are not skills readily accessible to most companies. It is no coincidence that  analysts and statistical modelling people are those skills in shortest supply and their  pay cheques can be quite high indeed. Sadly for entrepreneurs and smaller companies who don’t have scale; such analytical capability is going to come at a price beyond the affordability of most small (and even many large) businesses.

Wise people say that you learn the most by getting started on any endeavour. And this advice holds true when considering how you might leverage the vast amounts of data out there to your advantage. Be realistic though as to how much data you really have, how difficult it will be to create statistically valid behavioural models and how difficult it will be to find the skills to build and maintain these models. A starting point is to decide what data you need and create strategies for how you will gain access to this information.

The business case for investment in big data capability is not clear for most companies.  The reality is that only a handful of companies are prepared to make the real big investments necessary to build a serious big data capability.  The rest hope that they will be able to purchase this capability at some stage in the future.

Return on investment is very difficult to calculate using traditional ways because it is hard to quantify the probability that millions of people will actually respond to your targeted recommendations,  especially if your competitors are doing the same.

Human behaviour is notoriously difficult to predict – and even more so when the dominant consumer players like Facebook, Amazon and Google plan to continue to create new expectations of user behaviour.  The future will go far beyond targeted advertising.  If advertising is your strategy you need to recognise that the game will continue to change.  Intrusive advertising is thankfully becoming a relic of the past in favour of more personal interaction.

Understanding big data is therefore also about understanding transformational use of technology that will continue to disrupt business models in the future.

I contend that big data for many is actually not big at all, it is actually small data. And the more mature practices of traditional business intelligence (BI) projects apply to these smaller data sets. Before you leap into something brand new perhaps you should challenge whether you have the basic BI capabilities in place. This BI foundation will certainly help you along the maturity curve necessary to exploit big data potential in future.

Many companies are in the same place. Here are three recommendations for consideration for these companies:

(1) Envision what will be possible with big data in your specific industry to give you competitive advantage. Set this as a goal.
(2)Plan how you will actually get access to this big data and start developing clear strategies to do this immediately with your existing products/service lines.  Collect data starting today.
(2)Build a solid platform of BI capability. Look to extending this capability into strategic big data projects down the line.

If you have any thoughts on how you might utilise big data in your own industry, or related questions feel free to drop me a direct line or use the comments below.

Image by