learning engineering
July 18, 2018

Let’s start out with a few data points:

  • Data growth continues to outpace even the most aggressive projections
  • Survey research continues to show that ‘gaining deeper insights’ into data is the biggest single driver of projects in the space
  • And yet, somewhere between 60 and 85% of all ‘big data’ projects fail

What’s a curious, enterprising or even desperate executive to do? Are we really going to run 3 projects to see one possibly succeed just by ‘the law of averages?’ Is that truly business intelligence? 

In practice, the idea of “business intelligence” has become much more about data visualization with known tools than it is about what’s actually important. Decision support and visibility.

This visual-first approach via dashboards and reports filled with graphs and tables was effective even in the early days of the information age. However, the matters seems to have been given little thought beyond that.  Certainly since the evolution of ‘web scale’ and ambient data by way of ubiquitous technologies. For example, the Internet of Things and ‘connected everything.’



Can We Look At Things Differently?


To put it in fundamental terms: the current approach to business intelligence, seen as a communications channel for imparting information to the business, has long since been over-saturated.

To put it simply, we just have more ‘stuff’ than we know what to do with. A massive amount of data forms the foundation of the information from which we derive our interpretations from. And we’re struggling to figure out how to approach it.

We (humans) can’t make sense of that much information at once. It’s no wonder that terms like ‘information overload’ and ‘analysis paralysis’ are so common. These terms grew roughly along with the rise in popularity of ubiquitous computing; from consumer to enterprise applications and beyond.

To put it in terms of Claude Shannon’s take on information theory: If we can’t compress the data to make it useful with our current methodologies, clearly we need a better channel or process. Alternately, we need to change our expectations. We either forget about our ‘big data dreams’ of gaining insight from all of these rich data sources; or figure out how to consume this information differently.

We believe that the scale of change that Artificial Intelligence and Machine Learning represents an opportunity. A chance to change our approach to data and what “business intelligence” can mean moving forward.


Let’s Ask Better Questions


So where are we focused? We dig in on four things:

Finding the next data warehouse, reporting tool or other micro-evolution in Business Intelligence reporting in today’s world is like getting stuck in traffic on the freeway and finding your ‘secret back road’ way home: sooner rather than later, everyone else is going to discover it and you’ll be right back stuck in traffic.

  • What is the question being asked?
    • Fundamentally, even if the universe of possible analyses is a Cartesian or even multi-dimensional product, the list isn’t infinite. The more we understand these questions, the easier it is to identify the right approach to apply to similar question(s).
  • How wide?
    • From how many different data sources, contexts, etc. do we need to integrate to answer the questions being posed?
  • How deep?
    • Just how much detail do we need to capture from the variety of sources?
  • How fast?
    • What’s the tolerance between events happening ‘in the real world’ and your awareness of it in the context of the question being asked?