Beyond the “What” Stage of Analytics: Finally, after 30+ Years!

Do you ever wonder about the value of pretty dashboard and reports — and the time we spend on them? It seems like Business Intelligence (BI) tools have been stuck in the “What” world — i.e. reporting what happened — for the past three decades, though it’s gotten a bit easier and more colorful!

It was in 1969 that Ted Codd, from IBM, invented the relational database management approach that propelled business intelligence on to the corporate stage. The 1980’s saw the emergence of data warehouses and the 90’s saw the proliferation of Online Analytical Processing (OLAP) tools. More recently a number of modern in-memory BI tools have gained popularity and today these “excel on steroids” visualization tools are pervasive and their success is undeniable. It’s interesting that many people equate dashboarding to “data science” and suggest this as BI nirvana! What, though, is the true value generation potential of these new tools?

The chart below shows BI complexity versus Value Creation (axis should be considered as “log scaled”). In a nutshell, going from raw data to information is a 10X increase in complexity and 10X increase in value creation and each subsequent stage is 10X more complex and valuable.

With the recent explosion of unstructured and structured data, the two initial stages (Data and Information) are still a challenge for most companies (BTW, people seem to be realizing that the “Data Lake” Kool-Aid doesn’t help solve this issue). On the other hand, Visualization or “What Analytics” has seen a plethora of new BI tools and solutions. While they can create incremental value for most companies, we’ve had access to similar less friendly tools for decades and we’re still stuck in the “visualization” rut!

The next 10X value creation stage is the ability for our BI tools to address “Why Analytics”. For example: “Why have sales decreased in the North East?” A single report or chart can’t address these types of questions and BI analysts spend a significant amount of time extracting information and analyzing the data to find answers. This is not data science per-se, rather these are the daily questions handled by all business analysts — from manufacturing, to supply chain, to sales and marketing, to financial analysis. The need for “Why Analytics” automation is undeniable, but the challenge of enabling this is undeniable too — else we would have had a plethora of such products already in the market today.

At Dhiva.ai we’ve have been working on addressing this challenge for the past six years and have developed an integrated analytical BI platform to enable and automate the power of “Why Analytics”. The key components needed for this include:

1. A semantic knowledge architecture to enable the representation of key corporate knowledge artifacts — (historical and dynamic learning based). The use of a symbolic semantic layer is critical and could be an issue that in-memory and pure SQL based tools will have to overcome.
2. Various machine learning/statistical models and heuristics to identify causal drivers and significant patterns that could be driving an outcome. Note, that this is not “data science” per-se, but “business analysis” (which is worth its own separate blog discussion).
3. A visualization engine that connects to Datamarts, on-premise or in the cloud, and provide supporting charts and reports, and an assertion module to summarize the key findings resulting from ML model output and semantic knowledge artifacts.
4. A natural language generation (NLG) engine to generate a cogent and grammatically accurate written (or spoken) summary of the findings from the analysis.

An automated workflow engine that drives these components in a sequential (pruned) process requires an integrated platform since external linking could result in the loss of semantic context. It’s a hard problem and the breadth of the platform needed to handle all of the mentioned moving parts is formidable, but we’ve finally pulled these pieces together and have our first generation “Why” virtual analyst ready to go!

The holy grail for BI, however, is answering the “How” question (e.g. “how can I increase my sales in the North East?”), which is an order of magnitude more difficult than the “Why” question. An integrated platform that addresses the “What” and “Why” questions, if designed right, should enable us to scale this challenge too in the near future. This is our mission, our passion and we’re on our way to climbing the “Why” and “How” mountains of insight creation. Truly exciting times!