Skip to content

February 18, 2016


Big Data is Entering the Trough of Disillusionment

by Ravi Kalakota

hype cycleBig data, data lakes and becoming data-driven is no longer news or novel. Hype is not enough anymore.

The challenge today for leaders in every enterprises is (a) how to monetize data? (b) how to create enterprise class platforms instead of sandboxes? Basically, how can data, analytics and insight drive digital operations and digital transformation?

The paradox is interesting.  While leadership is struggling with value creation, the long term data trends are favorable: (1) data continues to grow exponentially and outstrip our ability to convert into insight;  (2) data consumption is evolving… Consumers and employees have more interactions with data through mobile apps than they do through desktop browsers; (3) Analytics – predictive and prescriptive – is gaining traction in several industries and business processes.

In the first wave of excitement around big data, there is a massive amounts of investment in stand-alone pilots and sandboxes.  Some experiments worked, but many failed to deliver.  The linkage between analytical projects and the everyday business applications (systems of record, systems of engagement, systems of insight) have mostly been missing.

In the next wave, we are seeing a tighter alignment or foundational underpinning between analytics (even machine learning) and traditional business applications. Take for instance Salesforce. Machine learning has become increasingly important to Salesforce, which has acquired PredictionIORelateIQ and Tempo AI, among other companies.

This implies a massive transformation wave (and upgrade cycle) across existing:

  • Systems of insight  (Reporting, BI, Analytics platforms)
  • Systems of engagement (CRM, SFA)
  • Systems of record (ERP)

The bigger transformation challenge is around how to systematically clear the bottlenecks in each of the above so that end-users can (1) access real-time data;  (2)  slice and dice the data for actionable insights from any device, anywhere; (3) convert the insights into guided decisions.

The directional strategy is clear, but can leadership get behind it and implement it swiftly and effectively?

This impending transformation is both exciting and daunting at the same time.  Application development and delivery (AD&D) teams are overwhelmed.  Leadership (in most corporations) in terms of vision or directional clarity is often weak or missing.  Strained relationships and misaligned business and technology teams is unfortunately the norm, not the exception.

Something is out of whack.

What I have observed in multiple engagements and research is that every year, billions of dollars are being spent with the consulting industry on establishing a corporate data strategy. Millions of hours of leadership time is invested in the strategy effort.

Even more billions are being spent on the core foundational “data lake” strategies – using Hadoop to create a large scale data dump. The more advanced form of this is to add Master Data Management (MDM) on top to create a “single golden view” of customers, employees, products or accounts. The hope (and prayer) is that this will enable companies to derive a more accurate picture of their business.  This is the hypothesis behind many “data lake” initiatives. The results and business value have been sketchy so far. So, is the strategy wrong or execution flawed?

Executing the data/analytics strategy around systems of engagement, record and insight is where the pain and the largest costs to the organization come into play. This is where discipline and talent becomes critical, and where competitive advantages are either won or lost.  But most firms starve the application development and delivery (AD&D) teams — limited budgets, basic talent to do cutting edge solutions,  unrealistic deadlines, constantly changing requirements. Few invest in product or program management to ensure the entire organization is involved, aligned, and ready to make it happen?

As a result, most corporations today have an awfully poor batting average when it comes to analytics projects or data informed business processes. This also may lead to the frequency of failure of most large-scale data-driven corporate change initiatives.  

The hype cycle is not a new phenomenon, but one that repeats itself with each innovation that somehow captures people’s imagination.  We have to be patient in the next 2-3 years and dig ourselves out of the big data Trough of Disillusionment🙂

I am bullish on data long term.  The future is becoming more data-driven everything. Every industry (financial services, healthcare, retail, industrial) is going digital powered by data.


Source: It’s Time To Upgrade Business Intelligence To Systems Of Insight Supercharge BI With Agility, Big Data, And Insights To Drive Action by Boris Evelson, July 20, 2015


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments

%d bloggers like this: