Skip to content

August 20, 2012


Innovation and Big Data: A Roadmap

by Ravi Kalakota

The bleeding edge of data and insight innovation is around next generation digital consumer experience.  Consumer behaviors are rapidly evolving….always connected, always sharing, always aware. Obviously new technology like Big Data drives and transforms  consumer behavior and empowerment.

With the influx of money, attention and entrepreneurial energy, there is a massive amount of innovation taking place to solve data centric problems (such as the high cost of collecting,  cleaning, curating, analyzing, maintaining, predicting) in new ways.

There are two distinct patterns in data-centric  innovation:

  • Disruptive innovation like predictive search which brings a very different value proposition to tasks like discover, engage, explore and buy and/or creates new markets!!
  • Sustaining innovation like mobile dashboards,   visualization  or data supply chain management which improves self service and performance of existing products and services.

With either pattern the managerial challenge is moving from big picture strategy to day-to-day execution.  Execution of big data or data-driven decision making requires a multi-year evolving roadmap around toolset, skillset, dataset, and mindset.

Airline loyalty programs are a great example of multi-year evolving competitive roadmaps. Let’s look at BA’s Know Me project.

British Airways “Know Me” Project

British Airways (BA) has focused on competitiveness via customer insight. It has petabytes of customer information from its Executive Club loyalty program and its website. BA decided to put customer big data to work in its Know Me program. The goal of the program is to understand customers better than any other airline, and leverage customer insight accumulated across billions of touch points to work.

BA’s Know Me program  is using data and applying it to customer decision points in following ways:

  • Personal recognition—This involves recognizing customers for being loyal to BA, and expressing appreciation with targeted benefits and recognition activities
  • Personalization — based on irregular disruptions like being stuck on a freeway due to an accident – A pre-emptive text message… We are sorry that you are missing your flight departure to Chicago. Would you like a seat on the next one at 5:15PM.  Please reply Yes or No.
  • Service excellence and recovery—BA will track the service it provides to its customers and aim to keep it at a high level. Given air travel constant problems and disruptions, BA wants to understand what problems its customers experience, and do its best to recover a positive overall result
  • Offers that inspire and motivate—BA’s best customers are business travelers who don’t have time for irrelevant offers, so Know Me program analyzes customer data to construct relevant and targeted “next best offers” for their consideration.

The information to support these objectives is integrated across a variety of systems, and applied in real-time customer interactions at check-in locations and lounges. Even on BA planes, service personnel have iPads that display customer situations and authorized offers. Some aspects of the Know Me program have already been rolled out, while others are still under development.

The Need for New Data Roadmaps

New IT paradigms (cloud resident apps, mobile apps, multi-channel, always-on etc.) are creating more and more complex integration landscapes with live, “right-now” and real-time data. With data increasingly critical to business strategy, the problems of poor quality data,  fragmentation, and lack of lineage are also taking center stage.

The big change taking place in the application landscape: application owners of the past expected to own their data. However, applications of the future will leverage data – a profound change that is driving the data-centric enterprise.  The applications of the future need one “logical” place to go that provides the business view of the data to enable agile assembly.

Established and startup vendors are racing to fill this new information management void.  The establish vendors are expanding on this current enterprise footprint by adding more features and capabilities. For example, the Oracle BI stack (hardware – databases – platform – prebuilt content) illustrates the data landscape changes taking place from hardware to mobile BI apps.  Similar stack evolution is being followed by SAP AG, IBM, Teradata and others.  The startup vendors typically are building around disruptive technology or niche point solutions.

To enable this future of information management,  there are three clusters of “parallel” innovation waves: (1) technology/infrastructure centric; (2) business/problem centric; and (3) organizational innovation.

IBM summarize this wave of innovation in this Investor Day slide:


Data Infrastructure Innovation

  • Data sources and integration — Where does the raw data come from?
  • Data aggregation and virtualization- Where it stored and how is it retrieved?
  • Clean high quality data — How does the raw data get processed in order to be useful?

Even in the technology/infrastructure centric side there are multiple paths of disruptive innovation that are taking along different technology stacks shown below.  

  1. Enterprise Data Stack;
  2. The Hadoop Ecosystem (Big Data) Stack;
  3. Enterprise Information Management;
  4. BI platforms, Analytics Tools and Visualization.

The big enterprise data trend is that enterprise are moving beyond the traditional data warehouse.  Relational database management systems (RDBMS) have been deployed as enterprise data warehouses (EDW) for analytic applications when most of the questions were known up front. Their care and feeding required a sophisticated, multistep process and a lot of time. This process supported the need for strong information governance, verifiability, quality, traceability, and security.

The sweet spot for traditional EDW includes both running the same reports and queries and tracking the same set of metrics over time. But if the questions changed, things would break and big parts of the end-to-end process would require redeveloping — often starting with the collection of new source data.

While billions have been invested in traditional EDW architectures they are limited in addressing the new class of  analytic applications that require speed and flexibility.  Planned deployments complex and require lengthy roadmaps.  Data is created in different ways, stored in various locations, and accessed in non-traditional ways making EDW architectures antiquated quickly.

So a new wave of investment in next generation data architecture (such as data virtualization) has just begun.  Different firms are attacking different feature/functionality gaps or problems.  All relevant but inter-dependent if you are a CIO or CTO.

The fact that the tools and technologies to tame data and enable seamless integration of on-premise and public/private cloud-based data are improving. The vision is agile assembly that accomplishes three things:

  1. accelerates simultaneous federated access to multiple existing data sources;
  2. creates centralized and unified “virtual views” via configuration as opposed to development; and
  3. seamlessly enables connections to sources to create new virtual views.

While the Big Data infrastructure is evolving rapidly, there is a tremendous amount of prototyping and piloting taking place on the business side.

Process and Vertical Centric Innovation

The business innovation is happening along three fronts:

  1. point solutions;
  2. process solutions;
  3. vertical specific solutions.

We’re talking about innovations in business models and practices—everything from how a company generates insights to innovative ideas for creating new markets.

Sometimes new business models and data are converging in interesting ways around certain point solutions. Take for instance Splunk which solves the machine data problem. Under the covers Splunk is designed to manage log data.  Splunk has built-in functions to understand time stamps and time series data as well as other fields related to log data. Given its focus, Splunk has a predefined a set of dashboards to track certain well-known metrics.

Organizational Innovation – How to Execute Better

Technology and business innovation has to be supported by organizational innovation — how to get your organization to actually acquire, implement, and put the tools and technologies to use. The people, cultural, and political issues surrounding data are very big and very real.

Technology centric innovation invariably creates decision dilemmas between the “status quo” (e.g., Oracle, SAP or IBM stack) and emerging (e.g., Hadoop).   Dilemmas emerge when there is tensions between two apparent opposite ideas or concepts.  In business we face these dilemmas all the time:  cost vs. quality, centralization vs. decentralization, stability vs. change, short term results vs. long term competitiveness.  Dilemmas are dynamic but inevitable.  They don’t go away.  They must be managed by making choices and creating a comprehensive multi-year roadmap.

The challenge for management is how to integrate these various streams of innovation into a cohesive multi-phase roadmap that is relevant for their corporation.   How to articulate and put their version of the innovation pattern together.  Also how to version this pattern so the firm can execute without getting all tangled up in complexity.

What every corporation needs to do is create a relevant and unified roadmap that brings the technology and business elements together under one umbrella. This is essential for communicating and setting a directional strategy.

The emerging DATA opportunity is clear to most firms. Which path you take to innovation depend on two critical philosophical questions (or perspective and context):

  • is Big Data disruptive or incremental for your applications and infrastructure?
  • Is your approach a Platform-to-Solution (common services driven) or a Solution-to-Platform (business problem driven) approach?

We are in the early days of this virtuous cycle.  Technology innovation and business model innovation are complementary.  Technology standards and maturity will also encourage business-model innovation.  Business model innovation fuels another round of technology investments.

Right Org Structure for “Data-as-a-Service”, “Information-as-a-service” or “Analytics-as-a-service” Innovation

You have data. You’ve hired data scientists. Now, how do you structure your teams? Do you keep the data scientists together to allow them to learn from each other? Or do you assign them individually to project teams so they can share their knowledge and become closer to the business?

Also data needs context for it to be useful… Every organization requires custom-made data-driven solutions to a variety of business applications.  This means an understanding of business challenges and delivering action oriented solutions.  What tools and techniques are the best fit for solving a particular set of business problems?

Success depends on having the right leadership and talent in right place. Some ingredients include

  • Strong talent pool enabled by well-developed hiring, training and performance management processes
  • Analytical tools & techniques & methodology specifically applicable to your industry and context
  • Ability to ramp up and down at short notice
  • Integrated delivery resources across onshore and offshore offering economical cost structure

Ability to understand the data science value chain, handoffs, leverage points and talent needed is what makes the difference between good firms and great firms.

Big Data Value Chain

Execution of Innovation

All these have to be aligned with an effective and efficient solutions lifecycle – Design, Develop, Discover and Deliver.

  • In the ‘design’ stage, the target is to understand the core business challenge and lay out an analytics solution. In this stage the sources of data, volume of data and resources available are examined
  • The ‘develop’ stage consists of gathering and managing available data. In many cases, data from external sources are incorporated to make more meaningful inferences. This is followed by advanced data analysis and modeling with the use of a wide range of analytic software (SAS, CART, MARS, JMP, R, SPSS, Excel, etc.).
  • The ‘discover’ stage is about leverging modeling approaches such as classification and regression trees, multivariate linear regression and other advanced techniques are used in building each model. With some amount of manual intervention the models are fine-tuned and revalidated to improve performance. Improved performance leads to more reliable business decisions based on relationships within the data.
  • In the ‘deliver’ stage, the learned models are employed and the results are interpreted to form coherent business rules that are implemented in a client environment.

Every stage has to supported by a systematic quality assurance process to ensure the quality throughout the project design, development and delivery.


Big Data Analytics is all about technology & business model innovation & execution.

The unsexy way to describe what Big Data Analytics….  is “enterprise performance management”: The act of gathering up the data that business generates in the course of its operations and creating a visualized environment from the live data. Not a snapshot of what it was yesterday or last week, as existing software tools often do in this space, but from the live, “right-now” data.  The analytics can be presented on an iPad or iPhone or other smart mobile device.

Insight with simplicity is the key goal… Almost all companies talk about the need to mine their customer and transaction data.  Using a test-and-learn approach, leading firms tend to design, execute, manage, and measure market-facing DATA experiments. Every new idea has risk and the potential to be partly wrong and partly right, the goal is to quickly iterative-and-learn before committing to an expensive scaling of the idea. The goal is make data analytics more and more accessible and easy enough to use. In the end,  analytics   can only make a difference if it’s usable and understandable. Otherwise, all that effort is going to waste.

Roadmap to a DATA centric model is a journey…..requires a new way of thinking about what a business does, and how it does it.  However, what is not clear is the roadmap to becoming a DATA centric business. What is the complicating matters is the complexity and challenge of making phased investment decisions in a shifting technology and services landscape as the gap between the technology investment and the capture of value usually lags by several years.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments

%d bloggers like this: