“Running a company is an endless quest to find out things you don’t know“
– Jeff Immelt, CEO GE
What will 2012 bring? Recently, I attended the CIO Executive Leadership Summit in Greenwich, Connecticut. I was particularly intrigued by the presentation by the new CIO of IBM, Jeanette Horan where she presented the projects she was tackling and how IBM is thinking about business analytics.
IBM is making a bet that “true leaders” will develop the capabilities required for making good and timely decisions in unpredictable and stressful environments.
IBM is adapting to this new data analytics reality by a rapid-fire acquisition strategy: Cognos, Netezza, SPSS, ILog, CoreMetrics, Algorithmics, OpenPages, Clarity Systems, Emptoris, DemandTec (for retail). IBM also has other information management assets like Watson, DB2 etc. They are building a formidable capability around the value chain: “Raw Data -> Aggregate Data -> Intelligence ->Insight -> Decisions” . They see this as a $20Bln opportunity. Read more
Apple with its iCloud offering is attacking the consumer facing digital content big data problem. Big Data is challenging on many fronts from the insights (e.g., analytics and query optimization), to the practical (e.g., horizontal scaling), to the mundane (e.g., backup and recovery).
On June 6th, 2011 Apple Inc. launched its new purpose built digital locker service called iCloud for its 225 million iTunes accounts that frees the end-user from the tyranny of the device. The iCloud service is a cloud offering that would allow users to store digital files such as photos, MP3 music, videos and documents in the cloud and access them from Internet-connected devices like iPhones, iPads, iPods, iMacs and others.
So, what’s the big deal? They are addressing a classic BI data management problem: How to free up data trapped in “device and application jails” in a user-friendly way. The “scan and match” concept is quite applicable to large scale Enterprise Datawarehouses which suffer from data integrity issues as edge data capture and consumption devices proliferate.
Data ingestion, governance and management is a huge problem facing large organizations. As data volumes double every year, not having a basic data management strategy will become an Achilles heel. Most organizations unfortunately don’t know what data assets they have, where these assets are, how they are organized and how well they are secured. Apple shows a neat way to address the Big Data problem in personal cloud management.
Data growth curve: Terabytes -> Petabytes -> Exabytes -> Zettabytes -> Yottabytes -> Brontobytes -> Geopbytes. It is getting more interesting.
Analytical Infrastructure curve: Databases -> Datamarts -> Operational Data Stores (ODS) -> Enterprise Data Warehouses -> Data Appliances -> In-Memory Appliances -> NoSQL Databases -> Hadoop Clusters
In most enterprises, whether it’s a public or private enterprise, there is typically a mountain of data, structured and unstructured data, that contains potential insights about how to serve their customers better, how to engage with customers better and make the processes run more efficiently. Consider this:
- Online firms–including Facebook, Visa, Zynga–use Big Data technologies like Hadoop to analyze massive amounts of business transactions, machine generated and application data.
- Wall street investment banks, hedge funds, algorithmic and low latency traders are leveraging data appliances such as EMC Greenplum hardware with Hadoop software to do advanced analytics in a “massively scalable” architecture
- Retailers use HP Vertica or Cloudera analyze massive amounts of data simply, quickly and reliably, resulting in “just-in-time” business intelligence.
- New public and private “data cloud” software startups capable of handling petascale problems are emerging to create a new category – Cloudera, Hortonworks, Northscale, Splunk, Palantir, Factual, Datameer, Aster Data, TellApart.
Data is seen as a resource that can be extracted and refined and turned into something powerful. It takes a certain amount of computing power to analyze the data and pull out and use those insights. That where the new tools like Hadoop, NoSQL, In-memory analytics and other enablers come in.
What business problems are being targeted?
Why are some companies in retail, insurance, financial services and healthcare racing to position themselves in Big Data, in-memory data clouds while others don’t seem to care?
World-class companies are targeting a new set of business problems that were hard to solve before – Modeling true risk, customer churn analysis, flexible supply chains, loyalty pricing, recommendation engines, ad targeting, precision targeting, PoS transaction analysis, threat analysis, trade surveillance, search quality fine tuning, and mashups such as location + ad targeting.
To address these petascale problems an elastic/adaptive infrastructure for data warehousing and analytics capable of three things is converging:
- ability to analyze transactional, structured and unstructured data on a single platform
- low-latency in-memory or Solid State Devices (SSD) for super high volume web and real-time apps
- Scale out with low cost commodity hardware; distribute processing and workloads
As a result, a new BI and Analytics framework is emerging to support public and private cloud deployments.
The “Raw Data -> Aggregated Data -> Intelligence -> Insights -> Decisions” is a differentiating causal chain in business today. To service this “data->decision” chain a very large industry is emerging.
The Business Intelligence, Performance Management and Data Analytics is a large confusing software category with multiple sub-categories — mega-vendors (full stack, niche vendors, data discovery, visualization, data appliances, Open Source, Cloud – SaaS, Data Integration, Data Quality, Mobile BI, Services and Custom Analytics).
But the interest in BI and analytics is surging. Arnab Gupta, CEO of Opera states why analytics are taking center stage, “We live in a world where computers, not people, are in the driver’s seat. In banking, virtually 100% of the credit decisions are made by machines. In marketing, advanced algorithms determine messages, sales channels, and products for each consumer. Online, more and more volume is spurred by sophisticated recommender engines. At Amazon.com, 40% of business comes from its “other people like you bought…” program.” (Businessweek, September 29, 2009).
Here is a list of vendors who participate in this marketspace:
However, it took until 1980s when decision support systems (DSS) became popular and mid 1990s for BI started to emerge as an umbrella term to cover software-enabled innovations in performance management, planning, reporting, querying, analytics, online analytical processing, integration with operational systems, predictive analytics and related areas.
Gartner 2014 magic quadrant shows the key players in the BI market. The different players are differentiated based on five abilities— ability to handle large volumes of data, ability to deal with data velocity, variety (structured and unstructured), visualization capabilities and domain/vertical specific accelerators.
Analytics is becoming three different markets. First of all, there is the BI market which is actually going through quite a bit of change itself. This is a more consolidated market than we have seen in the past and there is a tremendous amount of work being done by Oracle, SAP, IBM and others to kind of retool it for the next generation of BI. So it is a growing market, lots of upgrade, replatform, modernization demand, lots of clients who are finally realizing that the tools (visualization etc.) are ready to give them some of the capability that they have historically cared about.
The second part of the market is what is called Advanced Analytics. Here you need PhD level data scientists who have backgrounds in machine learning, industry specific domain modeling, and different types of data science who can apply that in a very specific way to specific industry problems. This is a rapidly growing part of IT Services. Also, there are just not enough data scientists to go around.
The third part of the market is Analytics as a Service. This is about leveraging software-as-a-service platforms as opposed to on-premise. This is about a business model that is more like Business Process Outsourcing (BPO). Clients buy business outcomes; they don’t buy transactions and FTEs.
The analytics market has thousands of boutique consultants who are specialists in particular industries or specific technologies. It includes all the major technology providers, who are all trying to advance their business and capabilities that they are bringing to the market. And then there are vendors who are just bringing sheer capacity of data science skills to the market and they are coming in from a completely different angle of basically just renting the expertise of their data scientists into the market.
The market is incredibly fragmented. We are in the early stages of growth in the market. Every single one of our clients is building this capability internally and they are looking for more services from vendors, because the opportunity to apply analytics is in every single one function whether it is a customer analytics, industrial Internet, e-commerce platform, is growing. Analytics is embedded into literally every single business interaction.
BI, Analytics [and Big Data] Market Sizing
More recently to support a new generation of cost cutting and growth initiatives, corporations are investing heavily to gain near real-time actionable insights (historical and predictive), and from a mix of disparate spreadsheets and myriad of systems (legacy, internal silos, customer facing, suppliers, partners, etc.).