Skip to content

Recent Articles


Machine Learning and AI @ FaceBook

FacebookAIMachine Learning (ML) and AI are at the bleeding edge of data science, deep learning and predictive search today.

Everyone is jumping on this AI powered data-driven engagement (“ambient experience and convenience”) trend.

Salesforce CEO Marc Benioff said at a recent conference: This is a huge shift going forward, which is that everybody wants systems that are smarter, everybody wants systems that are more predictive, everybody wants everything scored, everybody wants to understand what’s the next best offer, next best opportunity, how to make things a little bit more efficient.”

Facebook is a case study of where AI/ML are being used to transform user engagement and experiences. I am starting to see many leading firms investing in ML Accelerators and Platforms as part of their data science strategy.

Read more »


Chief Data Officer Role & Challenges

Chief Data Officers, Chief Analytics Officers,  Chief Data Science Officers and Chief Digital Officers are showing up everywhere. The job is to leverage the latest in predictive analytics, data science, machine learning, and multi-tenant cloud architecture to bring innovation to traditional processes.

This is a pivotal moment in data driven business models but there is no getting around the inherent difficulties associated with either altering organizational behavior, data ownership politics or managing wholescale transformation of the data infrastructure. And while the challenges are real, many firms are getting closer to achieving a data science and data management environment.

What are they data and analytics officers overseeing… A variety of foundational strategies:

  • Data-as-a-Service:  Data Provisioning, Management, Lineage, Quality
  • Reporting-as-a-Service:  Dashboards, KPIs, Drilldowns/Aggregates…. Descriptive
  • Analytics-as-a-Service:  Predictive  Modeling and BI… Prescriptive analytics
  • Information-as-a-Service:  Threshold based Alerts, Exceptions, Mobile Prompts
  • Insights-as-a-Service:   ML/AI based…automated learning – ambient intelligence, Next best Offer/Action

At core of all these, Data Management and Data Science tools are core technical and business capabilities.  Some firms are more mature and further along than others.

Why Mature Data Management as a Function

Organizations live or die by the quality of their data.

Data is an underlying factor of input into business operations and essential in order to  facilitate process automation, digitize operations, support financial engineering and enhance customer facing analytical capabilities.

An effective data management program requires a planned strategic effort

  • Integrate multi-discipline efforts
  • Inculcate a shared vision and understanding
  • Data is a ‘thing’ – vital infrastructure element foundation of the n-tier architecture
  • Not a project, more than a program…it’s part of the core foundation

There is no question about it – the foundational levels of people, process, governance and technology required to establish data management on a  sustainable basis are coming together under the CDO umbrella.

What does a Chief Data Officer (CDO) do?

Read more »


Robotic Process Automation + Analytics

“Looking to the future, the next big step will be for the very concept of the “device” to fade away. Over time, the computer itself—whatever its form factor—will be an intelligent assistant helping you through your day. We will move from mobile first to an AI first world.” 

— Sundar Pichai, CEO Google


  • A global oil and gas company has trained software robots to help provide a prompt and more efficient way of answering invoicing queries from its suppliers.
  • A large US-based media services organization taught software robots how to support first line agents in order to raise the bar for customer service.

Software agents or Robotic process automation (RPA) is becoming a mainstream topic at leading corporations. I have seen a massive uptick in corporate strategy work in this area as C-Suite execs look at new ways to do more with less.

Software robots ∼ Apple Siri, Microsoft Cortana, IBM Watson, Google DeepMind, drones and driverless cars ∼ are now mainstream. What most people are not aware of is the rapidly advancing area of enterprise robots to create a “virtual FTE  workforce” and transform business processes by enabling automation of manual, rules based, back office administrative processes.

This emerging process re-engineering area is called Robotic process automation (RPA).

Machine Learning (ML) and graph processing are becoming foundations for the next wave of advanced analytics use cases. Speech recognition, image processing, language translation have gone from a demo tech to everyday use in part because of machine learning. Machine learning models, e.g., in driverless cars,  teaches itself how to discover relevant things like a stop sign with snow partially obscuring the sign.

The market opportunity of artificial intelligence has been expanding rapidly, with analyst firm IDC predicting that the worldwide content analytics, discovery and cognitive systems software market will grow from US$4.5 billion in 2014 to US$9.2 billion in 2019, with others citing these systems as catalyst to have a US$5 trillion – US$7 trillion potential economic impact by 2025.

RPA – What?

According to Blue Prism, “Robotic automation refers to a style of automation where a machine, or computer, mimics a human’s action in completing rules based tasks.”

RPA is essentially the novel application of analytics, machine learning and rules based software to capture and interpret existing data input streams for processing a transaction, manipulating data, triggering responses and communicating with other enterprise applications (ERP, HRMS, SCM, SFA, CRM etc.).

RPA is not a question of “if” anymore but a question of “when.”  This is truly the next frontier of business process automation, enterprise cognitive computing, predictive analytics and machine learning. To make a prediction, you need an equation and parameters that might be involved.

Industrial robots are remaking factory and warehouse automation by creating higher production rates and improved quality.  RPA, simple robots and complex learning robots, are revolutionizing the way we think about and administer business processes (e.g. customer service), workflow processes (e.g., order to cash), IT support processes (e.g., auditing and monitoring), and back-office work (e.g., data entry).

I strongly believe that as cognitive computing slowly but surely takes off, RPA is going to impact process outsourcers (e.g., call center agents) and labor intensive white collar jobs (e.g., compliance monitoring) in a big way over the next decade. Any company that uses labor on a large scale for general knowledge process work, where workers are performing high-volume, highly transactional process functions, will save money and time with robotic process automation software.

RPA picture

Business Impact of RPA – Where?

Read more »


Omni-channel Retail Paradox: the Curse of Digital

Everyone knows that the retail industry is being transformed by digital, analytics and big data. Winning requires continual data-driven experimentation and transformation.  

Shortened time from idea-to-app is a constant challenge.

Evidence of this “digital disruption” by category are mounting every day. Wal-Mart closes 269 stores as it retools portfolio to compete with online natives like Macy’s said that it will shutter over 36 stores as store traffic declines faster than expected, and Finish Line said that it would close 150 stores by 2020.  Gap,  J.Crew,  American Apparel, Sears and Kmart are all facing similar headwinds.

Starbucks CEO Howard Schultz laid out his thoughts on the future prospects for retail business, “three years ago we began to envision that there would be a seismic change in consumer behavior, and that seismic change was due in large part to e-commerce, omni-channel and smartphone shopping.”

It’s fascinating to watch retailers trying to shift tech/platform strategies to deal with digital disintermediation, showrooming, asset-light models, physical-to-digital channel integration, mobile shoppers, same-day delivery/fulfillment, programmatic targeting, online native models and now the new buzz.. virtual and augmented reality.

Several retailers have invested in Big Data and Hadoop platforms to mine massive volumes of structured transactional,   operational data and unstructured data—web logs, clickstream data, geo-location data, social interactions and sensor data.

While most retailers understand the mega-shift and seems to know what to do….they are unable to execute consistently or effectively. A talent gap in many cases. A platform gap in others. Others are hindered by legacy IT systems or don’t have the necessary technology capabilities in place.

I think the digital induced pain is going to get worse in 2016 and 2017.  Consumers will continue to diversify their retail activity across channels in search of the best value, forcing retailers to spread out their digital investments.  This puts additional stress on execution and leadership.

Read more »


Analytics and ML Use Case – Robo-Advisors in Wealth Management

New data driven FinTech business models built on Hadoop, Spark and Machine Learning are rapidly emerging and disrupting wealth management.  Here is my recent posting from about one such use case… Robo-Advisors.

We are in the early stages of a generational shift in wealth management, especially “plain vanilla” investing for the mass affluent and millennial segment.  Until recently, you had only two options when investing:

  • Do-it-yourself (DIY)
  • Hire a registered investment advisor (RIA)

Now there is a third option.  Robo-advisors are new class of personal financial advisors that provides online, algorithm based portfolio management with minimal human intervention. Robo-Advisors going after the low-end of brokerage/RIA business with automated asset allocation using Modern Portfolio Theory.

The Robo-Advisors market leaders who are serving the mass affluent include are:

  • Wealthfront (with over USD 2.6bn in assets under management (AuM) and 20,000 investors);
  • Betterment (with over USD 1.4bn in AuM and 70,000 investors); and
  •  FutureAdvisor (With over $600 million in AUM).

The timing for this market shift coincides with three trends: consumerization, digital tools, and disillusionment with status-quo investment advisors.  The gyrating stock market driven by program trading is increasingly bringing Robo-Advisors, algorithmic portfolio management to the forefront.  Investors are getting disillusioned with traditional investment advisors who simply track the market indices (SPY, QQQ or Russell 2000) by purchasing ETFs at best.

Many banks and brokerage firms over the years have shifted their focus to serve ultra high net worth (UHNW) and high net worth (HNW) investors, leaving an opportunity for firms to target the “mass affluent” investors, or those with less than $1 million in investable assets. Younger investors are increasingly interested in online digital advice (trial-and-error bets), as opposed to hiring an adviser.

wealthfront3 Read more »


Big Data is Entering the Trough of Disillusionment

hype cycleBig data, data lakes and becoming data-driven is no longer news or novel. Hype is not enough anymore.

The challenge today for leaders in every enterprises is (a) how to monetize data? (b) how to create enterprise class platforms instead of sandboxes? Basically, how can data, analytics and insight drive digital operations and digital transformation?

The paradox is interesting.  While leadership is struggling with value creation, the long term data trends are favorable: (1) data continues to grow exponentially and outstrip our ability to convert into insight;  (2) data consumption is evolving… Consumers and employees have more interactions with data through mobile apps than they do through desktop browsers; (3) Analytics – predictive and prescriptive – is gaining traction in several industries and business processes.

In the first wave of excitement around big data, there is a massive amounts of investment in stand-alone pilots and sandboxes.  Some experiments worked, but many failed to deliver.  The linkage between analytical projects and the everyday business applications (systems of record, systems of engagement, systems of insight) have mostly been missing.

In the next wave, we are seeing a tighter alignment or foundational underpinning between analytics (even machine learning) and traditional business applications. Take for instance Salesforce. Machine learning has become increasingly important to Salesforce, which has acquired PredictionIORelateIQ and Tempo AI, among other companies.

This implies a massive transformation wave (and upgrade cycle) across existing:

  • Systems of insight  (Reporting, BI, Analytics platforms)
  • Systems of engagement (CRM, SFA)
  • Systems of record (ERP)

The bigger transformation challenge is around how to systematically clear the bottlenecks in each of the above so that end-users can (1) access real-time data;  (2)  slice and dice the data for actionable insights from any device, anywhere; (3) convert the insights into guided decisions.

The directional strategy is clear, but can leadership get behind it and implement it swiftly and effectively?

This impending transformation is both exciting and daunting at the same time.  Application development and delivery (AD&D) teams are overwhelmed.  Leadership (in most corporations) in terms of vision or directional clarity is often weak or missing.  Strained relationships and misaligned business and technology teams is unfortunately the norm, not the exception.

Something is out of whack.

What I have observed in multiple engagements and research is that every year, billions of dollars are being spent with the consulting industry on establishing a corporate data strategy. Millions of hours of leadership time is invested in the strategy effort.

Even more billions are being spent on the core foundational “data lake” strategies – using Hadoop to create a large scale data dump. The more advanced form of this is to add Master Data Management (MDM) on top to create a “single golden view” of customers, employees, products or accounts. The hope (and prayer) is that this will enable companies to derive a more accurate picture of their business.  This is the hypothesis behind many “data lake” initiatives. The results and business value have been sketchy so far. So, is the strategy wrong or execution flawed?

Executing the data/analytics strategy around systems of engagement, record and insight is where the pain and the largest costs to the organization come into play. This is where discipline and talent becomes critical, and where competitive advantages are either won or lost.  But most firms starve the application development and delivery (AD&D) teams — limited budgets, basic talent to do cutting edge solutions,  unrealistic deadlines, constantly changing requirements. Few invest in product or program management to ensure the entire organization is involved, aligned, and ready to make it happen?

As a result, most corporations today have an awfully poor batting average when it comes to analytics projects or data informed business processes. This also may lead to the frequency of failure of most large-scale data-driven corporate change initiatives.  

The hype cycle is not a new phenomenon, but one that repeats itself with each innovation that somehow captures people’s imagination.  We have to be patient in the next 2-3 years and dig ourselves out of the big data Trough of Disillusionment🙂

I am bullish on data long term.  The future is becoming more data-driven everything. Every industry (financial services, healthcare, retail, industrial) is going digital powered by data.


Source: It’s Time To Upgrade Business Intelligence To Systems Of Insight Supercharge BI With Agility, Big Data, And Insights To Drive Action by Boris Evelson, July 20, 2015



Big Data Evolving Landscape – 2016

2015 was a year of significant change in the “Art of the Possible” with Analytics.   Norms about Analytics are evolving, and as they do, leading to wholesale business model transformation and cultural change at some workplaces. This change is driven not only by fast-moving technology, but also by new techniques to get value from data.

2016 is turning into the year of “democratization of data”, AI and Deep Learning. The state of the art is around Deep learning (also known as structured learning, hierarchical learning or machine learning), which are essentially efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

Customer intelligence is the hot space. Commoditization pressures and shifting consumer expectations have inspired many companies to creatively use data and analytics to enrich their core products and services, a phenomenon called “wrapping”. Companies wrap offerings with information to differentiate them and to add value to customers. The best companies build distinctive competencies for wrapping.

However, there’s a lot of technology in the world, my friend.  The 2016 edition of Matt Turck & Jim Hao Big Data landscape supergraphic is shown below.



Additional References:

A post by Matt Turck, a partner at FirstMark Capital, with a previous version 2014 –   Big Data Landscape v. 3.0.

Click here for a High-Res version of original


Customer Journey Analytics and Data Science

  • DigitalJourneys-WhyWhere do customers abandon the shopping process? Is it the same in every geography?
  • Audience of One…. Who are your fans versus haters in the marketplace?
  • How do customers feel about your products? How engaged are customers with your brand versus your competitors’ brands across social media and web channels?

Fortune 500 companies are making large investments around Programmatic Marketing, Sales and Service (“marketing that learns”). Programmatic marketing is the application of automated technology through which media buyers and sellers may align organizational processes in support of ongoing, channel-agnostic customer engagement (and to allow for the continuous optimization of that effort as business strategies evolve) in order to drive revenue.

One of most often implemented use case in Programmatic Marketing is customer journey mapping and analytics.

Why? Because, deciphering the nuts-and-bolts” of individual customer journeys online (and deducing intent) is core to improving customer experience and driving brand loyalty.

Specifically, the objectives are:

  • Visualize and map the end-to-end customer journey by personas
  • Optimizing on the right journey attributes to increase yields  by >30% lift… Uncover the right combination of web, mobile and physical channels, content and experiences that  best achieves the target goals
  • Enable marketers to identify journey bottlenecks for individuals and aggregates
  • Leverage actual behavior data to enhance and personalize the experience for each individual customer


Read more »


Consumerization of BI: Data Visualization Competency Center

heatmap-pgWhat do users want? Self-service, interactive analytics with all kinds of datasets with instant response times, no waiting.

Today there is a strong move towards “Consumerization of BI” as business users are demand the same speed, and ease of use from their enterprise applications as their at-home software.

Consumerization of BI (“data at your fingertips”) means:

  • Help employees see and understand data
  • Helping employees gain insight into their data using simple drag-and-drop operations to solve business problems
  • Ability to quickly change filters and query conditions and conduct top down analysis via drill down
  • Make analytics fast, easy, interactive, and most importantly – useful

In every major corporation there is a renewed push to industrialize and improve data visualization and reporting capabilities.

The challenge is not in procuring the next greatest tool or platform but how to organize the people, process and assets effectively to create value, reduce training and support costs.  In other words, how to facilitate and create a flexible operating model for data mining and visualization delivery that provides discipline at the core while giving the business the agility that they need to make decisions or meet client needs?

Decision making is a core business activity that requires facts and insights. Slow, rigid systems are no longer useful enough for sales, marketing and other business users or even IT teams that support them. Competitive pressures and new sources of data are creating new requirements. Users are demanding the ability to answer their questions quickly and easily.

So the new target state is to empower business users along the Discover, Decide and Do lifecycle:

  • Discover new insights by rapidly accessing and interrogating data in ways that fit how people naturally think and ask questions.
  • Decide on best actions by publishing dashboards, collaborating with others, discussing insights and persuading others through data presented in an interactive application (“app”) rather than in a static view.
  • Do what is best at each decision point with confidence, based on the consensus that develops when new data is aggregated and explored with multiple associations and different points of view. Teams can take action more rapidly and move projects forward more effectively when everyone understands the data underlying decisions.

The challenge for business users is data discovery and ease-of-use. They want to focus on business questions that require aggregation and visualization. They want the interactive ability to quickly change filters and query conditions.

The challenge for infrastructure and application teams in every corporation is to deliver new easy-to-use platforms to their business partners quickly and consistently while maintaining governance and control.

To meet both sets of requirements, best practice firms are creating Data Mining and Visualization Competency Center or Centers of Excellence (DV-CoE) to ensure that the people, process and technology investments are not duplicated and addressed in a way that maximizes ROI and enhances IT-Business partnership. I have seen many cases where not having a proper structure leads to redundant projects and sub-optimal results. Read more »


Data/BI/Analytics Evolution @ NetFlix

More data + Better models + More accurate metrics + Better approaches & architectures = Lots of room for improvement!

netflixIt’s amazing to watch how quickly the data engineering / analytics/ reporting/ modeling/ visualization toolset is evolving in the BI ecosytem.

There are clearly massive foundational shifts taking place around big data. I am not sure how large conventional Fortune 500 firms can innovate and keep up with what’s going on.  I have run into CIOs who have not heard of Hadoop in some cases.

It’s also fascinating to see how data-driven “bleeding” edge firms like NetFlix are pushing the envelope.  Netflix stats are amazing:  1/3+ Internet traffic (NA / peak);  100+ Million hours per day; 65+ Million members / 50+ countries; 500 Billion Events / Day.

NetFlix is clearly reinventing Television and targeting 90 million potential subs in the US market alone.  Binge-watching, cord-cutting are now part of our everyday lingo. What most people don’t realize is how data-driven Netflix is…. from “giving viewers what they want” to “leveraging data mining to boost subscriber base”.

Viewing -> Improved Personalization -> Better Experience is the virtuous circle.

Here is a glimpse at how their BI landscape has evolved in the past five years as they integrate 5 million to 6 million net adds for several years now.  The figures are from a presentation by Blake Irvine, Manager Data Science and Engineering.

BI tools @ NetFlix pre-Hadoop

Read more »

%d bloggers like this: