Skip to content

August 3, 2011


Big Data, Analytics and KPIs in E-commerce and Retail Industry

by Ravi Kalakota

  • How to convert Lookers to Bookers…
  • How to create unique and effective Digital Experiences that impact probability of purchase or likelihood of return.
  • What offers might result in higher “take rates”

The change in consumer behavior and expectations that e-commerce, mobile and social media are causing is hugely significant – big data and predictive analytics will separate brand/retail winners from losers. This won’t happen overnight but the transformation is for real.

Retail Industry makes up a sizable part of the world economy (6-7%) and covers a large ecosystem –  E-commerce, Apparel, Department Stores, Discount Drugstores, Discount Retailers, Electronics, Home Improvement, Specialty Grocery, Specialty Retailers and Consumer Product Goods suppliers.

Retail is increasingly is looking like a barbell – a brand oriented cluster at the high-end, a very thin middle, and a price sensitive cluster at the low end. The consumerization of technology is putting more downward pricing pressure in an already competitive “middle” retail environment. The squeeze is coming from e-commerce and new “point, scan and analyze” technologies that give shoppers decision making tools — powerful pricing, promotion and product information, often in real-time. Applications in iPhones and Droid, like  Red Laser can scan barcodes and provide immediate price, product and cross-retailer comparisons. They can even point you to the nearest retailer who can give you free shipping (total cost of purchase optimization). This will lead to further margin erosion for retailers that compete based on price (a sizable chunk of the market in the U.S, Europe and Asia).

Data analytics is not new for retailers. Point of sale transactional data obtained from bar-codes first appeared in 1970s. A pack of Wrigley’s chewing gum was the first item scanned using Universal Product Code (UPC) in a Marsh Supermarket in Troy, Ohio in 1974.  Since then, retailers have been applying analytics to get even smarter and speedup the entire industry value chain.

Consumer Goods Value Chains

More recent use cases of retail analytics  include:

  • Corporate Express is using its market basket application to track the baskets of all of its online customers, and can identify which SKUs are most profitable or poor performers when purchased in combinations.
  • Liz Claiborne analyzes financial, sales, and inventory data across 30,000 points of sale worldwide — department stores, specialty stores, and catalogs/Internet — to uncover revenue optimization opportunities.
  • Limited Brands currently uses MicroStrategy for customer analysis, financial analysis, inventory management, sales analysis, and category management reporting across all of its brands (including Victoria’s Secret, Express, Bath & Body Works, and Henri Bendel).
  • Netflix is a heavy user of big data… The company issued a $1M analytics challenge in October 2006: come up with recommendation software that could do a better job predicting the movies customers would like than Netflix’s in-house software, Cinematch. To qualify for the prize, entries had to be at least 10 percent better than Cinematch. (for more detail see notes at end).
  • BestBuy had a clickstream analysis problem — 3.5 billion records, 71 million unique cookies, 1.7 million targeted ads required per day. How to make sense of this data? They hired Razorfish to implement a analytic solution on Amazon Web Services and Elastic MapReduce. Solution was a 100 node cluster on demand; processing time was reduced from 2+ days to 8 hours.

In a business with razor-thin margins, small, mid and large retailers are constantly looking for the performance improvement edge. Retailers are racing to become better at slicing and dicing reams of data they collect from multiple channels – online, stores, call center, catalogs and smartphones.  They are using business analytics to get insight into retail KPIs (outcome measures) or Metrics (operational activities that serve as leading or lagging indicators). The goal is to drive significant topline and bottomline improvements through improved merchandising decisions, more timely responses to information requests, cost reduction initiatives, enhanced employee productivity, and better service to customers.

Types of Big Data Analytics in Retail

We analyzed the typical retailers and found some interesting patterns of how raw data is being converted into signals and actions. Most of the Retail Analytics and Big Data retail tend to cluster around the following areas:

  • Customer Analytics and KPIs
  • Merchandizing KPIs — Eliminate Stockouts and Overstocks
  • Store Operations Analytics and KPIs
  • Vendor and SKU Management Scorecards and KPIs
  • Marketing Analytics and KPIs
  • Returns, Fraud and Loss Prevention Analytics and KPIs

It’s important to distinguish between retail KPIs and metrics.  We view KPIs as clusters of metrics that in aggregate provide a report card.  Metrics on the other hand are simply snapshots of an activity. Think of KPIs as overall health vs. metrics as taking temperature or measuring shortness of breadth.

KPIs are like latent variables statistics (as opposed to observable variables). KPIs are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables that are observed (directly measured).

Customer Analytics and KPIs

Objective:  Understand Your Most Valuable Customers. Target Them to Maximize Profits and Loyalty

Every retailer is focused on understanding and leveraging all customer touch points, from granular in-store transactional data to online store and web search information to mobile data, in order to most effectively target customers, improve repeat purchase, drive cross-sell. All to maximize profits and customer loyalty.

Providing state-of-the-art customer analytics, retail business analytics enables retailers to integrate and analyze the ever-expanding universe of data to obtain valuable insights about customer behavior, buying patterns, and consumer preferences.

Area   Type of Analytical Insights
Customer and Market Basket Analytics and KPIs –Understanding and leveraging all customer touch points
  • Behavior profiling
  • Preference modeling
  • Affinity tracking
  • Attachment Rates
  • Basket-Register Correlation
  • Brand Switching
  • Core Item Frequency
  • Core Items
  • Customer Loyalty
  • Demographic Baskets
  • In-Basket Price
  • Items per Basket
  • Revenue Contribution
  • Shopper Penetration


What Can You Do With Customer Analytics? 

The most important assets for every retailer are customers. The challenge is to look for ways optimize the value from each relationship.

  • Discover who your customers are. Identify their unique characteristics, and know what products they purchase and why.
  • Customer journey analysis, expectations and sentiment tracking.
  • Track the impact of promotions on customer baskets as they move through your stores and provide a holistic view of all of their behaviors.
  • Tap into their transactional data to connect the dots between customers, stores, products, and promotions.
  • Move beyond basic segmentation, personas, and list pulls to create targeted microsegments – highly-refined groups of customers based on user-defined characteristics.

Once you understand your customer at this level, you can align effective tactics to tailored customer segments to surround them with a holistic shopping experience. Enable business users in marketing and merchandising to seamlessly integrate insight with promotion planning, weekly flyers, planograms, store assortments, and more. Boost offer performance. Drive larger baskets.  End goal – increase sales, customer profitability, and customer loyalty.


Merchandizing KPIs

Objective:  Significantly reduce costs, eliminate the expense of stock-outs and overstocks, and make powerful, rapid decisions 

Retailers face immense pressure to balance inventory levels with in-stock positions while maintaining assortments that are relevant and fresh. Buyers and merchandise managers need to be constantly on-the-go either in team meetings, traveling overseas for a buying trip, or working hand-in-hand with vendors.

Whether at their desk or on-the-go, this job requires mobility and constant connectivity to respond to consumer demand, seasonal trends, and fashion styles, all to ensure that customers get the right products.

Merchandise Analytics and KPIs — Eliminate Stockouts and Overstocks
  • Department Contribution
  • Hot Item Report
  • In Stock %
  • Inventory Turns
  • Lineal Feet
  • Mark Down %
  • Pull-Through
  • Reallocations
  • Seasonal Buying
  • Sell-Through
  • Weeks of Supply

The Merchandise Analysis scenarios are designed to improve inventory decisions. Users can quickly accelerate shipments by evaluating top-selling products, make markdown decisions based on seasonal sell-through, cancel shipments for bottom-selling products, and communicate more effectively with vendors.

Store Operations Analytics and KPIs 

Objective:  Keep Store Managers on the Selling Floor, Not Behind a Desk. Give Store Operations the Right Information at the Right Time to Make the Right Decisions.

Store managers are more effective when they are out on the selling floor, not in a back office. The old paradigm of paper reporting or even web/desktop is now becoming obsolete. Mobility empowers field managers in stores to make near real-time decisions regarding inventory, promotions, pricing, staffing, and store design.

Store managers can now run daily analyses of store performance where they can aggregate by Store sales, SKU sales and sales per employee.  In some advanced retailers, the data is refreshed in real-time so they can deal with just-in-time replenishment more cleverly.

Mobile store operations reporting keeps the store managers where they should be: in the store, on the selling floor. Store managers can see, at regular intervals throughout the day, how their sales are performing compared to last year, giving them the information and tools to improve performance.

Area   Type of Analytical Insights
Store Operations Analytics and KPIs
  • Asset Turnover
  • Competitor Stores
  • Comp Store Sales
  • Front Store Sales
  • Inventory Turnover
  • Labor Cost Analysis
  • Reallocation / DSD
  • Register Usage Analysis
  • ROA
  • Sales & Margin
  • Sales / Square Foot

Store traffic information delivers detail on a store’s performance: Units per Transaction, Average Transactions (Units and $), Conversion Rates, and Sales per Payroll Hour are all provided in the mobile BI app.

Relevant, filtered information delivered when and where it is needed helps store personnel identify anomalies in their operations and drive immediate action in the areas of labor scheduling, customer service, merchandise display, promotional signage, potential stock shortfalls caused by local market conditions, and more. 

Vendor and SKU Management Scorecards and KPIs 

Objective:  Analyze Vendor Performance, Drive Improvement, and Strengthen Negotiations. Improve Performance Across the Supply Chain

Managing vendors in today’s environment requires you to know what is happening in the supply chain real-time, enabling you to make rapid decisions to optimize performance and to strengthen your negotiations on price and quality.

Area   Type of Analytical Insights
Category Management KPIs
  • Assortment Optimization
  • Category Contribution
  • Category Scorecards
  • Item Definitions
  • Market Comparison
  • Pricing Modules
  • Role Analysis
  • Season Changeover
  • Share of Shelf
  • SKU Rationalization
  • Vendor Performance

A comprehensive Vendor Scorecard provides the right information to take the right action at the right time. A scorecard helps jointly improve performance by providing a common view of the following:

  • How are my products / categories performing?
  • Where do I have issues with product sales, returns, or excess inventory?
  • How am I performing operationally in delivering product?
  • What is the true profit contribution of each product?
  • What are the results of my promotion efforts?
  • How can I be more effective with the use of promotional funds?
  • How are my products forecasted to sell? How many products to produce?
  • What is the profit contribution of private label products?

A Vendor Portal provides a platform for collaborating with vendors, enabling timely, valuable business insight, driving efficiencies in cost containment and improving sales performance.

Marketing Analytics and KPIs

Marketing is going through tremendous changes. As more customers move online and social becomes a big factor in how people are influenced to make a purchase, new techniques are emerging around cross-selling, location-based marketing, in-store behavior analysis, customer micro-segmentation, sentiment analysis and enhanced multichannel consumer experience.

Retailers are using big data to integrate marketing promotions, location data and differential pricing. Promotions via large-scale e-mail targeting (many retailers have 25 to 50 million household lists) is still a big challenge for many retailers. Open rates on targeted e-mails with custom catalogs with tailored personalized storefronts are 10 times more effective than those that are not targeted.

Another interesting emerging area of retail big data is sentiment analytics. Combination analytics from integrating profiles, peer sentiment, recommendations and comments on Twitter, Yelp and Facebook are playing a bigger role in influence and purchase decisions. As a result, there is a race among market departments to leverage the voluminous streams of data in social media to drive effective promotions or messages at various touch-points. Social discussion of brands, products and services, and endorsement of them, is very influential. Consumers have always been influenced by the opinions of others, and ignoring a channel that can influence so many consumers so quickly would be unwise.

Competition for the 16-35 age group customers is intense. Emerging social marketing services like Foursquare are changing the dynamics of real-time customer interaction in that age group. Foursquare is a web and mobile application that allows registered users to connect with friends and publish their location.  Users “check-in” at venues using a mobile website, text messaging or a device-specific application by clicking from a list of venues. Each check-in awards the user points and sometimes “badges”.  As of June 2011, the company reported it had 10 million registered users.  Foursquare is trying to become a sticky portal by moving from social connection to promotions/coupons to even payments.

A list of classic marketing KPIs is provided below.

Area   Type of Analytical Insights
Marketing Analysis KPIs
  • Attrition and Fading
  • Ad Blocks
  • Channel Share
  • Coupon Distribution
  • End-Cap Efficiency
  • Feature vs. Display
  • Marketing Areas
  • Market Share
  • Price Points
  • Promo Lift
  • Seasonal Events
  • Site visitors, Click-thrus

Returns, Fraud and Loss Prevention Analytics and KPIs

Ease of returns has always correlated well to customer satisfaction. Thanks to Amazon Prime or  Zappos’s  365 day return policy the dynamics of the industry are shifting.  Customers love this policy and are eager to migrate to vendors that provide flexibility.  But the challenge is predicting return trends by product and sku;  and keeping necessary reserves.

In a low margin industry, loss prevention is often a big focus.  Retail theft jumped by $3.6 billion to $37.1 billion in 2010, according to National Retail Federation. More goods (44 percent) were swiped by employees, often working with organized groups of thieves, than shoplifters, which accounted for $12 billion, or 33 percent of lost goods. Paperwork errors or vendor fraud accounted for $6.8 billion, or 18 percent.

A list of classic Loss Prevention KPIs is provided below.

Area   Type of Analytical Insights
Loss Prevention KPIs
  • Backorders
  • Cancels
  • Distribution Shrink
  • Hot Customer
  • Hot Staff
  • Hot Stores
  • Markdowns
  • Natural Losses
  • Return Rates
  • Shrink Analysis
  • Vendor Rationalization


The use of IT and digital data in the retail industry has been instrumental in boosting the profitability and productivity of players like Walmart, Tesco, for years.  In the coming years, the increased adoption of big data has the potential to increase sector-wide productivity.

In an industry with notoriously tight margins (typically 2-6% range), retailers are facing a set of challenges that require a new set of solutions to address them.  Margin declines can no longer be offset by improvements in selling, general, and administrative (SG&A) costs and reduced inventories.

Retailers will need to selectively employ a set of levers that they have utilized in the past to manage costs in the short-term, and a series of more-advanced levers to build a sustainable competitive advantage.

Notes (and Interesting Factoids)

1. uses collaborative filtering to generate “you might also want” prompts for each product bought or visited.

2. Neiman Markus uses behavioral segmentation, economic modeling, a multi-tier membership reward program to understand buying patterns.

3. Geographic retailing is gaining a foothold with better tools. Placecast, a provider of location-based marketing, claims that 50% of users have made purchases as a result of location-based ads (targeted ads combined with preference, profile and location)

4. Williams-Sonoma, all things kitchen and cooking, has a database of 60M households tracking variables like income, number of children, housing values, etc. They leverage these variables in e-mail targeting programs.

5. The Netflix movie recommendation contest (blending of different statistical and machine-learning techniques) has been widely followed because its crowdsourcing lessons could extend beyond improving movie picks. The researchers from around the world were grappling with a huge data set — 100+ million movie ratings — and the challenges of large-scale predictive modeling.

6. Interestingly several people think that “what your friends thought” feature to be extremely accurate in predicting and suggesting movies…more than the recommendation feature.

7. Netflix announced a second recommendation contest. Contestants were asked to model individuals’ “taste profiles,” leveraging demographic and behavioral data. The data set — 100 million entries will include information about renters’ ages, gender, ZIP codes, genre ratings and previously chosen movies. Unlike the first challenge, the contest will have no specific accuracy target.  $500,000 will be awarded to the team in the lead after six months, and $500,000 to the leader after 18 months. This contest was cancelled in May 2010 after a legal challenge that it breached customer privacy with the first contest.

8. Typical problems retailers face:

  • Low repeat purchase
  • Fading and attrition
  • No cross-selling or low expansion into new categories
  • Pricing inefficiencies
  • Post-promotion inventory gluts
9.   Healthcare Analytics Challenge — California physicians group Heritage Provider Network Inc. is offering $3 million to any person or firm who develops the best model to predict how many days a patient is likely to spend in the hospital in a year’s time. Contestants will receive “anonymized” insurance-claims data to create their models. The goal is to reduce the number of hospital visits, by identifying patients who could benefit from services such as home nurse visits.

10. As one our comments noted…. It’s important to distinguish between retail KPIs and metrics.  We view KPIs as clusters of metrics that in aggregate provide a report card.  Metrics on the other hand are simply snapshots of an activity. Think of KPIs as overall health vs. metrics as taking temperature or measuring shortness of breadth. KPIs are like latent variables statistics (as opposed to observable variables). KPIs are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables that are observed (directly measured).

6 Comments Post a comment
  1. Shawn Noonan
    Aug 3 2011

    another GREAT blog Ravi


  2. Aug 3 2011

    Excellent resource, very detailed. But I wonder if you perhaps will overwhelm most readers. I think youve catalogue metrics, not Key Performance Indicators. Most of these track activities, not outcomes, so it’s very difficult to determine performance. I know this will seem like a semantic argument to some but i think the distinction is critical to true performance management success. Here’s an article I wrote 5 years ago about it and there are lots of examples on


  3. Ashutosh Sinha
    Feb 6 2013

    Good detailed read however, one of the significant department i.e. IT’s ( Information Technology) KPI is missing. If the same be included into it, it would be more insightful for the reader (Top Management).



Trackbacks & Pingbacks

  1. Retail section: Types of Data Analytics in Retail | Hansa-Cequity's Knowledge Centre
  2. Big Data, Analytics and KPIs in E-commerce and Retail Industry - RetailFit :: Vietnam Mobile Marketing & Mobile Advertising for Retail Industry with Big Data
  3. Electronic Commerce Ravi Kalakota Pdf | Technology Nowadays

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Note: HTML is allowed. Your email address will never be published.

Subscribe to comments

%d bloggers like this: