Self-tracking, Seamless Engagement and Personal Efficiency improvement’s new frontier is Personalized Big Data and Digital Health. This is really becoming a viable idea around wearable and sensor computing and the basis for new data platform wars.
The new platforms for digital life or data driven life — that collect, aggregate and disseminate — will cover a wide range of new User Experience (UX) use cases and end-points… medical devices, sensor-enable wristwear, headset/glasses, tech-sensitive clothing. All of them are going to collect a lot of data, low latency analytics, and enable data visualization. Several new firms are entering the activity tracker market LG (Life Band Touch), Sony (the Core), Garmin (Vivofit), Glassup, Pebble, JayBird Reign etc.
Data collection is just one piece of the solution. The foundation for personalized big data is Descriptive and Predictive Analytics. Ok…What do i next? what is the suggestion? in the form of predictive search (automated deduction or augmented reality).
How do i discover useful patterns, analyze, visualize, share, query and mobilize the collected data? A wide range of start-ups – Cue, reQall, Donna, Tempo AI, MindMeld, Evernote, Osito, and Dark Sky – and big companies like Apple, Google, Microsoft, LG and Samsung are working on predictive apps — aimed at enabling new robo-assistants that act as personal valets, anticipating what you need before you ask for it.
As a data engineer and scientist, I have been following the NSA PRISM raw intelligence mining program with great interest. The engineering complexity, breadth and scale is simply amazing compared to say credit card analytics (Fair Issac) or marketing analytics firms like Acxiom.
Some background… PRISM – “Planning Tool for Resource Integration, Synchronization, and Management” – is a top-secret data-mining “connect-the-dots” program aimed at terrorism detection and other pattern extraction authorized by federal judges working under the Foreign Intelligence Surveillance Act (FISA). PRISM allows the U.S. intelligence community to look for patterns across multiple gateways across a wide range of digital data sources.
PRISM is unstructured big data aggregation framework — audio and video chats, phone call records, photographs, e-mails, documents, financial transactions and transfers, internet searches, Facebook Posts, smartphone logs and connection logs – and relevant analytics that enable analysts to extract patterns. Save and analyze all of the digital breadcrumbs people don’t even know they are creating.
The whole NSA program raises an interesting debate about “Sed quis custodiet ipsos custodes.” (“But who will watch the watchers.”) Read more
At the Analytics Executive Forum, I facilitated a session on Omni-channel analytics. It struck me how every leading consumer facing firm seems convinced that mobile is becoming the dominant B2C interaction channel. Mobile is the gateway to insight based marketing and the “always addressable customer”….
Insight-based interactions – The company knows who you are, what you prefer, and communicates with relevant, timely messages, using the power of analytical intelligence to detect patterns, decode strands of information and create meaningful offers and value.
The “always addressable customer.” This is a consumer who fits the bill on three fronts simultaneously: (1)
- Owns and personally uses at least three connected devices; (2)
Goes online multiple times throughout the day; (3)
- Goes online from at least three different physical locations
The opposite of insight-based is “spray-and-pray” marketing – The company has very limited knowledge about who you are, forgets what you prefer, and tries to reach you with off-target communications that alienate you – based on fragmented data, poor data quality and inadequate integration, resulting in confusing, chaotic interactions. A good example: “I have 2 million frequent flyer miles with your airline and still do not get any recognition, respect or value from this loyalty.”
As companies architect new insight based mobile use cases I suggest that they look at what is coming next. With IOS 7, Apple is delivering several new features – Passbook, Beacon.
Retailers, banks and other customer facing firms/brands better pay attention. 100+ million iPhones are automatically getting this feature with the new OS upgrade making this a mega-disruptor in the coveted target segment everyone is chasing. Read more
Data growth curve: Terabytes -> Petabytes -> Exabytes -> Zettabytes -> Yottabytes -> Brontobytes -> Geopbytes. It is getting more interesting.
Analytical Infrastructure curve: Databases -> Datamarts -> Operational Data Stores (ODS) -> Enterprise Data Warehouses -> Data Appliances -> In-Memory Appliances -> NoSQL Databases -> Hadoop Clusters
In most enterprises, whether it’s a public or private enterprise, there is typically a mountain of data, structured and unstructured data, that contains potential insights about how to serve their customers better, how to engage with customers better and make the processes run more efficiently. Consider this:
- Online firms–including Facebook, Visa, Zynga–use Big Data technologies like Hadoop to analyze massive amounts of business transactions, machine generated and application data.
- Wall street investment banks, hedge funds, algorithmic and low latency traders are leveraging data appliances such as EMC Greenplum hardware with Hadoop software to do advanced analytics in a “massively scalable” architecture
- Retailers use HP Vertica or Cloudera analyze massive amounts of data simply, quickly and reliably, resulting in “just-in-time” business intelligence.
- New public and private “data cloud” software startups capable of handling petascale problems are emerging to create a new category – Cloudera, Hortonworks, Northscale, Splunk, Palantir, Factual, Datameer, Aster Data, TellApart.
Data is seen as a resource that can be extracted and refined and turned into something powerful. It takes a certain amount of computing power to analyze the data and pull out and use those insights. That where the new tools like Hadoop, NoSQL, In-memory analytics and other enablers come in.
What business problems are being targeted?
Why are some companies in retail, insurance, financial services and healthcare racing to position themselves in Big Data, in-memory data clouds while others don’t seem to care?
World-class companies are targeting a new set of business problems that were hard to solve before – Modeling true risk, customer churn analysis, flexible supply chains, loyalty pricing, recommendation engines, ad targeting, precision targeting, PoS transaction analysis, threat analysis, trade surveillance, search quality fine tuning, and mashups such as location + ad targeting.
To address these petascale problems an elastic/adaptive infrastructure for data warehousing and analytics capable of three things is converging:
- ability to analyze transactional, structured and unstructured data on a single platform
- low-latency in-memory or Solid State Devices (SSD) for super high volume web and real-time apps
- Scale out with low cost commodity hardware; distribute processing and workloads
As a result, a new BI and Analytics framework is emerging to support public and private cloud deployments.