Skip to content

Posts from the ‘Big Data’ Category

11
Jun

NSA PRISM – The Mother of all Big Data Projects


Prism9As a data engineer and scientist, I have been following the NSA PRISM raw intelligence mining program with great interest.  The engineering complexity, breadth and scale is simply amazing compared to say credit card analytics (Fair Issac) or marketing analytics firms like Acxiom.

Some background… PRISM – “Planning Tool for Resource Integration, Synchronization, and Management” – is a top-secret data-mining “connect-the-dots” program aimed at terrorism detection and other pattern extraction authorized by federal judges working under the Foreign Intelligence Surveillance Act (FISA).  PRISM allows the U.S. intelligence community to look for patterns across multiple gateways across a wide range of digital data sources.

PRISM is unstructured big data aggregation framework — audio and video chats, phone call records, photographs, e-mails, documents,  financial transactions and transfers, internet searches, Facebook Posts, smartphone logs and connection logs – and relevant analytics that enable analysts to extract patterns. Save and analyze all of the digital breadcrumbs people don’t even know they are creating.

The whole NSA program raises an interesting debate about “Sed quis custodiet ipsos custodes.” (“But who will watch the watchers.”) Read more »

1
May

Big Data needs a good storyteller….like Gary Vaynerchuck


Gary Vaynerchuck

In an episode of Mad Men, Don Draper talks about pitching the Kodak Carousel. “Technology is a glittering lure, but there is the rare occasion when the public can be engaged on a level beyond flash, if they have a sentimental bond with the product….Nostalgia. It’s delicate but potent. Switch it on.”

Combine the storytelling prowess of Don Draper with the high-pitched vitriol of Jim Cramer and add a dose of emotional intelligence to get Gary Vaynerchuck,  social media guru, best-selling author, wine librarian and marketer par excellence of the internet age. Gary Vaynerchuck rose to prominence in social media a few years ago with his video log, wine library tv which he used to grow his family wine store into a mulit-million dollar business. He currently runs VaynerMedia, a social media strategy and production company.

Gary is an avid supporter of the use of quantitative analytics in marketing. Carpe Datum Rx caught up with Gary to ask him a few questions about big data, marketing and technology adoption in the enterprise. Here are his paraphrased comments.

Is Big Data ready for the 99 per cent ?

Read more »

17
Jan

RegTech – Regulatory/Risk Data Management, AML, and KYC Analytics


Financial Services value chainEveryone is abundantly aware of the changing risk and regulatory landscape within the financial services industry.

Over the past seven years, we’ve seen a massive regulatory overhaul and an industry-wide push to enhance trust and confidence and encourage investor participation in the financial system.

To roadmap Wall Street regtech priorities, we have been having ongoing meetings with MDs and leading architects in global banks and investment services firms. RegTech (e.g., regulation as a service) is a subset of FinTech. Companies include

  • Fintellix offers a data analytics platform allowing banks to convert internal data into regulatory reporting formats
  • Suade offers banks “regulation as a service” interpreting real time regulatory knowledge so that banks can better manage and respond to regulation
  • Sybenetix combines machine learning with behavioral science to create a compliance and performance tool for traders

No longer business as usual.  It is clear that banks are devoting more resources to Know Your Customers (KYC),  Anti-Money Laundering (AML), fraud detection and prevention, Office of Foreign Assets Control (OFAC) compliance. FINRA is at the beginning stages of the process for building the Consolidated Audit Trail, or CAT for trading surveillance.

To enable compliance with variety of Risk/Regulatory initiatives, AML and KYC initiatives…the big RegTech related investments are:

  1. Strengthening the Golden Sources – Security Master, Account Master and Customer Master.
  2. Standardized, common global business processes, data, systems and quantitative solutions that can be leveraged and executed across geographies, products, and markets to manage delinquency exposures, and efficiently meet Regulatory requirements for Comprehensive Capital Analysis and Review  (CCAR), FDIC Reporting, Basel, and Stress Loss Testing.
  3. Various enterprise data management initiatives – Data Quality, Data Lineage, Data Lifecycle Management, Data Maturity and Enterprise Architecture procedures.

Regulatory reporting improvements via next generation Enterprise Datawarehouses (EDW) (using Oracle, IBM, NoSQL or Hadoop)– Reporting on top of EDW addresses the core problems faced by Finance, Risk and Compliance when these functions extract their own feeds of data from the product systems through which the business is conducted and use differing platforms of associated reference data in support of their reporting processes.

Lot of current investments are in the areas of Finance EDW which delivers common pool of contracts, positions and balances, organized on an enterprise wide basis and completed by anointed “gold” sources of reference data which ensure consistency and integration of information.

Crawl, walk, Run seems to be the execution game-plan as the data complexity is pretty horrendous. Take for instance, Citi alone….has approximately 200 million accounts and business in 160+ countries and jurisdictions. All risk management is made incredibly complex by the numerous banking mergers that took place over the past 3-4 decades.

Banking Mergers

Banking Mergers

 

The type of data challenges global banks like Citigroup, Goldman, Wells Fargo, Bank of America and JP MorganChase are wrestling with include: Read more »

15
Jan

Big Data Fatigue and Company Shakeout?


hype cycleBig Data is the latest “next big thing” transforming all areas of business, but amid the hype, there remains confusion about what it all means and how to create business value.

Usually when there is so much hype…there is an inevitable boom-bust-boom cycle. Hence my question:  Is the Big Data shakeout inevitable?

Are we in a big data tech bubble? If you are an enterprise customer, how do you prepare for this? What strategies do you adopt to take advantage of the situation? Can you move from lab experiments to production deployments with confidence?

The sheer number of companies that are chasing “the pot of big data gold” is astounding (see below).  While the innovation has accelerated the ability of the typical Fortune 1000 enterprise to absorb and assimilate has not. They tend to be 5-10 years behind the curve. As a result, many big data startups are either running out of cash or they are being folded by VCs into other firms.  This boom-bust cycle is a typical pattern in innovation.

http://www.bigdata-startups.com/open-source-tools/

BigDataUniverse

Source: Big Data Universe v3.. Matt Turck, Sutian Dong & FirstMark Capital

The Case of Drawn to Scale

Drawn to Scale, the four year-old startup behind Spire, shut down recently. Co-founder and CEO Bradford Stephens announced the news in a blog post. Drawn to Scale raised .93M in seed funding.

Spire is a real-time database solution for HBase that lets data scientists query Hadoop clusters using SQL. According to Stephens, the system has been by deployed by American Express, Orange Flurry, and four other companies.

Drawn to Scale showed that its technology was viable in enterprise environments and established a “presence against  competitors who raised 10-100x more cash,” but even that wasn’t enough to save the startup from its financial woes.

As Hadoop evolves and different layers of the data analytics stack get commoditized, specialized vendors like Drawn to Scale will have problems surviving.   SQL-on-Hadoop was a unique feature set…but over time it has become a must-have feature, that is becoming embedded in the stack – e.g., Impala in Cloudera CDH stack.  As a result, firms like Drawn to Scale once unique functionality becomes difficult to monetize.

Startup to Viable Ventures

The Big Data ecosystem is exploding with exciting start-ups, new divisions and new initiatives from established vendors.  Everyone wants to be the vendor/platform of choice in assisting firms deal with the data deluge (Data growth curve: Terabytes -> Petabytes -> Exabytes -> Zettabytes -> Yottabytes -> Brontobytes -> Geopbytes), translate data to information to insight, etc.

In both U.S and Europe, several billion dollars of venture money has been invested in the past three years alone in over 300+ firms.  Firms like Splunk had spectacular IPOs. Others like Cloudera and MapR have raised gobs of money. In the MongoDB space alone – a small market of less than 100M total revenue right now, over $2 Billion is said to have been invested in the past few years.

Read more »

11
Dec

20 must read Infograhics on Big Data


Big Data emphasizes the exponential growth of data volumes worldwide (collectively, >2.5 Exabytes/ day).

Big Data incorporate the following key tenets: diversification, low latency, and ubiquity. In parallel, the emerging field of data science introduces new terms including, predictive modeling, machine learning, parallelized and in-database algorithms, Map Reduce, and data monetization.

A variety of infographics have been published around Big Data, Data Scientists.  Here is a compendium of some very interesting ones.

The Real World of Big Data  (Click image to see a larger version and article)

Real World of Big Data

Big Data Big OpportunityBig Data Big Opportunity

A Data Scientist StudyA Data Scientist Study

31
Oct

Email Marketing is a Predictive Analytics Problem


targeted segmentation for email using big dataDigital Marketing from 1999 to 2012

In his book Permission Marketing, Seth Godin referred to email marketing as “the most personal advertising medium in history”.  That was 1999.

Where does email marketing stand in 2012 in the age of social media, omni-channel marketing and big data analytics? Here are some interesting data points.

Read more »

2
Jul

Enabling SoLoMoMe + Omni-channel Analytics


At the Analytics Executive Forum, I facilitated a session on Omni-channel analytics. It struck me how every leading consumer facing firm seems convinced that mobile is becoming the dominant B2C interaction channel.  Mobile is the gateway to insight based marketing and the “always addressable customer”….

Insight-based interactions –  The company knows who you are, what you prefer, and communicates with relevant, timely messages, using the power of analytical intelligence to detect patterns, decode strands of information and create meaningful offers and value.

The “always addressable customer.” This is a consumer who fits the bill on three fronts simultaneously: (1)

  • Owns and personally uses at least three connected devices; (2)

Goes online multiple times throughout the day;  (3) 

  • Goes online from at least three different physical locations

The opposite of insight-based is “spray-and-pray” marketing – The company has very limited knowledge about who you are, forgets what you prefer, and tries to reach you with off-target communications that alienate you – based on fragmented data, poor data quality and  inadequate integration, resulting in confusing, chaotic interactions.  A good example: “I have 2 million frequent flyer miles with your airline and still do not get any recognition, respect or value from this loyalty.”

As companies architect new insight based mobile use cases I suggest that they look at what is coming next. With IOS 7, Apple is delivering several new features – Passbook, Beacon.

Retailers, banks and other customer facing firms/brands better pay attention. 100+ million iPhones are automatically getting this feature with the new OS upgrade making this a mega-disruptor in the coveted target segment everyone is chasing. Read more »

19
Jun

Organizing for BI, Analytics and Big Data: CoE, Federated or Departmental


  • Data-as-a-Service:  Data Provisioning, Management, Lineage, Quality
  • Reporting-as-a-Service:  Dashboards, KPIs, Drilldowns/Aggregates…. Descriptive
  • Analytics-as-a-Service:  Predictive  Modeling and BI… Prescriptive analytics
  • Information-as-a-Service:  Threshold based Alerts, Exceptions, Mobile Prompts
  • Insights-as-a-Service:   ML/AI based…automated learning – ambient intelligence, Next best Offer/Action

Which strategy are you implementing?

Data is valuable. Data is plentiful. Data is complex. Data is in flux. Data is fast moving. Capturing and managing data is challenging.

So, if you are a senior leader in a Fortune 2000 company.  How do you structure your group to deliver effective BI, Analytics or Big Data projects? Do you have the right structure,  toolset, dataset, skillset and mindset for analytics and Big Data?

Organizing for effective BI, Analytics and Big Data is becoming a hot topic in corporations.  In 2012, business users are exerting  significant influence over BI, Analytics and Big Data decisions, often choosing analytics and visualization platforms and products in addition to/as alternatives to traditional BI platform (reporting and visualization tools).

Read more »

4
Jun

Omni-channel Conversion Optimization with Social Marketing/Media Analytics


Brooks Brothers is investing in tools & testing to improve the online experience – and ↑sales. In a test involving one product category page: men’s shirts. The retailer using Bazaarvoice Ratings & Reviews software, used customer reviews to sort items on the product page. Items with five ♥♥♥♥♥- the highest rating – appeared on the top of the page. The result:  a 9% lift in conversions   [Adobe Digital Marketing Symposium]

Are you ready to anticipate and influence your audience in a whole new way? Value migration from traditional marketing to 24×7 digital marketing is happening in leading firms.  Real-time marketing and conversion is now becoming possible.

Read more »

9
Apr

Bloomberg on Business Analytics


Interested in slicing, dicing, measuring, and analyzing data for customer and business insights?

According to a recent survey by Bloomberg, 97% of companies with revenues of more than $100 million are using some form of business analytics, up from 90% just two years ago.

While businesses have embraced the idea of fact-based decision-making, a steep learning curve remains. Only one in four organizations believes its use of business analytics has been “very effective” in helping to make decisions. Data is not just ignored but often discarded in many organizations as the business users can’t figure out how to extract signal from data noise.

Read more »

%d bloggers like this: