Big Data Knows You Like Losers

Bloomberg View , July 29, 2015

Most of the data captured about our everyday transactions isn't very exciting. Take frequent shopper cards. When I visit the Ralphs supermarket website, it highlights sales on avocados and Hunt's diced tomatoes. CVS calls my attention to deals on Glide dental floss and Neutrogena skin-care products. The stores know I buy these things because I've swiped my cards in exchange for discounts on previous purchases. This is just the kind of customer-specific record that expert salespeople at places like Neiman Marcus were keeping long before computers -- and that small-town shopkeepers used to simply remember. It's small data on a large scale.

But when you can compare all that information across millions of consumers and products and thousands of outlets, you enter the realm of big data, which can reveal previously unknown patterns. A new case in point: A paper forthcoming in the Journal of Marketing Research identifies a segment of customers, dubbed the "harbingers of failure," with an uncanny knack for buying new products that were likely to flop.

Researchers Eric Anderson, Song Lin, Duncan Simester, andCatherine Tucker analyzed more than 10 million transactions by nearly 128,000 customers of a chain selling consumer packaged goods over a two-year period. The customers represent a random sample of frequent shoppers at 111 different stores. Using this mountain of data, the marketing scholars asked whether who bought a new product made any difference to its success. Were there systematic connections between specific groups of customers and whether a new item sold well over time?

They found that strong early sales -- the traditional indicator of product success -- in fact didn't matter as much as who the early buyers were. And one startling finding was the emergence of an identifiable segment of customers more prone to buying new products destined to survive less than three years, as well as unpopular "very niche" existing products.

"Because these guys are so consistent in behavior, if you're selling to a lot of them you're really in trouble," said Anderson, a marketing professor at Northwestern University's Kellogg School of Management, in an interview.

The new products that harbingers bought weren't crazy -- all had made it through test marketing into national distribution -- yet a mere 40 percent survived for three years after introduction, costing retailers and manufacturers big bucks. "Introducing a flop and keeping it for one year results in lost profits [to the retailer] equivalent to 49% of the average annual profits for an existing item in the category," the researchers write. So if a retailer knew that harbingers were accounting for early sales, it could save money by pulling the product earlier and replacing it with something likely to do better.

To make sure that harbingers weren't just a fluke of one chain's customer base, the researchers are repeating the analysis using an even larger data set: six years of records from all major U.S. grocery store chains, collected by the market-analytics firm IRI. So far, says Anderson, "everything holds up -- all results replicate." Strong initial sales to the wrong people foretell product doom. It even turns out that harbingers of failure in one category -- food, say -- often pick losers in other departments, such as health and beauty.

The larger data set offers more information about who the harbingers of failure are: wealthier, more highly educated, with larger families than other customers. These factors, Anderson suggests, make it easier for people to take chances on new items, including unpopular ones.

Most enticingly, the larger data set points to possible harbingers of success, although the connection isn't as strong. "To get a little better at predicting success could have a big payoff for firms," says Anderson. "That's where we're headed in the future -- to try to work not only the failures but also the harbingers of success."

He'd also like to move beyond consumer packaged goods to other businesses with high-stakes product introductions. "We would love to do this with movies," he says when I bring up AMC Theatres' Stubs card. "We would love to do it with AMC."

The trick, he emphasizes, isn't compiling the data. It's thinking of how to connect it to what marketers need to know. "If you had the research idea, you could have done this a long time ago," he says. "The breakthrough was not the data; it was the idea."