When ‘quick wins’ in data science add up to a long fail

I’ve worked with many clients to help them get a data science operation up and running for the first time. This is a major challenge for any business, no matter whether it’s a scrappy startup or a Fortune-500 behemoth.

A lot has been written about why so many of these initiatives fail. But I think there’s a failure mode for these projects that doesn’t get nearly enough attention. This is when a focus on “quick wins” eventually creates a “long fail.”

Why quick wins?

If an organization is attempting to apply data science for the first time, then there is a common set of challenges it must overcome.

First, there is no institutional knowledge of data science. This means that the stakeholders throughout the organization have no way of knowing how or whether data science can be applied to their problems. They’ll have an entrenched way of doing things; and their way of doing things may even be quite good. But data science isn’t even on their radar, so you have to help them understand a little bit about data science before you can even begin to think about applying it.

Second, the organization’s operations and technical infrastructure will be inappropriate to support data science. Data will be spread across silos that were built to answer specific questions. No single person will have a high-level view of all the information available. Procedures for running the business, interacting with customers, processing transactions, and so on will be tightly coupled to people’s roles within the company as well as to the present infrastructure.

In short, there’s a lot of education and foundational work to be done. And if you’re in an organization that doesn’t have experience with data science, it’s likely there will be some skepticism about the necessary up-front investment of time and money.

Thus, the “quick win.” Find a project that has few technical or operational requirements. Apply data science methods to it, generate some measurable results, and show value as quickly as possible. Use the quick win to recruit allies and to justify the large investments that will be necessary.

Failure modes for quick wins

This is a perfectly reasonable strategy, even a necessary one. And it can work. But I’ve also seen it fail in subtle ways that are difficult to detect because it’s possible to fail in the long run by repeatedly being successful in the short run.

The nature of the quick win is that it does not require any significant overhaul of business processes. That’s what makes it quick. But a consequence of this is that the quick win will not result in a different way of doing business. People will be doing the same things they’ve always done, but perhaps a little better.

For example, suppose Bob has been operating a successful chain of lemonade stands. Bob opens a stand, sells some lemonade, and eventually picks the next location to open. Now suppose that Bob hires a data scientist named Alice. For their quick win project, Alice decides to use data science models to identify the best locations for opening lemonade stands. Alice does a great job, Bob uses her results to choose new locations, and the business sees a healthy boost in profit.

What could possibly be the problem? Notice that nothing in the day-to-day operations of the lemonade stands has changed as a result of Alice’s work. Although she’s demonstrated some of the value of data science, an employee of the lemonade stand business wouldn’t necessarily notice any changes. It’s not as if she’s optimized their supply chain, or modified how they interact with customers, or customized the lemonade recipe for specific neighborhoods. The only difference is that instead of Bob poring over spreadsheets to find the next location for a lemonade stand, he now looks at reports that Alice’s model has generated.

This is an almost necessary aspect of the quick win strategy. But it has some dangerous consequences.

First, nobody has been challenged to imagine new ways of operating the lemonade stand business. Instead, they’ve been shown only that there may be more effective ways of doing the same things they’ve always done. If anything, the existing business processes have inadvertently become even more entrenched because Alice has shown everyone how to extract a little more incremental value. When this happens, Alice will probably get a lot of requests from other people throughout the organization to help them do their jobs a little bit better.

Second, quick wins are rarely game-changing. They are not the 10X improvements that data science can often provide when it’s done correctly. Let’s say that as a result of Alice’s work, new lemonade stands are 5% more profitable. That’s a very good result, and impactful to the business. But now, we cannot blame people for thinking that incremental improvements are what data science is good for. Because Bob made the investment to hire Alice, and she generated a 5% boost in profit, we can’t blame Carol for thinking that this is the level of impact that can be expected from data science teams. People’s expectations become anchored on incremental improvements.

Third, a similar anchoring happens with respect to investment in data science. Alice chose this initial project because it didn’t require a lot of time, money, or personnel. So again, we can’t blame Carol if she starts to think that data science doesn’t require much investment.

So Alice’s quick win has backfired. Even though it was intended only to be the business’s initial foray into data science, and even though the project was successful, it has now become more difficult to do data science in the long run. Alice risks being dragged into low-impact, incremental work in an organization that becomes steadily less likely to make the necessary investments in data science.

In my experience, this is a very common failure mode for new data science teams. Businesses invest in data science because it promises to be transformative. But instead, it turns into nothing more than a shiny new way to do the same dull stuff, providing merely incremental improvements in efficiency. And this long-term failure sneaks up on people because it’s the result of repeatedly succeeding in generating quick wins.

Avoiding the Pyrrhic victories of quick wins

I said earlier that the quick win strategy can work. And it can. But you have to think long-term, even as you aim to generate short-term results.

The key to avoiding this trap is to build a long-term plan into the quick win project. Make the quick win an incremental step toward a larger, truly transformative goal.

Let’s return to Alice. Suppose she had gone to Bob with the following proposal:

“Our biggest expense is the high cost of sugar. If we optimize our supply chain and bidding process for sugar, we can transform the business. We’ll need real-time intelligent bidding, just-in-time delivery based on dynamic demand forecasting, and smart routing of deliveries to our lemonade stands.

We’re not in a position yet to do that. But we can take a step toward that goal by optimizing the locations of new lemonade stands. That way, when we get to the point where we can optimize our supply chain and delivery network, the lemonade stands will be in locations that are best suited for those changes.”

Then Alice can do the same quick win project as before. Along the way, she’ll create a tidy increase in profits for the business. But now the lesson that people learn is totally different. If Alice does a good job of continually reinforcing her long-term vision for the company, people will see her work as just one step toward a much more ambitious goal. The quick win can be leveraged into a compelling argument for making the investments Alice needs in order to transform the business.

In short, this is the way to avoid having a quick win backfire into a long fail. But in order to pull this off, everyone has to be much more strategic from the outset. Here are a few specific tips:

1. Establish a partnership between data science and the people who understand the opportunities for long-term transformation. Data scientists need to learn to listen to those people so they understand where the long-term opportunities are for 10X improvements.
2. Pick a quick win project because it’s a step toward that goal, not just because it could generate some value quickly. If you can’t frame your quick win in terms of moving toward a long-term goal, then it’s not the right project. This may mean your quick win isn’t quite as quick as it could be. But that’s okay.
3. Relentlessly reinforce the vision. Talk about the long-term transformation every time you report on the status of the project. People who aren’t used to thinking about data science need to have the vision reinforced. Help people understand that their jobs might change significantly and that this is a good thing.

Upshot

In short, avoiding the trap of the quick win requires two elements. First, you need to focus on the long-term, transformative goal even as you try to sell a project that has limited scope. Second, even as you focus on the short-term work, you have to help everyone keep their eye on the long-term vision by consistently reinforcing the message that this is only one step of a long journey. The end result is an opportunity for major transformation and some fascinating data science along the way.

Zac Ernst is Head of Data Science at car insurance startup Clearcover.