How to launch—and scale—a successful AI pilot project

At the US Patent & Trademark Office in Alexandria, Virginia, artificial intelligence (AI) projects are expediting the patent classification process, helping detect fraud, and expanding examiners’ searches for similar patents, enabling them to search through more documents in the same amount of time. And every project started with a pilot project. “Proofs of concept (PoCs) are a key approach we use to learn about new technologies, test business value assumptions, de-risk scale project delivery, and inform full production implementation decisions,” says USPTO CIO Jamie Holcombe. Once the pilot proves out, he says, the next step is to determine if it can scale. From there, it’s about the actual scaling and then full production.

Indian e-commerce vendor Flipkart has followed a similar process before deploying projects that allow for text and visual search through millions of items for customers who speak 11 different languages. Now it’s testing conversational bots that use deep learning to build models that incorporate user intent detection, language translation, and speech-to-text and text-to-speech functions. And both Flipkart and the USPTO are rapidly expanding the application of computer vision, natural language processing, machine learning (ML), and other AI technologies to other parts of the business.

But despite all the excitement around AI and ML, many initial pilot and PoC projects fail to move to full production. Successful projects need to be part of a strategic plan, garner executive sponsorship, have access to the right data, have the right teams in place, have the right technical and business metrics and milestones in place, go through many iterations—and fail fast. “This process can take a year or two to get to a high level of quality. That’s the level of patience you need,” says Ganapathy Krishnan, VP of engineering at Flipkart.

Set the stage for success

Enterprises are moving quickly to stage successful AI pilot projects, move them to production, and produce results. “We’ve seen AI projects enter the mainstream,” says Rowan Curran, analyst for AI. ML and data science at Forrester. “Fifty-seven percent of enterprises are implementing or extending their AI projects and 70 to 75% are seeing clear value from those projects.” Also, according to a recent EY survey, 53% of CIOs and IT leaders said data and analytics, under which AI fits, will be a top area of investment over the next two years.

But many of those pilot projects are doomed to fail before they get started for several reasons, starting with a lack of top-down support. “You need an executive champion, and you have to have the right funding,” says USPTO’s Holcombe.

Initiating projects from the middle of the organization or from the bottom up reduces your chances of success, IT executives say. The most successful projects happen when the CIO has executive support with a commitment to fund the project, and integrates AI into the organization’s overall digital transformation strategy.

Setting clear expectations is also key, says Flipkart’s Krishnan. “You should not have the expectation that you’ll deploy this thing and it will radically transform the business. It’s a lengthy process that takes time.”

A PoC can also be an exercise in building capability within the organization. That’s an approach that Eli Lilly has taken. “From PoCs, we experiment with and learn the dimensions of scale for technical and project delivery,” says Tim Coleman, VP and information officer for information and digital solutions at the pharmaceutical firm. The team is applying natural language processing capabilities for natural language discovery, generation and translation in areas of the business, ranging from clinical and scientific content authoring to product development, advanced search, and general administrative functions.

But don’t confuse those capability building exercises with pilot projects that need to generate broad transformational value, cautions Dan Diasio, global AI consulting leader at EY. “You want to build your ability so it can do this, but when it’s time to make the kind of impact required to compete with disruptors in the future that’s meaningful to investors, then you have to take a top-down approach.”

That’s how Atlantic Health System approaches AI and ML projects. The healthcare provider has scaled successful pilots in image evaluation to assist radiologists, and in preauthorization automation, which takes an order for imaging and moves it through several process steps through to scheduling. “AI should be part of a digital transformation, not an isolated initiative,” says SVP and CIO Sunil Dadlani. “We have a formalized governance structure and investment plans on how to go about AI and ML.” And over at Eli Lilly, project proposals should pass through three criteria before moving forward: offer business value in terms of ROI, have an acceptable probability of success, and the outcome must align with business strategy and priorities, Coleman says. For example, the primary driver for Mosaic PV, one of the company’s first AI projects, focused on adverse drug reaction reporting, was “to increase productivity and reduce the cost of processing adverse events, while maintaining a high standard of quality and compliance,” he says.

What’s the question?

A successful pilot starts by defining the business problem. “Don’t end up with an answer looking for a question,” says Sanjay Srivastava, chief digital strategist at global professional services firm Genpact, which consults with large companies on AI-based projects. “Projects focused on business outcomes that start with a question rather than an answer generally do well,” he says.

Then decide if AI is the best answer. “Does the project fit that bar that it’s complex enough to be worth doing?” says Krishnan. “If you can do it with a simple rules-based approach, do it. But when you have hundreds of thousands or maybe millions of rules, it’s not feasible to use a software-based approach.”

Do you have the right metrics and data?

Back at the USPTO, AI projects require two sets of metrics: The technical ones in terms of how the model performs, and metrics that quantify business value of your AI project.

Then Atlantic Health System ensures success by implementing a pilot with clear business KPIs for a small segment of the business. Its imaging evaluation system, for instance, started with a small pilot deployment in the neurology department that quickly scaled out to cardiology and other areas. In eight weeks the team created a successful pilot for neurology, demonstrated results, and got buy-in from cardiology and all of its other service lines.

And like the USPTO, Flipkart first focuses on the technical model metrics, then runs A/B tests to find out what impact it will have on the business. Currently, the team is working on developing and testing an AI-assisted conversational bot. They started with the metric of “answerability,” or how good the bot is at answering questions. They’re now running A/B tests to determine whether that will have a measurable impact on the business.

AI projects are heavily reliant on big data, and you need the right velocity, volume and variety, says Dadlani. “If your data quality is not great you won’t see those [expected] results.”

Genpact’s Srivastava agrees: “Data ingestion, harmonization, engineering and governance are 90% of the work that goes into building an AI system. If you focus on the 10% and let go of the 90%, you’re dead from the start. So build that foundation of data.”

You also need to be able to deliver continuous feedback between A/B tests—getting data back in real-time so you can tune the model. But your organization may not be set up to provide the data quickly and in an automated way. For example, if you’re working on a forecasting model and the team isn’t automatically capturing information on what customers are buying, you can’t close that loop. It’s also essential to continue the feedback loop after full deployment, as customer preferences can change over time. If your model hasn’t accounted for that, you won’t get the results you’re hoping for—an outcome known as “model drift.”

Will it scale?

While preliminary expectations may be that a pilot will be able to scale to a full rollout, the proof is in the pilot. So do you have the right resources to scale from pilot to full deployment? “To scale, you may need to streamline code, bring in new technologies, push your AI or ML to the edge versus having one data repository, needing to employ new teams, and set up a data labeling factory,” says EY’s Diasio. “There’s a whole suite of engineering skills that are required.”

Execute the pilot

Flipkart leverages the cloud and associated MLOps capabilities for its pilots. “To get started,” says Krishnan, “pilots need a lot of engineering support, must iterate frequently and fail fast, and to do that you need an MLOps infrastructure, which the big cloud service providers offer.” He recommends that the pilot team reports in with regular progress updates on how close they are to hitting targets, and make sure expectations are set correctly during the pilot.

“If you move the needle by 3% during your initial pilot you’re doing well,” he adds. And don’t expect to see gains right away. It’s difficult for a complex pilot to see an impact in three months. Deploy, find the gaps, deploy again, and keep moving up incrementally.

A failure along the way doesn’t necessarily mean the end of a pilot. The USPTO’s augmented classification system failed initially. “We started with a data set that wasn’t properly curated,” Holcombe says. But the team was able to readjust and proceeded with the pilot until the system performed substantially better than the manual process. “If you fail, don’t just give up. Figure out why you failed,” he says.

The final assessment

These CIOs, IT executives and consultants used a variety of methods to assess their pilot projects. At Atlantic Health System, once the initial pilot is completed it’s time to assess the results—and decide whether to extend the pilot, move forward to production or cut their losses. “A pilot must deliver the perceived measure of success,” says Dadlani. “Only when we see a promising result do we say, ‘What would it take to scale this up, how much time will it take, what will be the time to value, what investments will be needed for tech infrastructure resources, and how will we operationalize it.”

Eli Lilly’s Coleman says pilots fail for several reasons: insufficient AI skills, not enough labeled data, unclear project vision or value proposition, lack of an agile, fail-fast mindset, and a lack of executive sponsorship and organizational change management to drive business adoption.

Make sure you’re reporting out the metrics that matter to the bottom line. For example, if a pricing algorithm is projected to save $50 million, there may be a gap between what’s been realized to date versus what the expected potential is, says Diasio. “When you talk about big dollar projects, pilots often lack the credibility of generating that much value, so do the hard work to track realized value to the extent you can,” he says.

This is also the time to reassess whether the pilot will scale. “A lot of PoCs are very successful technically, but not economically when you scale it,” says Genpact’s Srivastava. Other considerations include how long it will take to scale, and what resources will be required.

But that picture might change when you take the long view. “Even in situations where scale may not be achievable in the short-term, a smaller project scope with high probability of delivery success may still deliver near-term business value while the technology capabilities and skills mature to address barriers to scale,” says Coleman.         

Then there’s infrastructure. Make sure you check all of your assumptions when scaling, including configurations, network bandwidth, storage and compute. “You’ll need a lot of engineering support to scale, and this is where cloud-based MLOps infrastructure can help,” says Krishnan.

Finally, make sure you can integrate AI into your upstream and downstream workflows. For example, predicting failures isn’t helpful if you haven’t integrated that into your upstream supply chain system to ensure that the spare parts are there when and where you need them. Likewise, that information should be used downstream to adjust maintenance schedules.

Start slow, fail fast, be patient

The key to a successful AI/ML pilot starts with initial planning. Get top executive buy-in and financial support before moving forward. “You have to have that top cover,” says Holcombe, and make sure you have all stakeholders onboard from the start.

An AI/ML pilot project should be undertaken as part of an overall digital transformation strategy, with a compelling business use case, says Dadlani. Achieving results from expectations takes patience. Create both technical and the business impact metrics that define success and know your capabilities as you make sure you have the right resources in place. Build the right team and be prepared to fail fast. So having the right mix of skills and domain expertise on the team is key to a successful AI pilot project. “You need a cross-functional team, even at the pilot stage,” he says. “We make sure everyone is involved [in the pilot] because this becomes part of the clinical workflow. They have to be involved from the beginning.”

Organizations that don’t have all of that talent on staff should consider building a hybrid team with external partners, while small and mid-sized companies will probably need to outsource more roles — if they can find the talent. “If you don’t have the right AI/ML engineers and data engineers, it’s super-hard to outsource that,” says Srivastava. What’s more, you need people on your team who understand both ML and your industry, such as manufacturing. That’s not a skills combination that’s easy to find, so cross-training is critical.

Ultimately, consider a targeted project that can produce real business results, then scale to other areas of the business, as Atlantic Health System did with its ML-based imaging evaluation system.

Once a pilot moves to full production, build on what you’ve accomplished. Keep the business up to date on pilot progress, showcase the project’s capabilities once fully deployed, and create platforms that other business units can leverage for their own applications. “The pace of change today is the slowest it will ever be,” says Srivastava. “Corporations that want to disrupt and grow have to change the way they drive value, and you can’t do this without AI. If you don’t invest in it, you’ll have one hand tied behind your back.”