How organizational learning can unlock more business value from machine learning

We routinely underestimate the effects that new technology has in the long run while also overestimating its impact in the short term. What has become known as Amara’s Law, in honor of the late researcher and scientist Roy Amara, is playing out now in many organizations that adopt artificial intelligence and machine learning (AI/ML) technologies. AI/ML has capabilities that seemed science fiction not long ago, from autonomous driving to image recognition and advanced robotics, with surely many more on the way. At the same time, even advanced enterprises can struggle to achieve the anticipated return on their AI/ML investments. Although expectations might have been set high, following several guidelines has helped our customers maximize the business value delivered by their AI/ML initiatives. We’ll share them in this AI/ML blog post mini-series.

There Is No Magic: Acknowledge the Uncertainty

The pressure to deliver solutions faster and at lower cost while ensuring compliance, security, and reliability has caught many IT departments between a rock and a hard place. It’s no surprise, then, that they’re looking for new solutions to break through old barriers, and rightly so: modern technologies coupled with new ways of working can help transform IT from a cost center to an innovation driver. However, too often the latest technology is heralded as the panacea that’ll cure all the previous technology’s shortcomings. It’s therefore important to remind ourselves that while modern IT solutions can overcome past limitations, there is no “increase my customer satisfaction” algorithm, neither within AI/ML nor in any other technology.

AI/ML solutions model human reasoning by allowing the computer to make judgement calls (“inference”) based on past positive or negative outcomes. Just as for human reasoning, the result remains an educated guess. Building more sophisticated models can increase accuracy, but the business value of that accuracy needs to be weighed against the cost of building and training the model.

Upfront business cases that treat AI/ML initiatives as if they were deterministic software programs run the risk of overinflating expectations. You can’t expect an out-of-the-box AI/ML model to already be perfectly calibrated to your business problem and immediately make good decisions. The value the solution delivers depends largely on how well you’re able to balance investment and gain in the face of uncertainty. For example, if you’re looking to use a new data source for ML (or any analytical process for that matter), you’ll face a high degree of uncertainty, both for the required investment and the possible benefits. Let’s say you want your ML product to use daily stock levels from your warehouses around the globe to decide which items to advertise. You can’t be sure whether the data quality is sufficient for automatic processing, how much harmonizing data from the different warehouses will cost, whether daily stock levels are even relevant within your business processes, or how you can use that data to improve your top line or bottom line KPIs.

Many enterprises got burned by taking an approach of “put all the data into a data lake and find value later.” Just adding “artificial intelligence” into the mix won’t change that. Instead, reduce the uncertainty step by step while learning which investments impact your business value. Building on top of a cloud analytics and AI/ML platform is a great way to limit the initial investments as your learning journey begins. Once on your way, use the following three mechanisms to continually improve and refine.

1) Estimate the Value of Right Decisions and the Cost of Wrong Ones

Let’s assume you’re looking to apply ML to improve sales forecasts for your consumer product, a common use case we assist many customers with. Such a system can achieve several positive business outcomes, including higher net sales by avoiding out-of-stock situations, increased customer satisfaction thanks to shorter delivery times, and cost savings due to operational efficiencies. A perfect ML model would work toward all three equally, but ML models aren’t perfect.

ML models aren’t clairvoyants. No matter how well trained, they make erroneous judgement calls, in both directions! The costs of those errors are determined by your business and are likely asymmetric. For example, overstocking seasonal goods might be as bad for your business as understocking them. Also, some customers might happily wait five days for a high-demand article as long as they have a firm delivery date. Given your cost of goods sold and storage cost, it might be more cost-effective for your organization to store items in a central warehouse and pay for premium delivery rather than maintain specialized or regional warehouses. As with any IT initiative, a clear understanding of your business economics is critical to improving your bottom line.

Identifying potential high-cost mistakes at the start of an AI/ML project calibrates expectations and allows adjustments early on. It will also help you avoid building ML models that are penny-wise but pound-foolish. Because such mistakes come in many shapes, let’s look at some possible scenarios for a retailer’s sales forecasting:

Goal: Increase net sales by forecasting demand and stocking high-demand articles beforehand.

• Overestimating an increase in demand can lead to excessive stock or unwanted price reductions when the forecast doesn’t account for seasonal demand changes or articles that already are at the peak of demand.
• Misaligned advertising campaigns, e.g., those that advertise low-in-stock articles, can invalidate demand forecasting. Instead, consider advertising articles when demand isn’t picking up as expected.

Goal: Decrease time to delivery and thus increase customer satisfaction. Use region-specific demand forecasts to replenish local warehouses.

• Error rates are typically higher for larger forecasting windows, but short-term errors can have greater impact on customer satisfaction, especially when aiming to fulfill delivery promises.
• Misinterpreting customer demand can diminish the positive effect. For example, customers might prefer a later guaranteed delivery date over premium shipping options.

Goal: Increase operational efficiency by automating processes. Use demand forecast to automate replenishment and transportation orders.

• Full automation might ignore expert know-how for high-revenue articles, such as fan gadgets or streaming devices before major sports events.
• On the other hand, not trusting the ML forecast places an additional burden on experts to validate recommendations, reducing the positive effect ML was expected to have on the business.

The cost of these mistakes will depend on the dimension of the business value you’re looking at. It’s tempting to tweak an AI/ML model to avoid each high-cost mistake, but doing so will increase costs, as completely new data sources might have to be integrated or spin-off models might have to be designed for certain article subgroups. So, look beyond the actual model to mitigate the high-cost mistakes. For example, you could couple the forecast more closely with your pricing strategy, campaign calendars, immediate customer feedback, or route optimization.

Don’t expect your model to solve all your problems. Positive impact might be more easily achieved with surrounding systems or processes, ideally supported or informed by your model.

Sound complex? That’s precisely the point. Estimating realistic cost and gain is not a one-dimensional, one-off calculation. That’s true for conventional analytics and AI/ML initiatives alike.

2) Measure and Improve with Time

Because the route to business value is not a straightforward path, you’ll want a continuous process that reduces uncertainty by gathering additional information from a working AI/ML model. The likelihood of good decisions will increase over time if you keep honestly measuring, learning, and recalibrating the model along each value dimension. That’s why Michelle Lee, VP of Amazon Machine Learning Solutions Labs, warns:

Developing machine learning applications is an iterative process, requiring experimentation. If your organizational culture doesn’t encourage experimentation or if it treats failure (AKA learning) as something to be avoided at all costs, then this will be a significant barrier to applying ML effectively.

This doesn’t mean recalculating business value after every sprint or small experiment. Instead, the AI/ML development life cycle provides several natural milestones to recap leanings:

1. Before implementation: Estimate upper and lower ROI bounds based on use cases and industry benchmarks early on. This can be done with minimal investment to prioritize use cases for ideation.
2. Proof of concept: Conduct a proof of concept to estimate ROI with your own data, your own IT landscape, and your own business processes. Although it requires a small investment, it can be used to identify the most promising use cases, worth a more substantial invest.
3. Minimum viable product (MVP): Build a minimum viable product to validate ROI in real business scenarios. This medium investment will help create a business case for a full-featured product.
4. Fully operational product: Establish a stable, cross-functional team to take ownership of the AI/ML product. Being operational doesn’t mean you’re done: the team will constantly measure the solution’s costs and gain to improve it and adjust it to changing conditions.

3) Know When to Pivot

Iteratively calculating your AI/ML solution’s business value lets you adjust the solution according to what you’re learning. Often, companies calculate a detailed upfront business case, which is prone to being based on unrealistic and fuzzy assumptions. When those assumptions meet reality later, companies then twist the project output to fit the original business case. The only thing this achieves, however, is cheat you out of business value.

You need to prepare to pivot and change course, whether that means adjusting the model type, acquiring additional (possibly external) data sources, or adapting the business process. It might mean switching to another use case altogether. Remember the above forecasting example: instead of trying to predict future demand ever more accurately, pivoting to a live view of all inventory (whether in production, in stock, or in transit) can enable a quick and lean reaction to customer demand, even if forecasted inaccurately. Similarly, one can pull levers to influence demand by dynamically adjusting pricing and campaigns, rather than seeing these as unconstrained inputs. This can not only decrease the forecasting error rate but also increase full-price sales. It might be worth experimenting whether search terms or customer comments on the website have predictive power for short-term demand.

No one can foresee what the best pivotal moves are for your ML model and your business. And that’s okay. ML solutions aren’t copy-paste exercises based on other organizations’ successes. They are tools to maximize the value you derive from your assets or to overcome your specific constraints.

Utilizing Emerging Tech Means Constant Recalibration

A deeper understanding of AI/ML methodologies can help you realize greater business value. An end-to-end understanding of how your business dynamics interact with the capabilities of emerging technologies, however, can achieve much more.

That’s why AI/ML isn’t just another technology project. Remember these core points:

• AI/ML models make mistakes. To properly estimate the value for your business, you need to be able to weigh the impact of mistakes against the gain of right decisions.
• Realistic business impact is based on complex dynamics across KPIs and is hard to capture precisely before starting. Instead, incrementally reduce uncertainty with experiments and continuous learning.
• You must be willing to make changes based on the learnings, even if it means abandoning what once looked like a good idea. Be prepared to pivot based on using an early stage of the solution in your actual business process.

These steps are closely tied to making your organization data driven, whether it includes AI/ML technology or traditional analytics. Any of the above calculations can and will change over time, along with your business and customer demand. Incorporating this evolution into a sustainable AI/ML software life cycle will be the topic of our next post.

AUTHOR BIOS:

Annina Neumann is AI/ML Strategist at AWS Professional Services. In her role, she supports AWS customers to utilize emerging technologies for business value generation. Annina has been a lead advisory across diverse industries, such as retail, media or manufacturing. She has 15 years of experience with all things data, ranging from hands-on data science to leading cross-functional data technology teams.

In his role as Enterprise Strategist at Amazon Web Services, Gregor advises technology leaders in the transformation of both their technology platform and their organization. Drawing on his experience as Smart Nation Fellow to the Singapore government and as Chief Architect at Allianz SE, he connects the corporate strategy with technical decision making and vice versa. He enjoys sharing his thoughts on architecture and architects in his books, including The Software Architect Elevator and Cloud Strategy.