How Booking.com measures the impact of GenAI on developer productivity

As an established name in online travel, Booking.com has long maintained its position as a global leader—today, they attract over 500 million monthly visitors and more than a billion annual bookings. But sustaining their market position going forward, in the increasingly fast-evolving travel industry, has required Booking.com’s leadership to shift its attention toward accelerating their pace of software delivery and innovation. To help achieve this, the company is heavily leaning into rolling out AI across its 3,000-person engineering organization.

“We’re very excited about GenAI because it has the potential to multiply the impact of our engineering organization,” explains Amos Haviv from Booking.com’s Developer Experience team. “Our vision is to give every developer the equivalent of a senior engineer sitting beside them to pair program and tackle problems,” adds Bruno Passos, Group Product Manager of the team responsible for spearheading their AI efforts.

Confronting early challenges with AI adoption

When Booking.com first began rolling out AI code assistants for developers, they faced challenges due to differing expectations of executives and developers. Due to the immense hype around AI, some executives had sky-high hopes for what could be achieved, while developers expressed skepticism toward the true utility of these tools.

“The whole world was talking about AI, and the numbers we were hearing were astronomical: claims of hundreds of thousands of hours saved for developers,” shares Passos. “But nothing was backing it up. So our initial goal was simply to understand what GenAI could actually do for us and figure out a way to measure the value it was delivering to the organization.”

In addition to measuring the true impact of GenAI on development, Booking.com also needed data to guide key decisions around vendor evaluations, as well as to focus its training and enablement efforts to drive successful adoption.

“We needed to understand how AI affected engineering velocity, satisfaction, and code quality,” says Passos. “Without that insight, we wouldn’t be able to confidently talk about the ROI of Booking’s investment in AI with the rest of the business. We also wouldn’t know for sure whether we should be driving more adoption for a specific tool or use case.”

Implementing a data-driven approach

To measure AI tool adoption and impact, Booking.com partnered with DX, the developer intelligence platform designed by leading researchers. Using DX, Passos’ team was quickly able to begin quantifying the impact that AI code assistants were having on their software development teams.

Their findings included:

1. Developers who used AI daily had 16% higher change throughput than those who did not.
2. AI users were saving time on routine tasks, allowing them to focus on higher-value work.
3. Developer satisfaction with AI tooling had risen 15 points in the previous six months, thanks to product improvements and internal enablement efforts.

The above findings were derived from both qualitative and quantitative data collected through the DX platform. By combining direct signals from developers, along with longitudinal and cross-sectional analyses of multiple productivity signals, Passos’ team gained an in-depth view into the current state of AI impact across the organization.

Zane Wright, Senior Product Manager, shares: “We’ve been able to use data to make tactical and strategic decisions on where to invest further in our GenAI program. Decisions like which vendors we should be going for, where we should be looking further to assess deeper impact on our company, and how we should be structuring our programs to best impact developers moving forward.”

One insight that surprised Passos’ team: a large number of developers were still hesitant to adopt AI tools due to skepticism and confusion. To address this, further investments in developer enablement were needed. “We realized that education would be just as important to increasing adoption as improvements to the technology itself,” shares Passos.

Moving beyond initial adoption to full usage

Armed with data showing the positive impact of AI on developer productivity, Passos’ team set its aim on driving toward 100% adoption, as well as finding ways to encourage developers to incorporate AI tools in their work more frequently.

“The data from DX showed us that developers were most effective if they used AI on a daily basis, or at least twelve days per month. So we set out on a mission to figure out how to drive not only adoption, but widespread daily usage,” says Passos.

With this goal in mind, Booking.com has implemented several strategies:

Segmentation and targeted outreach: The team uses DX to identify which developer segments are getting more or less value from AI tools. “We’ve found segmenting data particularly valuable for driving AI adoption,” explains Bailey Stewart, Principal Software Engineer. “This tells us which communities within Booking have not been finding as much value in GenAI tools. Then we reach out to them to figure out what we can do to help them find more value.”

Education and hands-on experiences: The company is running a series of two-day events where day one focuses on GenAI education (LLMs, prompting techniques, and context handling) and day two focuses on applying these techniques to solve real problems. “We pick up a business problem from a specific business unit and attempt to solve it with GenAI by bringing in internal and external experts,” Passos explains.

Continuous communication: The team regularly shares updates on new AI features and capabilities through internal content. “Every time we’ve updated our AI coding assistant with new features, we post and communicate what developers can now do,” says Passos. “For example, initially, we could only use one LLM; now developers can use several LLMs depending on their specific task. Each time we make a change like this, we communicate it internally to the rest of the organization.”

Office hours and training: The team hosts regular office hours where developers can get help with AI tools and learn best practices. “We now have almost 100% of our developers adopting GenAI. Some of the biggest keys to achieving this have been our office hours, as well as producing content on how to use GenAI: for example, what’s an LLM, how to prompt… There’s a lot of video content that we produce and post on a regular basis,” Passos notes.

Thanks to these targeted efforts, Booking.com is already seeing a significant increase in developer adoption and utilization of AI tools. This has led to further increases in productivity gains, with fully adopted teams showing 30% higher throughput than non-fully adopted teams.

Looking forward, Booking.com is focused on continuing to augment its software development lifecycle with AI tools, ensuring that the business remains competitive in a fast-moving landscape.

To learn more, visit us here.