Generative AI copilots: What’s hype and where to drive results

“Microsoft says Microsoft 365 Copilot is a general release, but it seems like it’s still in beta with features they advertise on their website that it doesn’t actually do yet,” says Kleinman. “They advertise a feature where you can follow a meeting, and then Copilot will join and take notes for you.”

Kleinman believes executives will want this added workflow functionality to make it easier for people to sit virtually in a meeting they want to be summarized, especially when they aren’t going to be active contributors. The ambiguity of what’s working today and which users will benefit is driving some CIOs to ask whether adding Copilot licenses to Microsoft 365 is worth the price.

Microsoft is heavily investing in AI capabilities and workflow integrations, so CIOs should expect and plan for improved capabilities. The biggest question CIOs should help answer is where to experiment and learn Copilot’s impacts on workflow. CIOs should seek out departments and employees that are heavy Microsoft 365 users and create opportunities for them to learn, try, and report on Copilot’s capabilities and benefits.

Who benefits from software development copilots

The benefits of using Microsoft Office 365 Copilot may lie in setting realistic expectations and evaluating whether the results improve productivity. For software developers, the benefits of using copilots and other generative AI capabilities may be more about who is using it and the cost-benefit of validating code results.

IT leaders are exploring how different gen AI tools transform the software development lifecycle. Many are preparing a new world of developers as AI agents, with software development being closer to a manufacturing process. Today, top AI-assistant capabilities delivering results include generating code, test cases, and documentation.

GitHub’s research shows that users accept 30% of code its Copilot suggests and that less experienced developers have a greater advantage with AI. The research claims developers are faster and more fulfilled when using Copilot.   

Ali Dasdan, CTO of ZoomInfo, says, “In just three months, nearly all of our individual contributors were onboarded to GitHub Copilot. We saw near-immediate success, as we accepted tens of thousands of lines of code suggested by Copilot with an accuracy north of 26%.”

Other tools for gen AI in coding include Amazon CodeWhisperer, Seek, and Tabnine.  

“I lead a team of 20 developers regularly leveraging generative AI as a coding copilot, and each one has seen productivity improve by 20% on the low end to 100% on the high end,” says Mike Finley, CTO and co-founder of AnswerRocket. He shares the allure of using gen AI, “I often just write a comment indicating what I want the next few lines to do, and AI fills it in,” but also the reality that they still need to review the code.

“We produce a lot of code, so additional efficiency and finding ways to improve the speed of how we develop solutions is crucial for us,” says Luis Ribeiro, head of engineering and digital solutions at CI&T. “Tabnine has boosted developer productivity, and our developers accept 90% of the tool’s single-line coding suggestions resulting in an 11% productivity increase across projects.”

Some CIOs I spoke with say they see fewer benefits in giving junior developers access, largely because of the skills required to prompt and validate Copilot’s code. CIOs may also want to consider each application’s usage, security, and risks to decide which devops teams should experiment with AI copilots.

“The secret to CTOs leveraging gen AI copilot tools is finding the right balance between leveraging AI assistance and maintaining human oversight and control to ensure optimal outcomes,” says Anurag Malik, President and CTO of ContractPodAi.

To drive results with copilots, IT leaders should weigh in on who should experiment, which business functions, what compliance considerations, and which AI gen tools. As copilot technology capabilities are changing rapidly, leaders should frequently identify metrics and evaluate strategies.