AI agents make great teammates, but don't let them code alone – here's why

Agentic AI may offer compelling productivity benefits, but it still falls flat when it comes to the heavy lifting of day-to-day operations and technology development. Still, technology leaders and proponents see great advantage in putting agents to work in many key areas of their businesses.

At the end of last year, Carnegie-Mellon University researchers released details on the performance of a mock company they assembled running entirely on AI agents. The experiment continues, but to date, the performance of the company, called TheAgentCompany, has been subpar, suggesting that agents aren’t quite ready to run completely on their own on a daily basis. 

Also: Can you build a billion-dollar business with only AI agents (yet)?

The agents, powered by both closed API-based and “open-weights” language models, were capable of completing a maximum of 30% of their tasks autonomously, but that’s about it.

“This paints a nuanced picture on task automation with LM agents — a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems,” the researchers wrote.

“While AI agents occasionally performed simple, isolated tasks well, the study makes it clear that they can’t yet handle the kind of complex, dynamic work that humans excel at,” said Dusan Simic, CEO and cofounder of 2immersive4u, in a LinkedIn post. “The researchers concluded that current AI is best described as a sophisticated extension of predictive text — good at pattern recognition, but lacking true understanding, adaptability, and independent problem-solving skills.”

Software development obviously falls into the category of complex, dynamic work. Are AI agents truly capable of more fully taking on such tasks? While industry observers have mixed feelings about their abilities so far, they are optimistic about the potential. 

“In recent months, there has been a huge shift to using AI tools for programming and in our day-to-day jobs,” said Ashwin Das Gururaja, senior engineering manager at Adobe. “AI agents, code assistants, and tools are changing the way we go about our daily activities. Agentic AI tools are great for prototyping and brainstorming, and I see experienced software engineers using them to accelerate their development cycles.”    

But many pieces of the development and deployment process remain well beyond the scope of agents, he continued. “AI agents provide an abstraction layer over complex software code. But this still requires skilled engineers to verify, guide, iterate, and refine the output. While many non-engineers may use AI agents to solve simple problems or for quick prototyping, they could struggle with debugging issues they run into — especially when they lack the understanding of the underlying code that AI abstracts away.”  

Also: AI agent adoption is driving increases in opportunities, threats, and IT budgets

The narrative of completely agent-driven software development “overlooks the deeply human and multifaceted nature of software development,” agreed Keith Kuchler, chief development officer at Sumo Logic. “While AI can undoubtedly automate certain repetitive tasks and even contribute heavily to code generation, the crucial aspects of understanding user needs, architecting complex systems, making nuanced design decisions, and ensuring the security and reliability of software will remain firmly in the realm of skilled engineers.”

If anything, AI agents “will free them from lower-level repetitive tasks to focus on higher-level strategic critical thinking, innovation, and problem-solving,” he added.

Rather than AI agents autonomously building software, “I think they’ll become something closer to a new kind of teammate,” said Spencer Kimball, CEO at Cockroach Labs. “You’ll be expected to manage intelligent agents as part of your development workflow — asking the right questions, curating the right context, and evaluating the output critically. That’s a skillset shift, not a replacement.”  

Also: How ChatGPT could replace the internet as we know it

Ultimately, AI agents bring “the potential to dramatically augment the work engineers do, such as compressing the cycle time of problem-solving, lowering the barrier to entry for certain tasks, and shifting where human time and creativity are most valuably spent,” he added. “That means engineers will be able to focus more on architectural thinking, system design, and solving harder, more ambiguous problems.”

To prepare for this new environment, software professionals “should start by adopting these tools on a daily basis and try them out with a trust-but-verify mindset,” Das Gururaja advised. “They should also actively look for new development in this area.”

As an example, he noted, “Anthropic launched MCP in November 2024 to connect AI assistants with other data sources and tools and to plug in different sources of context. This now has wide acceptance, and there is a rush within our teams to also build workflows that can use MCP and have our services expose data to MCP. Engineers are having to ramp up on such advances in a matter of days, unlike any advances in the past. Software engineering will have to constantly learn and relearn to keep up.” 

Want more stories about AI? Sign up for Innovation, our weekly newsletter.