Limitations of automation orchestrators and the rise of automation optimization

Without a doubt, one of the key drivers of the Fourth Industrial Revolution is Robotic Process Automation (RPA). Organizations worldwide have increasingly leveraged RPA technology and are now adopting multi-vendor strategies for a multitude of enterprise automation tools, beyond RPA. From recent conversations with the VOCAL Council, I estimate that almost 40% of the nearly 40,000 customers using RPA are deploying a multi-vendor strategy.

RPA tools have evolved from simple bots that automate single, micro tasks or activities to more complex end-to-end, unattended solutions that can automate entire processes and deliver unprecedented benefits. However, automation management is the automation that automation vendors forgot.

At the heart of RPA is the orchestrator. While orchestrators have improved and some have moved to become cloud-based, several have not been rearchitected. As a result, the design debt built over the years (with a focus on selling bots vs. managing them) has limited orchestrators to basic operational bot metrics.

The high cost of orchestrators prevents organizations from fully capitalizing on the current limited benefits, and integration challenges make it even harder to incorporate multi-vendor orchestrators into the tech stack. The swivel chair approach (input data from one system to another) that automation is supposed to eliminate is back, with precious resources swiveling to manage multiple automation vendors, extract metrics only to populate Excel and PowerPoints, meticulously and manually maintain bot schedules and reschedules, and painfully pray that bot failure may be detected early.

Lack of support, missing automation management capabilities, and inadequate/missing self-recovery are just a few of the serious challenges in managing automations.

Automation vs. orchestration: Same pod, different peas?

The easiest way to understand the automation vs. orchestration divide is in terms of “one” vs. “many”. Automation is often about automating a single repetitive task or activity within a process to run on its own and with minimal (or no) human intervention. For example, you could set up an RPA bot to automatically create IT service tickets, another to launch a web server, and a third to change a line of code in JSON files. Each of these bots would continually execute its specific task until you stop it. Repetition is the name of the game. Intelligence and logic – not so much.

Orchestration, on the other hand, is about automating multiple tasks to work seamlessly together as part of a larger workflow. The effort could involve multiple environments, devices, services, and people. And that’s why it is much more complex than a single bot for a simple task.

This complexity makes it vital to understand the many steps involved during orchestration and how these steps intersect. It also requires seamless coordination to prevent bottlenecks and ensure that enterprises can successfully derive the expected benefits from the effort, whether it’s process optimization, error-free output, accelerated innovation, improved employee experiences, or faster time-to-market.

“Process automation is not as simple as automating every task within that process, there is also a layer of orchestration and task interdependency where most judgement calls and other complexities live. Automating and managing that layer is the toughest part of the journey.”

Max Ioffe, Global Intelligent Automation Leader, WESCO Distribution

Optimization vs. orchestration

There are network optimization tools that came following the shortcomings with network orchestrators. Cloud optimization followed a similar path.  One can integrate orchestration and scheduling capability to create dynamic schedules, create task queues, initiate workflows, and monitor and track executions to self-recovering and predictive maintenance. You can also incorporate triggering events and advanced logic to automate multiple tasks within a process and ensure timely, efficient and consistent execution.

Over time, these tools can confer benefits like reduced IT costs, higher-quality output, and limited process downtime and going a bit further even provide self-sustainability. Automation promises to remove the mundane, yet automation management is laden with the manual and mundane.

With automation optimization, your precious resources don’t have to worry about mundane activities like managing, scheduling, or restarting bots. The automation optimization tool steps in and boosts your license, utilization and infrastructure efficiency while driving higher employee engagement. 

“For automation to succeed, having & sustaining optimal utilization, is key. Automation optimization tools that integrate with multiple vendors and offer a single pane of glass to manage automations will lead to a better Total Cost of Ownership and perhaps even increased RPA adoption.”

Akash Choudhary, Director Enterprise Architecture, ServiceNow

Sound without music

Benefits notwithstanding, just orchestration – which is almost always part of the overall RPA solution package – has some limitations. For one, many of these solutions are so complex and prohibitively expensive that companies with small IT budgets and teams struggle just to purchase the solution, much less leverage its benefits.

Costs do vary, depending on the instances you want to deploy and can be a TCO barrier[1]. Support is often an afterthought, which impacts the quality and timeliness of automation maintenance, change requests and incident management. The longer the response time to classify, handle and resolve incidents, the larger the amount of disruption and potentially losses for customers. Qualified support people who understand the architecture of orchestrators are limited and hence often customers face more annoying sound than music, with limited or no recommendations on how to apply appropriate solutions.

Some orchestrators don’t monitor all incidents or provide a holistic view of incidents, which is key for incident monitoring, investigations, and root cause analyses. While it is possible to automate these aspects, many orchestrators don’t provide “automation management” capabilities. Further, they don’t provide a single, centralized “incident box” that can help organizations keep track of all their automation initiatives. Under this scenario, determining the lifecycle metrics of an automation program is almost impossible.

Many customers seek a self-recovery/self-healing capability and sometimes want the orchestrator to simply restart a service that stops or becomes unresponsive. At other times, they need higher-level orchestrated workflows to automatically start up a new virtual machine (VM), check that its services start correctly, update the DNS put it into the load balancer, and even shift services to a different data center. In today’s automation environment, if a Virtual Desktop Interface (VDI) fails, it cannot automatically resolve the underlying problem, much less reboot on its own in the fastest possible time. Additionally, due to this limitation, the enterprise cannot monitor CPU or memory usage or self-clone machines to dynamically scale up capabilities to match peak requirements or scale down (“shutdown mode”) when requirements are low.

Organizations must custom-build an inventory management system for the orchestrator, which can be an arduous and resource-intensive effort. An automation management solution with a built-in or integrated inventory management tool increases the visibility into the automation ecosystem. It provides opportunities to streamline systems maintenance, improve automation efficiency, enhance output quality (lower errors), and reduce downtime.

“You need to reduce costs, streamline, and improve visibility of your RPA bots to scale your automation program. An automation optimization tool to monitor and control bots and derive tangible business, value and automation lifecycle metrics is must.”

Amol Rajamane, Global Digital Automation Leader, DuPont

The rise of automation optimization solutions

In an ideal world, automation workloads – whatever their heritage – should be able to move seamlessly between and be shared among, automation providers, wherever the optimal combination of performance, functionality cost, security, resilience etc. can be found – while avoiding the dreaded “vendor lock-in.” What if your automation can meet your demand without demanding more from you? Automation hopping, at the risk of introducing yet another term in a busy highway of terminology in automation, is not a far reality. Consumption pricing will pave the highway.

Automation optimization is a category that brings together end-to-end automation orchestration and management capabilities to cut cost, increase performance, and measure business impact – across multiple automation tools.

Several niche solutions tend to focus just on the control aspect of automation management, ie addressing portions of the orchestration limitations described above. Automation executives are often looking for a holistic, cost-efficient solution. CFOs are seeking a holistic, cost-efficient solution – an integrated, all-around athlete vs. buying multiple brands of shoes to potentially become an athlete.

An end-to-end automation optimization solution strings together the automation lifecycle: idea generation and classification, document gathering (discovery), building the bot, controlling the bots with dynamic scheduling, license/bots/utilization optimization with dynamic caching and AI-powered bot failure prediction, and deriving key value metrics. This approach provides the necessary operational (bot) metrics, value (KPI) metrics, and lifecycle (idea to value) metrics to determine and communicate the automation value to your organization.

Automation optimization solutions address the shortcomings of existing orchestrators but also offer the ultimate outcome challenging and shaping the automation industry today: improving adoption and scale.

A few solutions have emerged in recent years that deliver all these advantages, address the limitations of older platforms, and allow organizations to easily add or remove bots to match their evolving automation needs.

“The cost of bot license, infrastructure and automation management creates an significant dent in the technology budget. To drive customer centricity, automation technology vendors are better served helping their customers improve their existing license, infrastructure, scheduling utilization rather than pushing more bots. The rise of automation optimization solutions is evidence that there is a major gap in the market.”

Ankit Thakkar, Automation & Finance Digitization Leader, Thermofisher

The future of orchestration and optimization solutions

As automation and orchestration technologies develop, many more cutting-edge solutions will emerge for different enterprise use cases. The best automation optimization solutions will allow organizations to capture all the benefits of large-scale automation at the lowest possible TCO. A solution, for example, like Turbotic (disclosure: I sit on Turbotic’s Board) speeds up bot development by allocating development tasks and approval steps appropriately and systematically. It also clearly shows the entire automation opportunity, all the way from ideation to implementation in a single flow. Further, the solution automatically creates an automation business case, boosts bottom-up pipeline generation, and supports value tracking and monitoring to bring greater transparency into the automation ecosystem.

The best automation optimization solutions also provide visually compelling dashboards and real-time metrics to measure these benefits and assess the true value of the automation initiative across the enterprise. In addition, they will effortlessly integrate with existing systems to minimize disruptions and deliver all the advantages promised by enterprise-wide automation.

Advanced orchestrator solutions enable organizations to implement hyperautomation with support for cutting-edge technologies like AI, ML, and cognitive NLP. These “predictive” orchestrators leverage data and learning to better orchestrate all automation using improved predictions and decision support. These capabilities enable enterprises to optimize their automation licenses and resources in order to optimize costs, throughput, compliance, and ROI, and eliminate the chances of costly SLA breaches.