Building a Commercial Scale 4G/5G Cloud Native Architecture and Platform

Over the last thirty years, the concept of a mobile core has evolved dramatically. From analog origins relying on circuit switching to the introduction of packet switching in the early 1990s, the first generation of mobile packet cores were vendor appliances with specialized hardware.   A great example of this is the Cisco ASR 5500, which tightly integrated hardware with software to provide industry-leading reliability and performance.   Although the ASR 5500 performs admirably, the strategy of building, maintaining, and upgrading dedicated appliances is expensive, when each new generation requires new custom components like data processing boards for higher performance.

Advances in off-the-shelf hardware and open-source software, such as 25/40G NICs, SRIOV, DPDK, and VPP, have enabled the deployment of more cost-effective mobile packet cores that meet the performance demands of mobile network operators, and Cisco has led the industry in this area by developing the Cisco Ultra Packet Core for virtualized environments.   This network function virtualization (NFV) had a hardware cost advantage over traditional appliances but proved fragile due to the complex NFV deployment architectures required to deploy virtual network functions (VNFs).  As a result, NFV deployments often have more operational costs than traditional appliance-based models.

The transition to 5G provided an opportunity for the industry to leverage new technology developed to deploy applications across public and private clouds.  The 3GPP standards body encourages the use of cloud-native technologies and has emboldened the industry to focus on the decomposition of applications into composable microservices.  By embracing a cloud-native architecture, the industry is steering in a new direction, away from the unreliability and complexity issues that troubled the industry’s initial attempt at transitioning with virtualization.

Reliability, Operational Simplicity, and Scale

A Kubernetes-based cloud-native solution was the obvious choice for how we went about building our Converged Core. Embracing Kubernetes provides numerous benefits, such as rapid application development, new CI/CD delivery patterns, and better resiliency models.  While Kubernetes is helpful for managing the multitudes of containerized applications in this new cloud-native landscape, the pitfalls of reliability and complexity that plagued the early VNF deployments across the industry remained.  As promising as cloud-native software containers are, developing a converged core required marrying this new cloud-native approach with a comprehensive architecture—an architecture that had yet to be defined.  When we began defining what a Converged Core architecture may look like, we wrestled with many choices:

Choice 1 – BareMetal vs Virtualized Deployments

In evaluating how we should deploy our new Converged Core we considered the existing NFV architecture with Kubernetes embedded within the VNFs or a BareMetal deployment model.  BareMetal became the clear choice, it allowed us to simplify the solution and increase reliability by eliminating complex and failure-prone parts of the previous NFV architecture.  Gone were the VNF manager, NFV orchestrator, VIM, hypervisor, and all the complexity and friction that came with these components.  What was left?  A hardened Linux OS running on top of UCS M5 hardware.

Choice 2 – The Cloud-Native Stack

The Cloud Native Computing Foundation (CNCF) landscape provides an abundance of solutions for building a platform stack, even providing a helpful map (https://landscape.cncf.io/) that engineers can use to visualize options in building a cloud-native stack.

Our priorities in developing a new architecture are rooted in simplicity and reliability, so we focused on adding only necessary, mature CNCF components to the stack, such as Helm, ContainerD, Etcd, and Calico.  Our guiding rule in development was to add only necessary and mature solutions, aiming to maximize reliability and minimize complexity.  For example, to improve reliability the Converged Core uses only local storage volumes, as a result, we don’t require any cloud-native storage add-ons.

Choice 3 – Managing Day-0 Install and Day-N Upgrades

Managing day-0 install / day-n upgrades of NFV architectures can be challenging with multiple integration points into different orchestrators in the MANO stack, resulting in long integration times and a relatively fragile solution.  For the Cisco Converged Core team, a stable cloud-native stack was a critical component, as was automated lifecycle management for all layers – not just the application layer. As a result, Cisco developed a cloud-native cluster management layer that ensures consistent software and tunings across all layers – BIOS settings, firmware, host OS, Kubernetes, and application versions.    This experience is so simple that upgrading the Cisco Converged Core has become a two-step operation – step one, select your new software version and then step two, commit it to the cluster.  To facilitate automation, the cluster management layer provides CLI, REST, and NETCONF interfaces.  Support for a wide range of interfaces enables seamless integration into a mobile service provider’s existing automation solution – such as Cisco’s Network Service Orchestrator (NSO).

Choice 4 – Managing Application Configuration

When developing a solution like the Cisco Converged Core, recognizing when to and when not to use new technology is important. Application configuration management is one of these challenging areas. Traditionally, mobile service providers have managed application configurations using NETCONF/REST or CLI.  With our new Converged Core, we can leverage existing SP interfaces or use cloud-native options like Kubernetes CRD or configuration maps.  Our choice was the status quo because maintaining a traditional management interface would greatly simplify integration into the mobile service provider’s configuration automation solution.

Putting it together

By focusing on simplicity, reliability, and scale, we’ve developed an architecture that enables service providers to manage 100s of Kubernetes clusters across 1000s of servers while serving millions of subscribers.

For More Information

To learn more about the Cisco Converged Core, visit our product pages. To learn more about T-Mobile and Cisco’s Launch of the World’s Largest Cloud Native Converged Core Gateway, read the December 2022 press release.

Share: