Zeroing in on network performance issues, with ThousandEyes – Cisco Blogs

In collaboration with Charlie Radke and Brian Hollenbeck

Part 2 of a 2-part series

In part one of this series, we explained how we’re troubleshooting teleworkers’ performance issues using ThousandEyes Endpoint Agent. This blog continues the story, describing how we monitor network performance from branch offices all the way to our cloud providers’ networks, using ThousandEyes Enterprise Agent on Catalyst 9000 switches.

Imagine it’s Monday morning, and trouble tickets are pouring in about a sluggish cloud application. Until recently, we in Cisco IT could only see performance on our own network—from the branch to the WAN hub. The internet circuit and cloud provider network were like black holes. We were at the mercy of the circuit provider and cloud provider to identify and resolve issues. Sometimes it took weeks. Meanwhile, productivity took a dip and those trouble tickets kept coming.

Hop-by-hop performance stats at our fingertips

Now we can instantly see what’s happening at every hop on the path to our cloud providers. The difference is ThousandEyes Enterprise Agent, installed on our branch office Catalyst 9000 switches. We use it to test the connection to software as a service (SaaS) providers as well as public clouds where we host other applications. (Just now I ran an HTTP test and saw that cisco.sharepoint.com loads in 1.6 seconds.) If we’re getting complaints about an application hosted on AWS, for example, we can pinpoint whether the problem is happening on our network, the circuit provider network, or the AWS network. Talking to the right people sooner means faster resolution.

Use case 1: troubleshooting SD-WAN performance

After we turned on split tunneling in our branch office in Irvine, California, we started seeing Office 365 tickets. We suspected problems with the service provider circuit. Before, we couldn’t have proved our suspicion, so the service provider would have wanted us to rule out everything else first. That takes weeks. By running a test with ThousandEyes, within minutes we had the stats to show high packet loss and latency on the circuit from Irvine to the San Jose hub. We shared a link to the data with the service provider. Handed indisputable evidence, the service provider quickly resolved the issue.

Use case 2: measuring the impact of changes

When we started sending web traffic through Cisco Umbrella to block malicious domains and cloud applications, we used ThousandEyes to measure Umbrella’s impact. We ran the transaction test, which emulates a user doing a series of steps. We created a script to open a browser, enter a term in a search engine, and then open one of the search results. Then we scheduled the script to run every 5 minutes for 24 hours. From the tests we could see that Umbrella didn’t noticeably slow the user experience, giving us the confidence to roll it out to the workforce. It also removed the element of, “well, we’ll see if anybody complains in the morning” from changes we make remotely.

Deployment decisions

• Where. We decided to deploy ThousandEyes on branch office switches instead of our WAN routers or hubs because we wanted it as close to users as possible. If we had deployed it on WAN routers or hubs, we wouldn’t see performance for the first hop from the branch.
• How. We used Cisco DNA Center to deploy the agent as a docker container on our switches. It was as simple as selecting the switches and clicking a button, everything can be configured from within Cisco DNA Centre, even our account token. And to make it even better, no downtime.

Next steps

We’re starting to deploy ThousandEyes in branches outside the U.S. We’re also planning to deploy it at the hub, to better understand the application and user experience. Aggregating data from multiple branches will give us new insights. For example, it’s usually assumed that sending branch SaaS traffic directly over the branch’s direct internet link is more efficient than backhauling it to a data center. That might not be true in our case because our data centers have direct peering connections to major ISPs. With ThousandEyes deployed on the hub, we’ll be able to compare.

Learn more about our journey to an advanced network
architecture by clicking through our interactive journey map

Follow Cisco IT on social!

Twitter
Facebook
YouTube

Learn more

Part 1 of 2 in series: Employees troubleshooting with ThousandEyes

Cisco ThousandEyes Activation Guide

ThousandEyes webpage

ThousandEyes + SD-WAN strategy

Share: