Researchers show techniques for malware persistence on F5 and Citrix load balancers

Over the past several years, hackers have targeted public-facing network devices such as routers, VPN concentrators, and load balancers to gain a foothold into corporate networks. While finding remote code execution vulnerabilities in such devices is not uncommon, incidents where attackers were able to deploy malware on them that can survive restarts or firmware upgrades have been rare and generally attributed with sophisticated APT groups.

Because they use flash memory that degrades over time if subjected to many write operations, embedded network devices typically store their firmware in read-only filesystems and load their contents into RAM at each restart. This means that all changes and files generated by the various running services during the device’s normal operation are temporary because they only occur in RAM and are never saved to the file system, which is restored to its initial state when the device is restarted reboot.

The exceptions are configuration files and scripts that are generated through the device administrative interface and are stored in a limited area of storage known as NVRAM (non-volatile RAM). From an attacker’s perspective, this limitation makes compromising networking devices in a persistent way much harder, which is why mass attacks against home routers, for example, involve automated botnets that periodically rescan and reinfect routers that have been restarted.

However, in a targeted attack scenario against enterprise networks, attackers would prefer to remain stealthy and not attack the same device multiple times so they don’t trigger any detections that might be put in place after a vulnerability becomes public. They would also prefer to have long-term access to such devices and use them as bridges into the internal networks, as well as pivot points from where they could perform lateral movement and expand their access to other non-public devices.

Persistence opportunities in Citrix, F5 load balancers

Since 2019, there have been three critical vulnerabilities in Citrix and F5 load balancers (CVE-2019-19781, CVE-2020-5902 and CVE-2022-1388) that have been publicly documented and exploited in the wild, triggering warnings from the US Cybersecurity and Infrastructure Security Agency (CISA) and other organizations. Because of this, researchers from firmware security firm Eclypsium recently investigated the persistence opportunities attackers would have on such devices. Their findings were released in a report Wednesday.

In May 2022, security firm Mandiant reported that a cyberespionage threat actor – identified at the time as UNC3524 but since correlated with the Russian state-run APT29 (Cozy Bear) – compromised enterprise networks and remained undetected for long periods of time due to deploying backdoor implants on network appliances including load balancers that don’t support running detection tools such as endpoint detection and response (EDR) on them and run older versions of CentOS and BSD. While Mandiant didn’t name the appliances or their manufacturers, the Eclypsium researchers believe they were F5 and Citrix appliances, since F5 load balancers run CentOS and Citrix (formerly branded as Netscaler) runs FreeBSD.

“One characteristic of UNC3524 stuck out: Their TTPs were unreliable, they used modified open-source software to establish their backdoor, and seemed to only possess enough understanding of the systems to achieve the most basic of goals,” the Eclypsium researchers said in their report. “Their implants were so unreliable they installed web shells for the sole purpose of restarting them when they died. It was this characteristic that was the catalyst for the research, the unanswered being: Is it possible to use an off-the-shelf C2 framework on a load balancer? Can the malware be resilient enough to persist across reboots and even upgrades? Is it possible to infect the device so deeply that a clean wipe and reinstall isn’t sufficient?”

Many attack groups choose to use cracked versions of commercial attack frameworks such as Cobalt Strike or Brute Ratel, but the Eclypsium researchers wanted something that’s open source and easily available to less sophisticated attackers, so they chose Sliver, an open-source adversary emulation framework, for their test implant. Sliver is written in Go, so it’s cross-platform and provides pivoting and tunneling functionality.

To investigate what files F5 load balancers retain across reboots and firmware upgrades, the researchers looked into the configuration backup functionality available through the administration interface that can be used to generate an archive containing all the configs and settings that can later be deployed on a fresh install. Inside the archive, which included hundreds of files, the researchers settled on three executable scripts and configuration files that can execute scripts on certain events.

“An unexpected discovery during this research was vendor documentation; it proved to be a wealth of information on undocumented features and functionality shoehorned into these devices over the years,” the researchers said. “In credit to the vendors, had it not been for the documentation this research would have been significantly more difficult. It is important to understand how devices handle their configuration files.”

Three ways to store and start malicious scripts

After scouring the documentation and config files, the team now had three different ways to store and start scripts after reboot that would even survive reinstalled because they would be included in the config backups. Storing the 12MB implant directly inside the backup archive would have not been inconspicuous, so the researchers opted to store a script that would later download the implant from the internet, kill any existing versions, and deploy it.

“Opting to download the implant makes the assumption that the device can connect to the internet,” they said. “If the attacker didn’t have this luxury but had a foothold on another system in the network, a smaller implant could be stored inside the config directory structure without alerting the administrators. This implant could instead connect to the ‘jump box’ system under the attacker’s control.”

For further stealthiness, the researchers found that the runsv Linux service on F5 boxes was configured to run a service whose configuration pointed to a binary file called ​​restjavad that didn’t exist on the system. They used this file name for their implant so it doesn’t look suspicious in a process listing. If an administrator would spot the process and would search for the name, they would likely find the F5 documentation for the legitimate ​​restjavad service.

Researching the Citrix system proved a bit more difficult as the documentation was not as detailed. However, inside the user manual they found a note about setting up Network Time Protocol (NTP) synchronization. The instructions involved creating a file called rc.netscaler inside the /nsconfig directory, which does get saved during a backup, and then adding a line to it called /bin/sh /etc/ntpd_ctl full_start. However, the scripts in the /etc/ directory were not set as executable and there was no way to change that. During the process the researchers figured out that the system used a package called Monit to start, stop and monitor the status of system processes and Monit stored its configuration in /nsconfig.

“We ended up writing a wrapper for our implant to run like a service and reused the same logic from the F5 loader,” the researchers said. “From there we simply dropped this file and the modified monit file into /nsconfig and verified the implant would start on boot and that our wrapper would be included in backup files.” Another side effect of using Monit was that it made the implant even more persistent, with Monit automatically restarting the service every few seconds if it was ever manually killed.

The researchers also tested the pivoting ability, which allows attackers to use the compromised device as a proxy to access other devices inside the network that wouldn’t normally have access to the internet. The F5 system allowed binding the implant to a port on one of the IPs on the device and then use ACL to allow access to that port.

“The bar for advanced attackers keeps getting lower and as the imposed cost of attacking hardened systems like servers or workstations gets higher, attackers are turning to more novel ways of infiltrating systems,” the researchers concluded. “Gone are the days of proprietary, purpose-built firmware used by routers & switches, instead replaced with firmware which is a fully functional operating system. This evolution introduces the commodity-server level risk on devices that have historically been out of reach for all but the most skilled attackers.”