Mitigating Alert Fatigue in SecOps Teams

Security Operations Teams (SOCs) today are under attack by the very mechanisms meant to help them. A recent industry study revealed a few startling facts:

• SOCs spend a third of their workday hunting down false positives.
• Even then, SOCs only get to half of the alerts they need to every day.
• Out of all the teams currently using automation, only half apply it to threat hunting and incident enrichment.

And unsurprisingly, the vast majority (80%) say that manual processes are slowing them down. These stats lay out both the problem with and solution to alert fatigue today: too many alerts, too many bad ones, and not enough streamlined processes helping SOCs get ahead of the problem.

This blog will review a few strategies for reducing the burden of excessive alerts on security operations teams, including the use of automation techniques and alert tuning.

Alert fatigue: Too much for humans alone

The need for automation and force-multiplying technologies is widely agreed upon as current threats outstrip security teams’ capabilities to deal with them. We’ve arrived at a point where it’s unreasonable for all security operations to be entirely human-powered. IT infrastructure is sprawling with the prevalence of SaaS apps and public cloud usage, ransomware is being deployed within days, not weeks, of vulnerability exploitation, and security devices continue to expand their telemetry – and, as a side effect, inundate analysts with loads of alert data.

When analysts spend their valuable time (and education) chasing down worthless pursuits for over a third of their day:

• Productivity can drop
• SOC members can feel their jobs are in a rut
• Companies could face losing well-qualified cybersecurity professionals as experts look for greener pastures

Plus, finding more effective ways to reduce alerts is key to not only improving the efficiency of security operations, but ensuring talent retention at a time when there are plenty of competitive cybersecurity careers out there.

And let’s not forget one of the most obvious consequences: an unprotected enterprise. When SOCs have to pick and choose which alerts they address in a day (not even knowing, sometimes, which ones are the most important), that leaves a massive blind spot. What if the alert that notifies you of tomorrow’s impending data breach was the one you chose not to get to today? The potential for disaster is endless.

Strategies for fighting alert fatigue

So, how to stop the hemmorraging? In order to keep alerts from being forgotten, overlooked, ignored, or simply lost, they need to be properly organized. This is a job beyond humans alone. Here are a few force-multiplying strategies for shifting the alert burden to capable technologies and saving your valuable experts for where they are needed the most.

Automation & LLMs

While many tools and services provide automated threat detection today (hence so many alerts), automation can be applied differently to yield more specified results. Most of today’s tools can map to MITRE ATT&CK tactics, but sometimes you need more. Large Language Models (LLMs), for instance, can be used in building rule content when given a guided prompt. Just like using a Generative AI model like ChatGPT, the more customized the prompt, the narrower and more specific the results, leading to less overwhelm and more focused alerts.

AI-based technologies

The use of AI and ML can be another way to cut alert fatigue. In an XDR solution, for example, AI-infused automation can be used to run through playbook plays in route investigations, weeding out potential false positives and leaving the rest to your team.

However, these powerful finding capabilities can also contribute to the probelm as there may be too many anomalies to sift through once the enterprise has been scoured. The solution to this is to better prioritize the alerts you do have, and alert tuning can help with that.

Alert Tuning

In order to prioritize critical alerts, your SOC needs to arrange its alert thresholds to be sensitive enough to cut out the noise. While this may seem like an art, alert tuning does not have to be one big game of guess-and-check. In fact, it shouldn’t be.

• Base your adjustments on historical data, not “gut feel.” How many times did this type of alert result in an actual threat versus a false positive? Adjust accordingly.
• Have strict metrics by which alerts do or do not make it to production. Have these rules do the work for you so you’re not bending your brain on a new decision every time (because decision fatigue is a risk, too).
• Focus your tuning efforts on weeding out alerts that rarely yield True Positives. By cutting out these energy sandpits one by one, your team will progressively have more time to snowball into what matters.

Once you’ve rounded up your alert types and have begun investigating their efficacy, here are a few questions that can help speed the process:

1. Can this detection type alone identify a threat, or is it flagging it later in the kill chain (when another alert is flagging it sooner)? You don’t want double alerts.
2. Are your rules streamlined? “Runaway” alerts can be triggered by vague or too-far-reaching rules that cause needless notifications.
3. Has this alert ever yielded a True Positive? While the possibility is always there that it could, it might be more beneficial on your time to start by focusing on ones that have proven success, then expanding out from there.

SecOps today: Less is more – and keep humans in the loop

We’re certainly living in the information age, but too much of it is ironically dragging SOCs under and undermining our ability to defend ourselves. While a few years ago, the discussion might have hinged on “volume and real time,” the dialogue today now centers around figuring out which alerts are important and which ones to weed out. To paraphrase a line from the Pixar movie The Incredibles, “When every[thing] is special, no[thing] is.” However, much of the weeding-out process requires human decision-making.

To this point, Ali Hader, an internationally recognized cybersecurity architect and advocate, stated: “Automated systems are designed to augment, rather than replace, human analysts. Therefore, human oversight is essential for interpreting alerts, investigating potential threats, and making informed decisions.”

By leveraging automation, LLMs, AI and ML investigative capabilities, and savvy alert tuning, individual SOC members can supplement technology and increase the chances that the most important things come to light. 

Editor’s Note: The opinions expressed in this and other guest author articles are solely those of the contributor and do not necessarily reflect those of Tripwire.