Detection Engineering in Post SIEM and SOAR World

A few years back, my security team was tasked to create and maintain a green field environment for FEDRAMP compliance. We made a radical decision, we opted to forego a Security Information and Event Management (SIEM) system entirely. This decision was not made lightly, but it was driven by two primary considerations. First, we wanted to eliminate any instance that required patching, and second, there was no off-the-shelf SIEM solution that was FEDRAMP compliant.

Navigating SIEM-less Security and Compliance

In the absence of a SIEM, we had to navigate through the myriad of controls that a typical SIEM would cover. This included:

• Logging Controls: Every control mandates extensive logging.
• Alerting Controls: Particularly under Audit (AU) and System Integrity (SI) families.
• Correlation Controls: Necessary for detecting patterns and anomalies that many control families
• Threat Intelligence / Malware controls: SC related controls specifically related to malware.

We designed the architecture where subdivided the functionality that SIEM traditionally provided and distributed those functionalities to various components. Primarily –

• We leveraged AWS S3 (Amazon Simple Storage Service) for storage and
• We deployed numerous Lambda functions to funnel logs from various non-AWS-native applications into AWS S3.
• Several compute jobs to correlate logs from multiple sources

The effort is super successful. We managed the environment with barely one full time person, we come out of yearly audits in full colors, through multiple years of audits. Our success was largely due to the role security played in shaping the entire organization’s practices.

We collaborated with all departments to ensure the following –

• We were able to collaborate with all the application teams and get all the logs in specific standard formats we needed them to
• We were to work with devops teams and make the environment managed and operated only through CI/CD pipelines, the DevOps team delivered. This seamless integration made correlating production alerts to approved change tickets effortless.

Challenges Outside FEDRAMP Environments

However, replicating this architecture outside FEDRAMP environments proved challenging. I pondered about the reasons over multiple months as we encountered the hurdles. The reasons for the challenges are the same challenges that led the rise of SIEM and SOAR technologies in mainstream security engineering.

• Log Format Control: It is impractical to enforce a uniform log format across diverse sources, and so SIEM became a dump of all logs and a place to bring uniformity into all logs
• Rule Development: Writing rules against the various vendor-specific log formats at source is cumbersome, so learning a single SIEM DSL became preferable.
• Centralized Intelligence: Since logs are in a central place, SIEM systems evolved to provide intelligent log dumps, abstracting rule development and offering comprehensive dashboard capabilities.
• Reach into the sources with API and conduct actions (SOAR): As the SIEM management complexity grew, SOAR systems emerged to automate the response actions, checking sources, and conducting interventions.

Rethinking Detection with GenAI

Let’s flip the scenario. Imagine a world where all event and state data from internal and external sources neatly organized in well-structured JSON store. You have access to an unlimited, cost-effective log store that can organize these JSONs by various attributes like date and service. Correlating them becomes a manageable problem.

Let’s look at the three major technical problems that are acting as hurdles for the dream

• Normalization of Events: Converting unstructured log data from each of the sources into JSONS of predefined, well-structured schemas.
• Abstraction of Events: Picking out relevant logs to query based on a procedure in MITRE TTP parlance for understanding the detection coverage
• Writing rules for new attack patterns: Writing rules as new threats arise, intelligently picking the right set of procedures and corresponding logs

GenAI presents solid solutions to the key problems above.

Log normalization into structured schemas.

LLMs front ending some of the unstructured log content and making structured JSON to store into a JSON store

Correlating logs and identifying patterns.

Callable functions based on the schema of each of the micro structured logs and has the description that describes the capabilities (essentially each function can be thought as a procedure finder in Mitre attack TTP parlance)

Understanding threats and executing precise queries to detect emerging threats

An LLM agent that comprehends a described threat, makes queries various stores with specific schemas, and identifies emerging threats.

29

The above approach not only will make detection engineering possible without SIEM and SOAR but would make security engineers focus on actual development of code to detect new and emerging procedures as opposed to learn how to work with SIEM and write SOAR playbooks.

About the Author

Venkat Pothamsetty is the CTO of Network Intelligence. Venkat took companies through their first 100K, first 1M, first 10M and 100M years to successful exits as products and security leader. With extensive leadership experience in product, engineering, pre-sales, and services teams at both Fortune 100 companies and startups, Venkat consistently hires and leads high-energy teams to deliver successful products in both SaaS and on-premise worlds. Notably, all of the startups that Venkat worked with went through successful liquidity events while he served as product head.