Black Hat USA 2022 Continued: Innovation in the NOC

In part one of our Black Hat USA 2022 NOC blog, we discussed building the network with Meraki:

• Adapt and Overcome
• Building the Hacker Summer Camp network, by Evan Basta
• The Cisco Stack’s Potential in Action, by Paul Fidler
• Port Security, by Ryan MacLennan, Ian Redden and Paul Fiddler
• Mapping Meraki Location Data with Python, by Christian Clausen

In this part two, we will discuss:

• Bringing it all together with SecureX
• Creating Custom Meraki Dashboard Tiles for SecureX, by Matt Vander Horst
• Talos Threat Hunting, by Jerzy ‘Yuri’ Kramarz and Michael Kelley
• Unmistaken Identity, by Ben Greenbaum
• 25+ Years of Black Hat (and some DNS stats), by Alejo Calaoagan

Cisco is a Premium Partner of the Black Hat NOC, and is the Official Wired & Wireless Network Equipment, Mobile Device Management, DNS (Domain Name Service) and Malware Analysis Provider of Black Hat.

Black Hat USA is my favorite part of my professional life each year. We had an incredible staff of 20 Cisco engineers to build and secure the network. Also, for the first time, we had two Talos Threat Hunters from the Talos Incident Response (TIR) team, providing unique perspectives and skills to the attacks on the network. I really appreciated the close collaboration with the Palo Alto Networks and NetWitness team members. We created new integrations and the NOC continued to serve as an incubator for innovation.

We must allow real malware on the network for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites. So, context is what really matters when investigating a potential attack and bringing so many technologies together in SecureX really accelerated investigation and response (when needed).

All the Black Hat network traffic was supported by Meraki switches and wireless access points, using the latest Meraki gear donated by Cisco. Our Meraki team was able to block people from the Black Hat network, when an investigation showed they did something in violation of the attendee Code of Conduct, upon review and approval by the Black Hat NOC leadership.

Cisco Secure provided all the domain name service (DNS) requests on the Black Hat network through Umbrella, whenever attendees wanted to connect to a website. If there is a specific DNS attack that threatened the conference, we supported Black Hat in blocking it to protect the network. However, by default, we allow and monitor DNS requests to malware, command and control, phishing, crypto mining, and other dangerous domains, which would be blocked in a production environment. That balance of allowing cybersecurity training and demos to occur, but ready to block when needed.

In addition to the Meraki networking gear, Cisco Secure also shipped an Umbrella DNS virtual appliance to Black Hat USA, for internal network visibility with redundancy. The Intel NUC containing the virtual appliance also contained the bridge to the NetWitness on-premises SIEM, custom developed by Ian Redden.

We also deployed the following cloud-based security software:

We analyzed files that were downloaded on the network, checking them for malicious behavior. When malware is downloaded, we confirm it is for a training, briefing or demonstration, and not the start of an attack on attendees.

During an investigation, we used SecureX to visualize the threat intelligence and related artifacts, correlating data. In the example below, an attacker was attempting remote code execution on the Registration Servers, alerted by the Palo Alto team, investigated by the NOC threat hunters, and blocked by order of the NOC leadership upon the results of the investigation.

Cisco Secure Threat Intelligence (correlated through SecureX)

Donated Partner Threat Intelligence (correlated through SecureX)

Open-Source Threat Intelligence (correlated through SecureX)

Continued Integrations from past Black Hat events

• NetWitness SIEM integration with SecureX
• NetWitness PCAP file carving and submission to Cisco Secure Malware Analytics (formerly Threat Grid) for analysis
• Meraki syslogs into NetWitness SIEM and Palo Alto Firewall
• Umbrella DNS into NetWitness SIEM and Palo Alto Firewall 

New Integrations Created at Black Hat USA 2022

• Secure Malware Analytics integration with Palo Alto Cortex XSOAR, extracting files from the network stream via the firewall

The NOC partners, especially NetWitness and Palo Alto Networks, were so collaborative and we left Vegas with more ideas for future integration development

Creating Custom Meraki Dashboard Tiles for SecureX, by Matt Vander Horst

One of the biggest benefits of Cisco SecureX is its open architecture. Anyone can build integrations for SecureX if they can develop an API with the right endpoints that speak the right language. In the case of SecureX, the language is the Cisco Threat Intelligence Model (CTIM). As mentioned above, Cisco Meraki powered Black Hat USA 2022 by providing wired and wireless networking for the entire conference. This meant a lot of equipment and users to keep track of. To avoid having to switch between two different dashboards in the NOC, we decided to build a SecureX integration that would provide Meraki dashboard tiles directly into our single pane of glass: SecureX.

Building an integration for SecureX is simple: decide what functionality you want your integration to offer, build an internet-accessible API that offers those functions, and then add the integration to SecureX. At Black Hat, our Meraki integration supported two capabilities: health and dashboard. Here’s a summary of those capabilities and the API endpoints they expect:

Capability	Description	API Endpoints
Health	Enables SecureX to make sure the module is reachable and working properly.	/health
Dashboard	Provides a list of available dashboard tiles and, after a tile is added to a dashboard, the tile data itself.	/tiles /tile-data

 

With our capabilities decided, we moved on to building the API for SecureX to talk to. SecureX doesn’t care how you build this API if it has the expected endpoints and speaks the right language. You could build a SecureX-compatible API directly into your product, as a serverless Amazon Web Services (AWS) Lambda, as a Python script with Django, and so on. To enable rapid development at Black Hat, we chose to build our integration API on an existing Ubuntu server in AWS running Apache and PHP.

After building the API framework on our AWS server, we had to decide which dashboard tiles to offer. Here’s what we ended up supporting:

Tile Name	Description
Top Applications	Shows the top 10 applications by flow count
Client Statistics	Shows a summary of clients
Top SSIDs by Usage in GB	Shows the top 10 SSIDs by data usage in GB
Access Point Status	Shows a summary of access points

 

Finally, once the API was up and running, we could add the integration to SecureX. To do this, you need to create a module definition and then push it to SecureX using its IROH-INT API. After the module is created, it appears in the Available Integration Modules section of SecureX and can be added. Here’s what our module looked like after being added to the Black Hat SecureX instance:

After adding our new tiles to the SecureX dashboard, SecureX would ask our API for data. The API we built would fetch the data from Meraki’s APIs, format the data from Meraki for SecureX, and then return the formatted data. Here’s the result:

These dashboard tiles gave us useful insights into what was going on in the Meraki network environment alongside our existing dashboard tiles for other products such as Cisco Secure Endpoint, Cisco Umbrella, Cisco Secure Malware Analytics, and so on.

If you want to learn more about building integrations with SecureX, check out these resources:

Talos Threat Hunting, by Jerzy ‘Yuri’ Kramarz and Michael Kelly

Black Hat USA 2022 was our first fully supported event, where we deployed an onsite threat hunting team from Talos Incident Response (TIR). Our colleagues and friends from various business units, connected by SecureX integration, granted us access to all the underlying consoles and API points to support the threat hunting efforts enhanced by Talos Intelligence.

The threat hunting team focused on answering three key hypothesis-driven questions and matched that with data modelling across all of the different technology stacks deployed in Black Hat NOC:

• Are there any attendees attempting to breach each other’s systems in or outside of a classroom environment?
• Are there any attendees attempting to subvert any NOC Systems?
• Are there any attendees that are compromised and we could warn them about that?

To answer the above hypothesis, our analysis started with understanding of how the network architecture is laid out and what kind of data access is granted to NOC. We quickly realized that our critical partners are key to extending visibility beyond Cisco deployed technologies. Great many thanks go to our friends from NetWitness and Palo Alto Networks for sharing full access to their technologies, to ensure that hunting did not stop on just Cisco kit and contextual intelligence could be gathered across different security products.

Daily threat hunt started with gathering data from Meraki API to identify IP and DNS level requests leaving the devices connected to wireless access points across entire conference. Although Meraki does not directly filter the traffic, we wanted to find signs of malicious activity such as DNS exfiltration attempts or connections to known and malicious domains which were not part of the class teaching. Given the level of access, we were then able to investigate network traffic capture associated with suspicious connections and check for suspected Command and Control (C2) points (there were a few from different threat actors!) or attempts to connect back to malicious DNS or Fast Flux domains which indicated that some of the attendee devices were indeed compromised with malware.

That said, this is to be expected given hostility of the network we were researching and the fact that classroom environments have users who can bring their own devices for hands-on labs. SecureX allowed us to quickly plot this internally to find specific hosts which were connecting and talking with malicious endpoints while also showing a number of additional datapoints which were useful for the investigation and hunting. Below is one such investigation, using SecureX threat response.

While looking at internal traffic, we have also found and plotted quite a few different port-scans running across the internal network. While not stopping these, it was interesting to see different tries and attempts by students to find ports and devices across networks. Good thing that network isolation was in place to prevent that! We blurred out the IP and MAC addresses in the image below.

Here is another example of really nice port scan clusters that were running across both internal and external networks we have found. This time it was the case of multiple hosts scanning each other and looking to discovery ports locally and across many of the Internet-based systems. All of that was part of the class but we had to verify that as it looked quite suspicious from the outset. Again, blurred picture for anonymity.

In a few instances, we also identified remarkably interesting clear-text LDAP traffic leaving the environment and giving a clear indicator of which organization the specific device belonged to simply because of the domain name which was requested in the cleartext. It was quite interesting to see that in 2022, we still have a lot of devices talking clear text protocols such as POP3, LDAP, HTTP or FTP, which are easy to subvert via Man-In-The-Middle type of attacks and can easily disclose the content of important messages such as email or server credentials. Below is an example of the plain text email attachments, visible in NetWitness and Cisco Secure Malware Analytics.

In terms of the external attacks, Log4J exploitation attempts were pretty much a daily occurrence on the infrastructure and applications used for attendee registration along with other typical web-based attacks such as SQL injections or path traversals. Overall, we saw a good number of port scans, floods, probes and all kind of web application exploitation attempts showing up daily, at various peak hours. Fortunately, all of them were successfully identified for context (is this part of a training class or demonstration) and contained (if appropriate) before causing any harm to external systems. Given the fact that we could intercept boundary traffic and investigate specific PCAP dumps, we used all these attacks to identify various command-and-control servers for which we also hunted internally to ensure that no internal system is compromised.

The final piece of the puzzle we looked to address, while threat hunting during Black Hat 2022, was automation to discover interesting investigation avenues. Both of us investigated a possibility of threat hunting using Jupyter playbooks to find outliers that warrant a closer look. We have created and developed a set of scripts which would gather the data from API endpoints and create a data frames which could be modeled for further analysis. This allowed us to quickly gather and filter out systems and connections which were not that interesting. Then, focus on specific hosts we should be checking across different technology stacks such as NetWitness and Palo Alto.

Unmistaken Identity, by Ben Greenbaum

An unusual aspect of the Black Hat NOC and associated security operations activities is that this is an intentionally hostile network. People come to learn new tricks and to conduct what would in any other circumstance be viewed rightfully as malicious, unwanted behavior. So, determining whether this is “acceptable” or “unacceptable” malicious behavior is an added step. Additionally, this is a heavily BYOD environment and while we do not want attendees attacking each other, or our infrastructure, there is a certain amount of suspicious or indicative behavior we may need to overlook to focus on higher priority alerts.

In short, there are broadly speaking 3 levels of security event at Black Hat:

• Allowed – classroom or demonstration activities; i.e. a large part of the purpose of Black Hat
• Tolerated –C&C communications from BYOD systems, other evidence of infections that are not evidence of direct attacks; attendee cleartext communications that should be encrypted, but are not relevant to the operation of the conference.
• Forbidden – direct attacks on attendees, instructors, or infrastructure; overt criminal activity, or other violations of the Code of Conduct

When Umbrella alerted us (via a SecureX orchestration Webex workflow) of DNS requests for a domain involved in “Illegal Activity” it was reminiscent of an event at a previous conference where an attendee was caught using the conference network to download forged vaccination documents.

Using the Cisco Secure Malware Analytics platform’s phishing investigation tools, I loaded and explored the subject domain and found it to be a tool that generates and provides pseudo-randomized fake identities, customizable in various ways to match on demographics. Certainly, something that could be used for nefarious purposes, but is not illegal in and of itself. Physical security and access control is, however, also important at Black Hat, and if this activity was part of an effort to undermine that, then this was still a concern.

This is, however, also the kind of thing that gets taught at Black Hat…

Using the reported internal host IP from Umbrella, Meraki’s connection records, and the Meraki access point map, we were able to narrow the activity down to a specific classroom. Looking up what was being taught in that room, we were able to confirm that the activity was related to the course’s subject matter

Network owners and administrators, especially businesses, typically don’t want their network to be used for crimes. However, here at Black Hat what some would consider “crimes” is just “the curriculum”. This adds a layer of complexity to securing and protecting not just Black Hat, but also Black Hat attendees. In security operations, not every investigation leads to a smoking gun. At Black Hat, even when it does, you may find that the smoking gun was fired in a safe manner at an approved target range. Having the right tools on hand can help you make these determinations quickly and free you up to investigate the next potential threat.

25 Years of Black Hat – Musings from the show (and some DNS stats), by Alejo Calaoagan

Back in Singapore, I wrote about cloud app usage and the potential threat landscape surrounding them.  My original plan at Black Hat USA was to dig deeper into this vector to see what interesting tidbits I could find on our attendee network. However, given that this was the 25th anniversary of Black Hat (and my 14th in total between Vegas, Singapore, and London), I’ve decided to pivot to talk about the show itself.

I think it’s safe to say, after two difficult pandemic years, Black Hat is back. Maybe it’s the fact that almost everyone has caught COVID by now (or that a lot of people just stopped caring). I caught it myself at RSA this year back in June, the first of consecutive summer super spread events (Cisco Live Vegas was the following week). Both of those shows were in the 15-18k attendee range, well below their pre-pandemic numbers. Black Hat USA 2022 was estimated at 27,000 attendees.

If I remember correctly, 2019 was in the 25-30K range. Last year in Vegas, there were ~3,000 people at the event, tops. 2021 in London, was even lower…it felt like there were less than 1,000 attendees. Things certainly picked up in Singapore (2-3k attendees), though that event doesn’t typically see attendee numbers as high as the other locations. All in all, while the pandemic certainly isn’t over, Las Vegas gave glimpses of what things were like before the “Rona” took over our lives.

The show floor was certainly back to the norm, with swag flying off the countertops and lines for Nike sneaker and Lego giveaways wrapping around different booths.  The smiles on people’s faces as they pitched, sold, hustled, and educated the masses reminded me how much I missed this level of engagement.  RSA gave me this feeling as well, before COVID sidelined me midway through the show anyway.

Not everything was quite the same. The Black Hat party scene certainly is not what it used to be. There was no Rapid 7 rager this year or last, or a happy hour event thrown by a security company you’ve never heard of at every bar you walk by on the strip. There were still some good networking events here and there, and there were some awesomely random Vanilla Ice, Sugar Ray, and Smashmouth shows. For those of you familiar with Jeremiah Grossman’s annual Black Hat BJJ throwdown, that’s still, thankfully, a thing. Hopefully, in the coming years, some of that old awesomeness returns….

Enough reminiscing, here are our DNS numbers from the show:

From a sheer traffic perspective, this was the busiest Black Hat ever, with over 50 million DNS requests made…

Digging into these numbers, Umbrella observed over 1.3 million security events, including various types of malware across the attendee network. Our threat hunting team was busy all week!

We’ve also seen an increase in app usage at Black Hat:

• 2019: ~3,600
• 2021: ~2,600
• 2022: ~6,300

In a real-world production environment, Umbrella can block unapproved or high-risk apps via DNS.

The increases in DNS traffic volume and Cloud App usage obviously mirrors Black Hat’s return to the center stage of security conferences, following two years of pandemic uncertainty. I’m hopeful that things will continue to trend in a positive direction leading up to London and, hopefully, we’ll see you all there.

——

Hats off to the entire NOC team. Check out Black Hat Europe in London, 5-8 December 2022!

Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.

SecureX threat response, orchestration, device insights, custom integrations and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor

Umbrella DNS: Christian Clasen and Alejo Calaoagan

Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley

Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter

Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey

Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira

Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz

Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann

Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).

About Black Hat

For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.

---

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn

Share: