Black Hat USA 2022: Creating Hacker Summer Camp

In part one of this issue of our Black Hat USA NOC (Network Operations Center) blog, you will find:

• Adapt and Overcome
• Building the Hacker Summer Camp network, by Evan Basta
• The Cisco Stack’s Potential in Action, by Paul Fidler
• Port Security, by Ryan MacLennan, Ian Redden and Paul Fiddler
• Mapping Meraki Location Data with Python, by Christian Clausen

Adapt and Overcome, by Jessica Bair Oppenheimer

In technology, we plan as best as we can, execute tactically with the resources and knowledge we have at the time, focus on the strategic mission, adjust as the circumstances require, collaborate, and improve; with transparency and humility. In short, we adapt and we overcome. This is the only way a community can have trust and grow, together. Every deployment comes with its challenges and Black Hat USA  2022 was no exception. Looking at the three Ps (people, process, platform), flexibility, communication, and an awesome Cisco platform allowed us to build and roll with the changes and challenges in the network. I am proud of the Cisco Meraki and Secure team members and our NOC partners.

The Buck Stops Here. Full stop. I heard a comment that the Wi-Fi service in the Expo Hall was “the worst I’ve ever experienced at a conference.” There were a lot of complaints about the Black Hat USA 2022 Wi-Fi network in the Expo Hall on 10 August. I also heard a lot of compliments about the network. Despite that the Wi-Fi and wired network was generally very good the most of the conference, and before my awesome colleagues share the many successes of designing, building, securing, managing, automating and tearing down one of the most hostile networks on Earth; I want to address where and how we adapted and what we did to fix the issues that arose, as we built an evolving, enterprise class network in a week.

First, a little history of how Cisco came to be the Official Network Provider of Black Hat USA 2022, after we were already successfully serving as the Official Mobile Device Management, Malware Analysis and Domain Name Service Provider. An Official Provider, as a Premium Partner, is not a sponsorship and no company can buy their way into the NOC for any amount of money. From the beginning of Black Hat 25 years ago, volunteers built the network for the conference rather than using the hotel network. This continues today, with the staff of Black Hat hand selecting trusted partners to build and secure the network.

After stepping up to help Black Hat with the network at Black Hat Asia, we had only two and a half months until Black Hat USA, in Las Vegas, 6-11 August 2022. Cisco was invited to build and secure the network for the much larger Black Hat USA flagship conference, affectionally known as ‘Hacker Summer Camp’, as the Official Network Equipment Provider. There were few other options, given the short timeframe to plan, supply chain difficulties in procuring the networking gear and assembling a team of network engineers, to join the Cisco Secure engineers and threat hunters. All the work, effort and loaned equipment were a gift from Cisco Meraki and Cisco Secure to the community.

We were proud to collaborate with NOC partners Gigamon, IronNet, Lumen, NetWitness and Palo Alto Networks; and work with Neil ‘Grifter’ Wyler, Bart Stump, Steve Fink and James Pope of Black Hat. We built strong bonds of familial ties over the years of challenges and joint successes. I encourage you to watch the replay of the Black Hat session An Inside Look at Defending the Black Hat Network with Bart and Grifter.

In June 2022, adjacent to Cisco Live Americas, the NOC partners met with Black Hat to plan the network. Cisco Meraki already donated 45 access points (APs), seven MS switches, and two Meraki MX security and SD-WAN appliances to Black Hat, for regional conferences.

I looked at the equipment list from 2019, that was documented in the Bart and Grifter presentation, and estimated we needed to source an additional 150 Cisco Meraki MR AP (with brackets and tripods) and 70+ Cisco Meraki MS switches to build the Black Hat USA network in just a few weeks. I wanted to be prepared for any changes or new requirements on-site. We turned to JW McIntire, who leads the network operations for Cisco Live and Cisco Impact. JW was enthusiastically supportive in helping identify the equipment within the Cisco Global Events inventory and giving his approval to utilize the equipment. A full thanks to those who made this possible is in the Acknowledgements below.

Over the week-long conference, we used all but three of the switches and all the APs.

We worked off the draft floor plans from 13 June 2022, for the training rooms, briefing rooms, support rooms, keynote rooms, conference public areas, registration, and of course the Expo Hall: over two million square feet of venue. We received updated plans for the training rooms, Expo Hall and support needs 12 days before we arrived on site. There were about 60 training rooms planned, each requiring their own SSID and Virtual Local Area Network, without host isolation. The ‘most access possible’ was the requirement, to use real world malware and attacks, without attacking other classrooms, attendees, sponsors or the rest of the world. Many of the training rooms changed again nine days before the start of the network build, as the number confirmed students rose or fell, we adjusted the AP assignments.

For switching allocation, we could not plan until we arrived onsite, to assess the conference needs and the placement of the cables in the walls of the conference center. The Black Hat USA network requires that every switch be replaced, so we always have full control of the network. Every network drop to place an AP and put the other end of a cable into the new switches in the closets costs Black Hat a lot of money. It also requires the time of ‘Doc’ – the lead network engineer at the Mandalay Bay, to whom we are all deeply grateful.

The most important mission of the NOC is Access, then Security, Visibility, Automation, etc. People pay thousands of dollars to attend the trainings and the briefings; and sponsors pay tens of thousands for their booth space. They need Access to have a successful conference experience.

With that background, let’s discuss the Wi-Fi in the Expo Hall. Cisco has a service to help customers do a methodical predictive survey of their space for the best allocation of their resources. We had 74 of the modern MR57 APs for the conference and prioritized their assignment in the Expo Hall and Registration. Specifications for MR57s include a 6 GHz 4×4:4, 5 GHz 4×4:4 and 2.4 GHz 4×4:4 radio to offer a combined tri–radio aggregate frame rate of 8.35 Gbps, with up to 4,804 Mbps in 6GHz band, 2,402 Mbps 5 GHz band and, 1,147 Mbps / 574 Mbps in the 2.4 GHz band based on 40MHz / 20MHz configuration. Technologies like transmit beamforming and enhanced receive sensitivity allow the MR57 to support a higher client density than typical enterprise-class access points, resulting in better performance for more clients, from each AP.

We donated top of the line gear for use at Black Hat USA. So, what went wrong on the first day in the Expo Hall? The survey came back with the following map and suggestions of 34 MR57s in the locations below. Many assumptions were made in pre-planning, since we did not know the shapes, sizes and materials of the booths that would be present inside the allocated spaces. We added an AP in the Arsenal Lab on the far-left side, after discussing the needs with Black Hat NOC leadership.

In the Entrance area (Bayside Foyer) of the Expo Hall (bottom of the map), you can see that coverage drops. There were four MR57s placed in the Bayside Foyer for iPad Registration and attendee Wi-Fi, so they could access their emails and obtain their QR code for scanning and badge printing.

I believed that would be sufficient and we allocated other APs to the rest of the conference areas. We had positive reports on coverage in most areas of the rest of the conference. When there were reported issues, we quickly deployed Cisco Meraki engineers or NOC technical associates. to confirm and were able to make changes in radio strength, broadcasting bands, SSIDs, etc. to fine tune the network. All while managing a large amount of new or changing network requirements, as the show expanded due to its success and was fully hybrid, with the increased streaming of the sponsored sessions, briefings and keynotes and remote Registration areas in hotels.

As the attendees queued up in mass outside of the Expo Hall on the morning of 10 August, the number of attendee devices connecting to the four MR57s in the foyer grew into the thousands. This degraded the performance of the Registration network. We adjusted by making the APs closest to the registration iPads only dedicated to the Registration. This fixed Registration lag but reduced the performance of the network for the attendees, as they waited to rush into the Expo Hall. From the site survey map, it is clear that the replacement APs were now needed in the Entrance for a connected mesh network, as you entered the Expo Hall from the Bayside foyer. Here lies Lesson 1: expected people flow should be taken into account in the RF design process.

Another challenge the morning of the Expo Hall opening was that five of the 57MRs inside were not yet connected to the Internet when it opened at 10am. The APs were installed three days earlier, then placed up on tripods the afternoon prior. However, the volume of newly requested network additions, to support the expanded hybrid element required the deployment of extra cables and switches. This cascaded down and delayed the conference center team from finalizing the Expo Hall line drops until into the afternoon. Lesson 2: Layer 1 is still king; without it, no Wi-Fi or power.

A major concern for the sponsors in their booths was that as the Expo Hall filled with excited attendees, the connectivity of the 900+ iOS devices used for lead management dropped. Part of this congestion was thousands of 2.4Ghz devices connected to the Expo Hall network. We monitored this and pushed as many as possible to 5Ghz, to relieve pressure on those airwaves. Lesson 3: With Wi-Fi 6e now available in certain countries, clean spectrum awaits, but our devices need to come along as well.

We also adjusted in the Cisco Meraki Systems Manager Mobile Device Management, to allow the iPhones for scanning to connect securely to the Mandalay Bay conference network, while still protecting your personal information with Cisco SecureX, Security Connector and Umbrella DNS, to ensure access as we expanded the network capacity in the Expo Hall. Lesson 4: Extreme security by default where you can control the end point. Do not compromise when dealing with PPI.

Using the Cisco Meraki dashboard access point location heat map and the health status of the network, we identified three places in the front of the Expo Hall to deploy additional drops with the Mandalay Bay network team. Since adding network drops takes some time (and costs Black Hat extra money), we took immediate steps to deploy more MS120 switches and eight additional APs at hot spots inside the Expo Hall with the densest client traffic, at no expense to Black Hat. Lesson 5: Footfall is not only about sales analytics. It does play a role into RF planning. Thereby, allowing for a data-driven design decision.

Above is the heat map of the conference Expo Hall at noon on 12 August. You can see the extra APs at the Entrance of the Expo Hall, connected by the three drops set up by the Mandalay Bay to the Cisco Meraki switches in the closets. Also, you can see the clusters of APs connected to the extra MS120 switches. At the same time, our lead Meraki engineer, Evan Basta, did a speed test from the center left of the Expo Hall.

As I am sharing lessons learned, I want to provide visibility to another situation encountered. On the afternoon of 9 August, the last day of training, a Black Hat attendee walked the hallways outside several training rooms and deliberately attacked the network, causing students and instructors not to be able to connect to their classes. The training rooms have host isolation removed and we designed the network to provide as much safe access as possible. The attacker took advantage of this openness, spoofed the SSIDs of the many training rooms and launched malicious attacks against the network.

We must allow real malware on the network for training, demonstrations and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors from using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities and malicious websites.

The attack vector was identified by a joint investigation of the NOC teams, initiated by the Cisco Meraki Air Marshal review. Note the exact same MAC addresses of the spoofed SSIDs and malicious broadcasts. A network protection measure was suggested by the Cisco Meraki engineering team to the NOC leadership. Permission was granted to test on one classroom, to confirm it stopped the attack, while not also disrupting the training. Lesson 6: The network-as-a-sensor will help mitigate issues but will not fix the human element.

Once confirmed, the measure was implemented network wide to return resiliency and access. The NOC team continued the investigation on the spoofed MAC addresses, using syslogs, firewall logs, etc. and identified the likely app and device used. An automated security alerting workflow was put in place to quickly identify if the attacker resumed/returned, so physical security could also intervene to revoke the badge and eject the attacker from the conference for violation of the Black Hat code of conduct.

I am grateful to the 20+ Cisco engineers, plus Talos Threat Hunters, deployed to the Mandalay Bay Convention Center, from the United States, Canada, Qatar and United Kingdom who made the Cisco contributions to the Black Hat USA 2022 NOC possible. I hope you will read on, to learn more lessons learned about the network and the part two blog about Cisco Secure in the NOC

Building the Hacker Summer Camp Network, by Evan Basta

It was the challenge of my career to take on the role of the lead network engineer for Black Hat USA. The lead engineer, who I replaced, was unable to travel from Singapore, just notifying us two weeks before we were scheduled to deploy to Las Vegas.

We prepared as much as possible before arrival, using the floor plans and the inventory of equipment that was ordered and on its way from the warehouse. We met with the Black Hat NOC leadership, partners and Mandalay Bay network engineers weekly on conference calls, adjusted what we could and then went to Black Hat, ready for a rapidly changing environment.

Our team was able to remain flexible and meet all the Black Hat requests that came in, thanks to the ability of the Cisco Meraki dashboard to manage the APs and switches from the cloud. Often, we were configuring the AP or switch as it was being transported to the location of the new network segment, laptop in hand.

For the construction of the Black Hat network, let’s start with availability. Registration and training rooms had priority for connectivity. iPads and iPhones needed secure connectivity to scan QR codes of registering attendees. Badge printers needed hardline access to the registration system. Training rooms all needed their separate wireless networks, for a safe sandbox for network defense and attack. Thousands of attendees attended, ready to download and upload terabytes of data through the main conference wireless network. All the keynotes, briefings and sponsored sessions needed to be recorded and streamed. Below are all the APs stacked up for assignment, including those assigned to the Expo Hall in the foreground.

All this connectivity was provided by Cisco Meraki access points and switches along with integrations into SecureX, Umbrella, and other Cisco platforms. We fielded a literal army of engineers to stand up the network in six days.

Let’s talk security and visibility. For a few days, the Black Hat network is one of the most hostile in the world. Attendees learn new exploits, download new tools, and are encouraged to test them out. Being able to drill down on attendee connection details and traffic was instrumental in ensuring attendees followed the Black Hat code of conduct.

On the wireless front, we made extensive use of our Radio Profiles to reduce interference by tuning power and channel settings. We enabled band steering to get more clients on the 5GHz bands versus 2.4GHz and watched the Location Heatmap like a hawk looking for hotspots and dead areas. Handling the barrage of wireless change requests – enable or disabling this SSID, moving VLANs (Virtual Local Area Networks), enabling tunneling for host isolation on the general conference Wi-Fi, mitigating attacks – was a snap with the Cisco Meraki Dashboard.

Floor Plan and Location Heatmap

On the first day of NOC setup, the Cisco team worked with the Mandalay Bay networking engineers to deploy core switches and map out the switches for the closets, according to the number of cables coming in from the training and briefing rooms. The floor plans in PDF were uploaded into the Meraki Dashboard; and with a little fine tuning, aligned perfectly with the Google Map.

Cisco Meraki APs were then placed physically in the venue meeting and training rooms. Having the APs named, as mentioned above, made this an easy task. This enabled accurate heatmap capability.

The Location Heatmap provided the capability to drill into the four levels of the conference, including the Expo Hall, lower level (North Conference Center), 2nd Floor and 3^rd Floor. Below is the view of the entire conference.

Network Visibility

We were able to monitor the number of connected clients, network usage, the people passing by the network and location analytics, throughout the conference days. We provided visibility access to the Black Hat NOC management and the technology partners, along with full API (Application Programming Interface) access, so they could integrate with the network platform.

Alerts

Cisco Meraki alerts provide notification when something happens in the Dashboard. Default behavior is to be emailed when something happens. Obviously, emails got lost in the noise, at Black Hat Asia 2022, we made a web hook in Cisco SecureX orchestration to be able to consume Cisco Meraki alerts and send it to Slack (the messaging platform within the Black Hat NOC), using the native template in the Cisco Meraki Dashboard.

The alert kicked off if an AP or a switch lost connectivity. At Black Hat USA, we modified this to text alerts, as these were a priority. In the following example, we knew at the audio-visual team unplugged a switch to move it and were able to deploy technical associates from the NOC to ensure it was reconnected properly.

The Cisco Stack’s Potential in Action, by Paul Fidler

As we planned for Black Hat USA, the number of iOS devices to manage and protect rose from 300+ to over 900, and finally over 1,000.

The first amongst these was the use of the Cisco Meraki API. We were able to import the list of MAC addresses of the Cisco Meraki APs, to ensure that the APs were named appropriately and tagged, using a single source of truth document shared with the NOC management and partners, with the ability to update en masse at any time. Over three quarters of the AP configuration was able to be completed before arriving on site. 

Meraki Systems Manager – Initial device enrollment and provisioning

We’ll start with the positive: When it comes to creating the design to manage X number of devices, it doesn’t matter if it’s 10 devices, or 10,000… And this was certainly true for Black Hat. The requirements were straightforward:

• Have several apps installed on devices, which each had a particular role
• Have a passcode policy on some devices
• Use home screen layout to help the conferences associates know which app to use
• Use Name synchronization, so that the name of the device (on a label on the back) was also in the SM dashboard and under Settings > General > About
• Use restrictions to prevent modification of accounts, Wi-Fi and prevention of screenshots (to protect the personal information of attendees)
• Prevent the devices from having their management profile removed
• Ensure that the devices could connect to the initial WPA based network, but then also to the 802.1x based network (using certificates)

All this configuration was done ahead of time in the Meraki Dashboard, almost a month before the conference.

Now the negatives: Of all the events that the company who supplies the devices attends; Black Hat is the only one where devices are managed. Using mass deployment techniques like Apple’s Automated Device Enrollment, therefore, is not used. The company pre-stages the devices using Apple Configurator, which allows for both Supervision and Enrollment.

It became more difficult: Whilst the pre-staged devices were fine (other than having to handle all 1,000+ devices to turn Wi-Fi to Autojoin and opening the Meraki Systems Manager app [to give us Jailbreak and Location visibility]), an extra 100 devices were supplied that were not enrolled. As these devices were enrolled elsewhere from the prior Black Hat conferences, a team of around 10 people pitched in to restore each device, adding the Wi-Fi profile and then enrollment.

Fortunately, Apple Configurator can create Blueprints:

A Blueprint is essential a list of actions, in a particular order, that Apple Configurator can run through autonomously

But why did it need a team of ten? There were several limitations:

• Number of USB ports on a computer
• Number in USB-A to USB-C converters (the devices were supplied with USB-A cables)
• Downloading of the restore image (although Airdrop was used to distribute the image quickly)
• Speed of the devices to do the restore (the actual Wi-Fi and enrollment steps take less than 10 seconds)

However, the task was completed in around three hours, given the limitations! If there’s one lesson to learn from this: Use Apple’s Automated Device Enrollment. 

Command vs Profile

One of the slight nuances of Apple Mobile Device Manager is the difference between a ‘command’ and ‘profile’. Within the Meraki Systems Manager dashboard, we don’t highlight the difference between the two. But it’s important to know. A ‘profile’ is something that remains on the device: If there’s a state change on the device, or the user attempts something, the profile is always on there. However, a ‘command’ is exactly that: It’s sent once, and if something changes in the future, then the command won’t have any effect.

So, why is this highlighted here? Well, in some instances, some apps weren’t pushed successfully: You’d see them on the device, but with a cloud icon next to them. The only way to resolve this would be to remove the app, and then repost it. But we were also using a Homepage Layout, which put various apps on various pages. Pushing the app would result in it appearing on the wrong page. To ensure a consistent user experience, we would push the homepage profile again to devices to take effect.

Meraki BSSID Geolocation

We’ve mentioned this before in past Black Hat events, but, given the scale of The Mandalay Bay, it’s important to circle back to this. GPS is notoriously unreliable in conference centers like this, but it was still important to know where devices are. Because we’d ensured the correct placement of the Access Points on the floor plan, and because Systems Manager was in the same organisation, it ensured that the devices reported their location accurately! If one were to ‘walk’ we could wipe it remotely to protect your personal details.

Protection of PPI (Protected Private Information)

When the conference Registration closed on the last day and the Business Hall Sponsors all returned their iPhones, we were able to remotely wipe all the devices, removing all attendee data, prior to returning to the device contractor.

APIs

As mentioned elsewhere in this blog, this was a conference of APIs. Just the sheer scale of the conference resulted in the use of APIs. Various API projects included:

• Getting any ports down events with the getNetworkEvents API call
• Getting the port status of switches with a given tag with getDeviceSwitchPorts
• Turning off all the Training SSIDs in one go with getNetworkWirelessSsids and updateNetworkWirelessSsids
• From a CSV, claiming devices into various networks with tags being applied with claimNetworkDevices and updateDevice (to name it)
• Creation of networks from CSV with createOrganizationNetwork
• Creation of SSIDs from CSV with updateNetworkWirelessSsids: This was to accommodate the 70+ SSIDs just for training! This also included the Tag for the SSIDs
• Adding the Attendee SSID to every training network with updateNetworkWirelessSsids: This was due to us having several networks to accommodate the sheer number of SSIDs
• Amending the Training SSIDs with the correct PSK using updateNetworkWirelessSsids

From a Systems Manager perspective, there were:

• The renaming of devices from CSV: Each of the devices had a unique code on the back which was NOT the serial number. Given that it’s possible to change the name of the device on the device with Systems Manager, this meant that the number could be seen on the lock screen too. It also made for the identical of devices in the Systems Manager dashboard quick and easy too. The last thing you want is 1,000 iPhones all called “iPhone!”

Port Security, by Ryan MacLennan, Ian Redden and Paul Fidler

During the Cisco Meraki deployment, we had a requirement to shutdown ports as they went inactive to prevent malicious actors from removing an official device and plugging in theirs. This ability is not directly built into the Cisco Meraki dashboard, so we built a workflow for the Black Hat customer, using the Cisco Meraki API. To achieve this, we created a small python script that was hosted as an AWS (Amazon Web Services) Lambda function and listened for webhooks from the Cisco Meraki Dashboard when a port went down. Initially this did solve our issue, but it was not fast enough, about five minutes from the time the port went down/a cable was unplugged. This proof of concept laid the groundwork to make the system better. We migrated from using a webhook in the Cisco Meraki Dashboard to using syslogs. We also moved the script from Lambda to a local server. Now, a python script was scanning for syslogs from the switches and when it saw a port down log, it will immediately call out to the locally hosted python script that calls out to the Cisco Meraki API and disabled the port.

This challenge had many setbacks and iterations while it was being built. Before we settled on listening for syslogs, we tried using SNMP polling. After figuring out the information we needed to use, we found that trying to poll SNMP would not work because SNMP would not report the port being down if the switch to another device was fast enough. This led us to believe we might not be able to do what we needed in a timely manner. After some deliberation with fellow NOC members, we started working on a script to listen for the port down syslogs. This became the best solution and provided immediate results. The ports would be disabled within milliseconds of going downThe diagram below shows an example of what will happen: If the Workshop Trainer’s device is un-plugged and a Threat Actor tries to plug into their port, a syslog is sent from the Cisco Meraki switch to our internal server hosting the python listener. Once the python script gets the request, it sends an API call to the Cisco Meraki API gateway and the Cisco Meraki cloud then tells the switch to disable the port that went down very briefly.

However, what was apparent was that the script was working TOO well! As discussed, several times already in this blog, the needs of the conference were very dynamic, changing on a minute-by-minute basis. This was certainly true in Registration and with the Audio-Visual teams. We discovered quickly that legitimate devices were being unplugged and plugged in to various ports, even if just temporarily. Of course, the script was so quick that it disabled ports before the users in registration knew what was happening. This resulted in NOC staff having to re-enable ports. So, more development was done. The task? For a given network tag, show the status of all the ports of all the switches. Given the number of switches at the conference, tags were used to reduce the amount of data being brought back, so it was easier to read and manage.

Mapping Meraki Location Data with Python, by Christian Clausen

In the blog post we published after Black Hat Asia 2022, we provided details on how to collect Bluetooth and Wi-Fi scanning data from a Meraki organization, for long-term storage and analysis. This augmented the location data provided by the Meraki dashboard, which is limited to 24-hours. Of course, the Meraki dashboard does more than just provide location data based on Wi-Fi and Bluetooth scanning from the access points. It also provides a neat heatmap generated from this data. We decided to take our long-term data project a step further and see if we could generate our own heatmap based on the data collected from the Meraki Scanning API.

The Folium Python library “builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the leaflet.js library” to provide all kinds of useful mapping functions. We can take location data (longitude and latitude) and plot them on lots of built-in map tiles from the likes of OpenStreetMap, MapBox, Stamen, and more. Among the available Folium plugins is a class called “HeatMapWithTime.” We can use this to plot our Meraki location data and have the resulting map animate the client’s movements.

Step 1: Collect the data

During the previous conference, we used a Docker container containing a couple Flask endpoints connected via ngrok to collect the large amount of data coming from Meraki. We re-used the same application stack this time around, but moved it out from behind ngrok into our own DMZ with a public domain and TLS (Transport Layer Security) certificate, to avoid any bandwidth limitations. We ended up with over 40GB of JSON data for the conference week to give to Black Hat!

Step 2: Format the data

Folium’s HeatMapWithTime plugin requires a “list of lists of points of time.” What we wanted to do is generate an ordered dictionary in Python that is indexed by the timestamp. The data we received from the Meraki API was formatted into “apFloor” labels provided by the admin when the access points are placed. Within each “apFloor” is a list of “observations” that contain information about individual clients spotted by the AP scanners, during the scanning interval.

Here’s what the data looked like straight from the Meraki API, with some dummy values:

The “observations” list is what we wanted to parse. It contains lots of useful information, but what we wanted is MAC address, latitude and longitude numbers, and timestamp:

We used Python to iterate through the observations and to eliminate the data we did not use. After a lot of data wrangling, de-duplicating MAC addresses, and bucketizing the observations into 15-minute increments, the resulting data structure looks like this:

Now that the data is in a usable format, we can feed it into Folium and see what kind of map we get back!

Step 3: Creating the map

Folium is designed to project points onto a map tile. Map tiles can show satellite images, streets, or terrain, and are projected onto a globe. In our case, however, we want to use the blueprint of the conference center. Folium’s allows for an image’s overlay to be added, and the bounds of the image to be set by specifying the coordinates for the top-left and bottom-right corners of image. Luckily, we can get this from the Meraki dashboard.  

This enabled us to overlay the floorplan image on the map. Unfortunately, the map tiles themselves limit the amount of zoom available to the map visualization. Lucky for us, we did not care about the map tile now that we have the floorplan image. We passed “None” as the map tile source and finally received our data visualization and saved the map as an HTML file for Black Hat leadership.

We opened the HTML file, and we had an auto-playing heatmap that lets us zoom at far in as we want:

Detail at 1:30pm PT, on 10 August 2022 below.

To improve this going forward, the logical next steps would be to insert the data into a database for the Black Hat conference organizers, for quick retrieval and map generation. We can then start looking at advanced use-cases in the NOC, such as tracking individual a MAC address that may be producing suspicious traffic, by cross-referencing data from other sources (Umbrella, NetWitness, etc.).

——————————————————————————————————

Network Recovery, by Jessica Bair Oppenheimer

Once the final session ended, the Expo Hall closed and the steaming switched off, dozens of conference associates, technical associates, Mandalay Bay engineers and Cisco staff spread out through two million square feet and numerous switching closets to recover the equipment for inventory and packing. It took less than four hours to tear down a network that was built and evolved 11 days prior. Matt Vander Horst made a custom app to scan in each item, separating equipment donated to Black Hat from that which needed to be returned to the warehouse for the next global Cisco event.

Adapt and overcome! Check out part two of this blog, Black Hat USA 2022 Continued: Innovation in the NOC.

Until then, thanks again to our Cisco Meraki engineers, pictured below with a MR57 access point.

Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.

Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter

Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey

Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira

Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz

Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann

SecureX threat response, orchestration, device insights, custom integrations, and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor

Umbrella DNS: Christian Clasen and Alejo Calaoagan

Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley

Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).

About Black Hat

For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.

---

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn

Share: