How to Measure the Success of Your Phishing and Security Awareness Training (PSAT) Program
By Greg Crowley, CISO, eSentire
Ransomware and crippling cyberattacks are so pervasive they have permeated our social fabric down to the late night comedy. HBO’s Last Week Tonight with John Oliver covered the subject of ransomware and even how cybercriminals race their Lamborghinis through the streets of Moscow, lavishing their protection within the borders of their native country, Russia in this case.
Most of these crippling cyberattacks begin with a simple phishing email or business email compromise (BEC) scam. While most companies provide some form of annual cybersecurity awareness training to reduce the likelihood of employees being hooked, they focus on overly simplistic lures taken from public events that fail to represent the real danger of targeted cyberattack campaigns.
What’s worse is they rely on disengaged metrics like employee attendance and simple test results as a proxy to measure business risk reduction. However, there is little to no connection between the two. In fact, this short-sighted approach creates a false sense of security which only exposes the business further to cyber risk.
Stepping back for a moment, the purpose of phishing and security awareness training (PSAT) is to drive behavioral changes amongst your employees and the corporate ecosystem to improve business resiliency through reduced exposure to related cyber risks, alleviated resource constraints, and increased ability to meet regulatory compliance requirements.
So, why would you look at attendance lists and the so-called “smiley sheets” to assess business resiliency? This is why most programs fail–they don’t represent the cyber risks, they don’t provide the tools and skills necessary for employees to identify cyberattacks, and they focus on the wrong measures.
Plus, think about the wasted effort and costs. Whether you see it or not, there is a cost associated with your employees sitting in the PSAT classroom sessions. That salary easily eclipses the direct expenses associated with PSAT training sessions, such as the cost of lunch or even the speaker.
How do you know your PSAT program is working?
Successful PSAT programs move beyond the overly represented focus on compliance. They provide contextual phishing examples, empower employees to protect themselves and the business, and provide clear threat reporting mechanisms.
When it comes to measuring and reporting PSAT effectiveness there are two models you can explore: the SANS Security Awareness Roadmap and the Kirkpatrick model.
The SANS Security Awareness Roadmap
The SANS Security Awareness Roadmap provides a tiered approach to maturing your program similar to other maturity models, in that it helps your organization move from a state of disregard for cyber risk and towards a more integrated, team approach:
| Level | Focus | Leadership Approach | Employee Behavior |
| 1 | No program | Leadership does not recognize the cyber risks nor value of PSAT | Employees are oblivious and do not discuss cybersecurity |
| 2 | Compliance | Leadership understands cybersecurity obligations and risks | Employees do not understand cybersecurity, and attend PSAT as unproductive obligation |
| 3 | Behavior | Leadership understands cybersecurity obligations and risks and funds the program | Employees understand the risk and can identify threats and proactively report potential cyber threats |
| 4 | Culture | Long-term commitment to PSAT with metrics-driven decision making | Employees are empowered to make recommendations and proactively seek training and report cyber threats. |
| 5 | Outcomes | Leadership aligns business objectives, cyber risk prioritization, and PSAT resources | Employees are empowered to make recommendations and proactively seek training and report cyber threats. |
This is a good framework that can help you determine where your company is in terms of maturity and identify the gaps you need to fill to move up the safety stack.
The Kirkpatrick Model
The Kirkpatrick model is a long-standing standard for validating the efficacy of investments in training and learning. Like the SANS Security Awareness Roadmap, the Kirkpatrick model consists of four levels of maturity and indicators, focusing on Reaction, Learning, Behavior, and Results. Below, I’ve provided the levels and their focus, what they evaluate, and some specific examples of how to measure PSAT success.
| Level | Focus | Evaluation | Examples to Measure PSAT Success |
| 1 | Reaction | The degree to which participants find the training favorable, engaging, and relevant to their jobs. |
|
| 2 | Learning | The degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training. |
|
| 3 | Behavior | The degree to which participants apply what they learned during training when they are back on the job. |
|
| 4 | Results | The degree to which targeted outcomes occur because of the training and the support and accountability package. |
|
All too often, program coordinators proudly display their “smiley sheet” results and tout the number of employees trained as a measure of their value. Yet these results only measure subjective response, which in no way indicates the trainees’ ability to identify a cyber threat and avoid falling for the cybercriminal’s intent.
So, which metric should you depend on most? The answer is “Engagement”. It’s the most qualitative measure and it tells you that you’ve got through to your employees.
Ask your employees why they reported a suspicious email to see which indicators triggered the response. This will tell you if your employees feel empowered to protect the business and whether your program works:
- Was it an “external” flag on an email that looks like it comes from an employee colleague?
- Was it because the sender’s email signature or tone seemed insincere or unusual from a known colleague?
- Was it that the sender’s ask was out of bounds?
The second metric to look at is the time employees proactively spend with the IT/Security team. This action shows passion, which in turn shows their commitment to cybersecurity and their desire to protect the business.
Make it personal and make it worth it
When it comes to engaging employees, metrics can measure your success, but they normally don’t motivate attendance and participation. Nor do compliance requirements. We’ve all taken cybersecurity awareness courses with the class window in the background as we do our “real work”.
One of the best examples of a successful phishing and security awareness training program is a creating tailored campaigns, including posters, swag, and a competition ladder of escalating phishing testing to crown a cyber champion. It’s an innovative approach that can lead to record attendance, motivated employees, and reduced operational costs. More importantly, it also reduces the risk of phishing.
Remember, just because cybersecurity is a serious topic, doesn’t mean your company can’t have fun!
How eSentire’s Phishing and Security Awareness Training program can help
Designing, developing, and executing a fun–yet, educational–PSAT program is hard. Checking a box for compliance is easy, but moving the needle takes more leverage and more force. The better the PSAT program, the more effort is required since the complexity increases.
eSentire’s Managed Phishing and Security Awareness Training puts your organization on the forefront of user protection against the latest social engineering cyberattacks.
Our end-to-end service alleviates the resources required to operationalize an effective user resiliency program. By leveraging software paired with dedicated social engineering experts, we ensure that your users are continuously tested and hardened against even the most sophisticated phishing attempts. Our robust phishing libraries consist of hundreds of templates, ensuring users are tested against real-world scenarios, not commoditized and easy to spot templates.
More importantly, our PSAT training modules are automatically assigned to users that fall victim to simulated testing, ensuring education is integrated at the moment of failure and drives sustained behavioral change. Our PSAT program generates measurable results, extending well beyond checking a compliance box. We not only help you meet regulatory requirements, but also ensure that your organization is resilient against the latest social engineering tactics.
To learn how eSentire’s Managed Phishing and Security Awareness Training can help drive behavioral change with your employees across your organization, book a meeting with a cybersecurity specialist today.
About the Author
Greg Crowley is an accomplished executive with over 20 years in Information Technology and Cybersecurity with extensive experience in managing enterprise security and mitigating risk for global hybrid networks. Greg believes that as a leader in the cyber world, being able to communicate and execute a strategic vision to defend and protect is the most important part of his role. Prior to joining eSentire, Greg oversaw the overall cybersecurity function as Vice President of Cybersecurity and Network Infrastructure at WWE (World Wrestling Entertainment). He spent over 17 years in various leadership roles across engineering, infrastructure and security within that organization. Greg holds a Bachelor’s degree from Queens College. He is a Certified Information Security Manager (CISM) and a Certified Information Systems Security Professional (CISSP).
FAIR USE NOTICE: Under the “fair use” act, another author may make limited use of the original author’s work without asking permission. Pursuant to 17 U.S. Code § 107, certain uses of copyrighted material “for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright.” As a matter of policy, fair use is based on the belief that the public is entitled to freely use portions of copyrighted materials for purposes of commentary and criticism. The fair use privilege is perhaps the most significant limitation on a copyright owner’s exclusive rights. Cyber Defense Media Group is a news reporting company, reporting cyber news, events, information and much more at no charge at our website Cyber Defense Magazine. All images and reporting are done exclusively under the Fair Use of the US copyright act.
