Why Security Can’t Be An Afterthought in DevOps

Through the evolution of the software development process over the last decade, our industry has seen the wide adoption of a game-changing paradigm: DevOps. Sadly, while many organizations have commissioned shiny new DevOps teams, those same organizations continue along with their decades-old approach of implementing (or neglecting) security measures.

Shift left security

As organizations shift left with DevOps, what does it mean to embrace shift left security? “Shift left” means that operational responsibilities shift leftward on the development timeline, from the Operations Team to the Development Team.

With security as an integral part of the software development life cycle, what does it look like to combine security concerns with the DevOps paradigm?

Let’s dive in and find out.

The DevOps revolution

DevOps empowers developers to take full responsibility for the end-to-end process of developing code, deploying it to production, and monitoring it post-release.

DevOps has moved forward, security was left behind

Traditionally, developers wrote code and tested it locally. Then, they threw it over the wall to a QA team that tested it in their own environment. Then, before release (which happened once every six months or one year) all the code changes from all the different features were integrated, and the entire system was retested. After fixing integration errors and regression bugs, the updated system was finally released to customers.

This traditional pre-DevOps approach was slow, but it was tolerable as long as your software wasn’t too complex and didn’t need to scale. Today, with microservices and the cloud, applications have the complexity of countless moving parts, and they are expected to scale.

The DevOps approach evolved to meet the needs of scaling and agility, with better tools and practices for automated testing and infrastructure provisioning.

However, many organizations continue to practice security according to the old ways: a siloed security team that is uninvolved with design and development but is responsible for managing the mayhem of large-scale distributed systems developed at a high velocity and released at a high cadence.

Like operations, security needs to shift left.

The security arms race in the age of DevOps

The bad guys didn’t sit out the DevOps revolution. With software eating the world, more and more lucrative targets are popping up. Cryptocurrencies allow cyber criminals to extract value from their exploits with absolute anonymity and little risk. The explosion of sensors and Internet of Things (IoT) presents a huge attack surface.

All these factors lead to attackers that operate at a larger scale than ever before, and they’re equipped with high-quality kits to mount powerful attacks against more targets. As if that’s not enough to keep developers up at night, state-level actors that target highly defended targets like core physical infrastructure.

The defenders, fortunately, haven’t stayed behind. Academia, corporations, and governments invest a lot in research, standards, and processes to combat the proliferation and sophistication of attacks.

Parallels between the DevOps and security life cycles

When we take a step back, we notice that good security practices are actually very similar to good development and operational practices. Building reliable and highly-available systems often goes hand-in-hand with building secure systems.

The DevOps process can be divided roughly into three phases, and we can blend security practices into this process:

1. Development: everything that happens before deployment, which includes planning, coding, code review, local testing, and pre-deployment testing.
2. Day 1 operations: the deployment itself, including infrastructure provisioning and smoke tests, along with rollbacks if something goes wrong.
3. Day 2 operations: monitoring the system, detecting problems, incident management and alerting, and post-mortem.

All of these tasks are relevant for security as well. Many of the problems, risks, and mitigation techniques apply to both common failures and security issues.

For example, if your system depends on an out-of-date version of some library, this can cause operational issues if the library contains bugs that cause it to leak memory or crash occasionally. However, the library could also contain security vulnerabilities. The process to handle both operational and security issues is the same: Be aware of known issues with your dependencies, and upgrade to a newer version when discovering an issue.

That said, there is a notable difference between standard operations and security operations. With standard operations, you are concerned with mistakes; with security operations, you are concerned with adversaries. In security operations, you intentionally look for chinks in your armor.

DevSecOps – putting security into the DevOps pipeline

Let’s look at a typical software development life cycle to see how we might bake security into the pipeline..

Security-aware planning and design
System planning and design starts with threat modeling and is followed by deciding how to mitigate risks and what level of risk is acceptable. It follows with concerns such as levels of isolation, access control to data and systems, monitoring, and auditing.

Security-aware planning may also involve staffing changes, with security engineers embedded in development teams to inform the software development task, while a central security team oversees overall security and provides guidelines.

Security-focused code review
Introducing code changes typically introduces new security issues. Just as any code change should be reviewed for functional and operational issues, it should also be reviewed for security concerns.

Checking dependencies for vulnerabilities in the CI/CD pipeline
Today, enterprise systems commonly leverage open source libraries and entire third-party systems. Updating dependencies can lead to incompatibility issues and break your system. However, updating dependencies may be necessary in order to counter newly-discovered vulnerabilities.

On the other hand, updating a dependency may introduce new vulnerabilities. Checking dependencies for security issues against Common Vulnerability and Exposure (CVE) databases should be a step in any change management process. Many tools can help DevSecOps teams with this.

Artifact scanning
In today’s DevOps pipeline, software is packaged—prior to deployment—in different forms like VM images or containers with their own OS, along with other tools in which that software is layered. A strong DevSecOps approach would require scanning of these intermediate and final artifacts. This scanning process can be a separate step in the CI/CD pipeline just prior to deployment.

The next frontier: AI in security

As systems become more complex and accommodate remote access by users and other systems, DevSecOps teams find that automating the CI/CD pipeline with security measures alone is not enough to mitigate all of the potential risks. Already deployed systems must still face many dangers. Often, it seems that teams must make some sort of compromise between security and productivity.

In these cases, DevSecOps teams need a smarter security apparatus. Automatic detection—and even prevention—of issues is of utmost importance. The complexity and surface area of a system may be too great for teams to define specific rules for issue detection. This requires a more holistic approach.

This is where AI can step up to the plate and help to detect anomalies. With AI-assisted security tools, systems can block unusual user activity and even examine processes to suggest more secure processes. A good example of AI in security is Amazon Macie, which scans your data and uses machine learning to identify and alert teams to the presence of sensitive data such as personally identifiable information (PII).

Conclusion

Security has always been a paramount concern of software systems. As the approach to building software (now with DevOps teams involved) and to architecting software (now with microservices and cloud-native approach) has evolved, the engineering team’s approach to security should as well.

At the beginning stages of system planning—rather than as an afterthought or as a reaction—security needs to be a high-priority concern of DevSecOps teams. Teams need to take responsibility for the security of the components they own. The shifting left of operational responsibilities must now be accompanied by a likewise shift of security responsibilities, too.

Related resources

We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!

Twitter @CiscoDevNet | Facebook | LinkedIn

Visit the new Developer Video Channel

Share: