DevSecOps

What is DevSecOps?

Introduction

The modern software engineering team practicing DevOps faces a gauntlet of obstacles in the way of shipping secure production software. As the cloud, microservices, and CI/CD have changed the way we build software, refactoring security for DevOps requires a full-stack approach to build a collaborative security culture. In this guide, we present the most pervasive areas in DevSecOps where teams have invested to realize the most value for evolving their security program and approach.

Recent studies such as the BSIMM 11 (https://www.bsimm.com/) suggest that software security groups are increasingly prioritizing cloud and security activities while focusing on secure deployment parameters and configuration across their portfolios. As the modern “full-stack developer” frequently spans the operating system, networking, and software stacks from top to bottom, building security into your system from the ground up requires a comprehensive approach.

Observability

Logging, Metrics, and Tracing

As your distributed systems become more complex and as you leverage more IaaS and PaaS capabilities, having visibility into security events and understanding data flows as your infrastructure evolves, is an indispensable part of your security strategy. With a greater rate of change, microservices and serverless architectures require continuous monitoring to detect events across immutable infrastructure. Often, flaws within IAM implementations can allow an attacker to pivot between services. Understanding regular traffic patterns and anomalous trends comes from having a strong core for observability in place.

What is Logging in a DevSecOps environment?

Logs provide you with the ability to reconstruct a sequence of events after they’ve occurred, whether it’s due to an unexpected outage or because of a suspected security incident. There are many places within software systems where logging must occur. However, centralized log ingestion and analysis are required for tracking potential malicious activities across a broader enterprise environment. When focusing on logging, the primary places to ensure logging is in place:

  1. Cloud or infrastructure level events
  2. Operating system logs – virtual machines, containers, or bare-metal
  3. Networking/Device logs – load balancers, container orchestration systems, or network appliances
  4. Developer Systems – auditing of CI/CD, Git, container registries, artifact servers
  5. Applications – custom software logs and third-party products/libraries

What DevOps Metrics should you track?

Metrics allow us to measure various properties of a system’s performance and activities to better understand how a system behaves under normal and unexpected conditions. Using baseline behaviors and metrics, we can create alarms and alert when various thresholds are crossed or as we encounter generally unexpected scenarios that warrant further investigation.

Tracing Defined for DevSecOps

Tracing allows us to leverage logs and metrics to understand the path a request takes throughout a system. As micro-services and distributed systems create many additional paths and data flows to analyze, tracing gives us context to understand the behavior of our system as our requests and data move between components. For security, tracing gives us insight into the interconnected state of our system and components to better understand its attack surface. We can leverage data and context gained from tracing to better understand ongoing incidents and to reconstruct an attacker’s behavior while moving throughout a system architecture.

infrastructure

Infrastructure and Policy as Code

Software-defined infrastructure and networking give us the opportunity to build programmatic and declarative controls to automate many aspects of security. Building codified policies to manage and enforce security across the many moving layers of your distributed architecture, is an invaluable security tool in your goal to move quickly with low friction.

What is Infrastructure as Code (IaC)?

Infrastructure as Code (IaC) represents a shift in describing and deploying infrastructure as well as an opportunity to embed security into our core designs and implementations in a declarative and easily reproduced way. By having centralized and auditable configurations with strong change controls in place, this gives organizations an advantage in streamlining security while also requiring many organizations to retool the way they detect and prevent attacks.

What is Policy as Code?

Policy as Code allows us to codify our checklists and rather than continuously audit configurations, we can continuously enforce our baseline standards. Often, environments increasingly drift away from their desired state and this can introduce unexpected security risks and exposure points. Policy as Code allows us to express our desires within the form of programmatic constraints, and have continuous enforcement of these policies within a runtime environment.

CI/CD

The CI/CD pipeline you use to build and deploy software is a critical component to securing your organization. When considering security within the pipeline, we must focus on keeping our attack surface narrow within our development infrastructure itself, while making sure automated and manual checks occur as various stages are executed.

How to secure the DevSecOps Pipeline

To secure our software, we need to make sure the pipeline we ship it with is free of security defects and that our pipeline can enforce our procedural security constraints in an automated fashion. This requires hardening the CI/CD pipeline systems themselves as well as ensuring proper authentication and access control as it interacts with external systems such as cloud infrastructure, application servers, and container orchestration APIs. In addition to technology hardening across our common pipeline components, we must consider the realistic threats against each of our engineering team members and consider how to limit our attack footprint there as well.

In many organizations, software engineers and development teams are often given highly elevated privileges to production or pre-production environments. They are frequently granted elevated rights to their local systems where they run different development, build, and testing tools. Often, they may have sensitive data (ie- production databases) stored locally and they have access to view and modify source code across repositories.

How can we limit the likelihood of a compromised developer account causing us a large-scale security incident?

  1. Require multi-factor authentication everywhere possible. While multi-factor authentication does not prevent all account takeovers, it increases the difficulty in doing so and significantly reduces the likelihood. This includes at login, as well as using keys and cryptographic verification to authorize sensitive transactions such as a commit to main/master branches, etc.
  2. Ensure good monitoring and alerting – the better we understand what’s normal, the quicker we’ll be able to identify odd activities as they occur. This includes deploying directly to production rather than through the CI/CD pipeline, operating in an unexpected configuration/security posture, etc.
  3. Enforce the principle of least-privilege access across all of your architectural components. As you build your IAM policies and isolation between your virtual and logical assets, provide services with the minimal amount of privileges required to operate. In the event of a compromised account, this will limit an attacker’s ability to elevate their privileges or move laterally between similarly privileged services.

What tools are used for Securing the Software?

Securing software starts left, but unfortunately it does not stop there. Shifting left lets us get further ahead of security issues by addressing them earlier in development. In order to be successful, we need automated test coverage as well as well-designed deployment and runtime security controls for end-to-end security.

The types of testing we can perform at different phases of our CI/CD pipeline vary depending on the state of our software and built packages. The most commonly used testing techniques for security in CI/CD include:

  • Static analysis
  • Dynamic analysis
  • Software Composition Analysis/ Software Bill of Materials (SBOM)
  • Unit Testing
  • Chaos Testing
  • Interactive Security Testing (IAST)

For example, static analysis is typically run post compilation but prior to building an application, while unit testing requires building and running an application to perform various tests at runtime while either mocking out components or setting up an elaborate testing infrastructure.

Security testing can be built into automated workflows through plugins, integrations, and often, elaborate scripts and custom tooling to glue systems together. The goal of embedding security checks across our stages is to ensure we are identifying issues where we can achieve the highest levels of both speed and precision.

The goal of embedding security checks across our stages is to ensure we are identifying issues where we can achieve the highest levels of both speed and precision.

containerized microservices

Container Security

How to secure containerized microservices: Securing our containerized microservices requires a focus on orchestration systems, container registries, the container runtimes themselves, and any places where our containerized infrastructure interacts with cloud or authentication systems. Containers offer many security benefits over virtual machines, but can also be deployed in insecure ways that expose your services to attack.

Why you should Deploy by Digest

Deploying by digest ensures that containers are pulled by a specific hash rather than a tag number, and that the container is also validated against this hash to ensure its integrity. It is not recommended to pull containers directly from :latest, as this can often have unwanted consequences. While pulling from a tag is preferred over :latest, however tags can also be overwritten and without a digest, limited capabilities for integrity checks against the container to detect unauthorized tampering.

Containers free of known security vulnerabilities

Vulnerabilities within your containers will manifest themselves within your base images, configuration, the packages you install, and by the code you deploy to run within containers. Approaching security from each of these perspectives allows you to reduce the overall attack surface within your containers. These activities can all be run in automated fashion within your pipeline to ensure continuous checks are in place. The primary targets for these types of tests include local developer environments, source code repositories, container registries, and artifact servers.

Use policies to centrally manage security at deployment/runtime

Implementing security in an enforceable way at key trust boundaries within your architecture can reduce the amount of vulnerabilities that make it into production. Building enforceable policies that limit dangerous configurations or behaviors while providing a frictionless software developer and infrastructure management experience is a fine balancing act that many organizations attempt.

What is Container Isolation?

Isolating your containers at virtual and logical boundaries is critical to limiting the damage that can occur in the event of an attack. While you cannot prevent all attacks from occurring, you have many tools at your disposal to prevent attackers from elevating their privileges or pivoting into other services. Container Isolation techniques include:

  • User namespaces (https://www.man7.org/linux/man-pages/man7/user_namespaces.7.html) – Remapping a container’s privileges to a user namespace significantly limits the capabilities allowed on the host operating system. This limits the facilities available to an attacker to break out of the container and attack the host itself as well as other services running on that node.
  • No root – Allowing root users within the container gives attackers increased surface to break out of the container, through vulnerabilities within container runtimes as well as common misconfigurations which grant host system resource access. Root can be restricted within both container manifest files (ie- Dockerfile) as well as by container orchestration configurations (ie- Pod Security Policy, Pod Spec).
  • No host resource access – do not grant local IPC, network, or other privileged host resource access that can give an attacker the ability to bind to resources assigned to other containerized services or the host itself.

Linux Security Modules

Kernel modules can be used to enforce security and have first class support across leading container runtimes. These include SELinux, AppArmor, and Seccomp, which allow you to restrict dangerous functionality and run with a reduced set of privileges at runtime. These protections mitigate against attacks within the container as well as limiting the surface for performing a container escape. For example, vulnerabilities within Docker that have allowed attackers to break out of containers have been either partially or completely mitigated by effective usage of SELinux in tandem with secure configurations.

Secure Code and Artifact Management

Code and Artifact Management

How to Secure Code and Artifact Management:
Securing access to your code, packages, configurations, containers, and virtual machines spans many layers from IAM through configurations and policies. Our goals are to reduce the number of vulnerabilities within packages that can be deployed, limit the likelihood of code disclosure as well as to minimize the risks of malicious code being introduced into an environment. Many organizations use a combination of on-prem and cloud-based infrastructure for hosting and managing their code and artifacts. It is essential we protect each of these systems along our path to shipping secure software.

Git Hygiene

The ability to commit and deploy code as a privileged developer presents a serious risk to software development teams. To minimize these risks, there are several controls we can implement at the repository level that can make it harder for an attacker to read or write code in unauthorized locations.

Four Controls to Implement for Google Git Hygiene

Multi-factor authentication

Each account should be protected with multi-factor authentication, using SMS, using the Time-based One-time Password (TOTP) algorithm, or through other biometric or behavioral indicators. If a developer’s credentials were compromised in a public breach and the developer also used the same password to access your publicly available Git server, this would represent an exposure point for your organization without MFA. When your code repository supports it, MFA increases the difficulty for account takeover attacks without much extra effort.

Signing commits

Popular Git implementations support the ability to sign each commit using a PGP key. This results in an additional level of authentication for each commit beyond the developer’s login credentials themselves. In the event a developer’s password were compromised, requiring commit signing increases the difficulty for malicious or untrusted code to be introduced.

Branch protection

Frequently, branches are mapped to the environments they can be deployed to via CI/CD. Main or master branches commonly deploy to production, while other common branch names such as staging or development may typically correspond to a non-production environment. Protecting important branches by requiring additional verification or approvals can help reduce accidental deployments and malicious or backdoor code from being committed.

Scan for secrets, high confidence checks

In addition to the CI/CD checks performed against your code as it is tested across pipeline stages, there are checks that can be run directly against repositories that can give you fast and accurate results. Common issues that can be detected quickly and accurately in this manner tend to be of the greppable and lintable variety. These include:

  • Secrets – common formats including AWS or Azure keys, SSH private keys, plain text passwords, etc.
  • Sensitive data – often, production data accidentally gets checked in. You want to find the plain text credit cards, passwords, and SSNs before the bad guys do.
  • Security misconfigurations – known insecure modes of operation and common vulnerability patterns.

Container Registries

To ensure good container security, we can employ several techniques at our registries to keep dangerous or vulnerable code out of our registries, while also enhancing the trust level when deploying images to a production environment.

How to keep vulnerable code out of your container registries

Deploy by Digest

Deploying from :latest or by a specific tag often leads to unexpected results when the container does not match the expected configuration or packages. Often, teams will overwrite tagged builds when using incremental numbers, returning unexpected code and results.

Image Signing

Ensure that only signed images can be deployed to production clusters and orchestration systems. Signing authenticates the container as trusted, while also providing integrity by making sure the container’s base image or manifests have not been tampered with. Signing can be used throughout your pipeline to attest that containers were built by a specific system, which can include approved CI/CD pipelines as well as other trusted deployment tools. Implementing signing also decreases the likelihood of a malicious developer or attacker deploying rogue containers to your clusters.

Our goals are to reduce the number of vulnerabilities within packages that can be deployed, limit the likelihood of code disclosure as well as to minimize the risks of malicious code being introduced into an environment.

Cloud Security

Software built for the cloud leverages a combination of APIs, IaaS, and PaaS. Building security into a cloud architecture requires carefully planning how your infrastructure and software will grow over time. Designing IAM strategies including separate accounts or subscriptions across business units, application teams, or across test environments, as well as network design and segmentation, are important steps in building a security strategy that will evolve over time.

Organizations move to the cloud in different ways. In a lift and shift move to the cloud, organizations strive to make a few changes to the software and architectural design itself.

Identity & Access Management

IAM is woven into every aspect of your software delivery pipeline, from the time a developer checks in code through the time it’s deployed via CI/CD, through the capabilities and permissions granted to the software at runtime. As cloud native CI/CD patterns grow in popularity, IAM and role-based access controls (RBAC) propagate deeper across the stack. In general, you want to avoid developers and engineers from interacting directly with cloud architecture. Instead, through deployment artifacts and declarative configurations which launch infrastructure on demand, teams should be deploying through these workflows rather than by manually modifying infrastructure or platform services directly.

Secrets Management

Managing secrets across applications, systems, and environments requires a consistent approach to storing secrets and allowing applications, developers, or administrators to access them. Often, secrets are exposed through code repositories, configuration files, or logging. Frequently, passwords and API tokens are leaked through these channels and their disclosure often significantly undermines the security of production systems. Attackers are typically able to log in as privileged users or they are able to sign or decrypt values to gain access to confidential data.

About nVisium
The Intersection of Software and Security

nVisium empowers organizations to eliminate application and cloud security vulnerabilities before cyber threats exploit them with proven in-depth security assessments, remediation and training programs. Our experienced team of security-savvy developers and engineers guide organizations to build best practices with high ROI into their engineering and development lifecycles across applications, operating systems, networks, mobile, cloud and IoT through services, software solutions and R&D unique to business operations and compliance initiatives. Additionally, nVisum provides a fully managed platform for tracking and measuring performance as well as instructor-led and online training. Privately owned and founded in 2009, nVisium is headquartered in Falls Church, VA, and names Fortune 500 companies and household brands as customers.