12 Jun, 2019

Adapting Agile for Internal Security Operations

by Alex Useche

How we do software development have been evolving. This is not just because we no longer have to worry about maintaining clunky on-prem servers and can instead write serverless applications and push code to a cloud service like AWS or Azure. How we collaborate with other developers, plan, and schedule work has been changing as well. As more companies are adopting new technologies, so are they making agile the default methodology for getting work done, even if agile means something slightly different to everybody. However, despite all the talk about DevOps and agile, most security teams of any size are still operating in the same waterfall fashion.

While development teams get to have fun using DevOps tools and share ideas in daily 15 minute stand-ups as they continuously learn, security is relegated to a dark corner, buried in piles of Excel sheets, and forced through labyrinths built out of Gantt charts and overly complicated projects plans just to get their job done. In my experience working in agile software development teams and leading small security teams, I have found the following issues that keep security from adopting agile:

  1. Most agile and DevOps documentation out there is written for developers and operation teams, not for security teams.
  2. There is very little information on how we can do security in an agile fashion. Agile was created by developers for developers.
  3. Security and development teams often operate separately. This not only reduces the opportunities for security to learn from teams that are already doing agile, but it also exacerbates the disparity between what developers think is a priority and what security thinks is a priority.

Simply put, there is a lack of guidance on how security teams can operate in an agile way. Sure, recent documentation on DevOps tells us that security must be an integral part of the development process, but literature on agile and DevOps as it would apply to security teams is scarce. This is particularly true when it comes to tasks such as vulnerability management, event monitoring, and incident response. This presents a problem: the fact that security teams are not doing agile only increases communication gaps that exist between security and software engineering teams. If developers and security teams rely on different methodologies for doing their work, how are they to be expected to collaborate productively? This post exposes how security teams can operate in an agile fashion, as well as the reasons why organizations can benefit by implementing agile methodologies beyond development tasks.

Before we continue, it is important to answer one question: why does security need to become agile? Combining security operations and agile methodologies can result in the following benefits:

  • Centralized backlog and prioritization of security-related tasks
  • Regular short meetings that allow issues to be brought up sooner
  • Increased visibility of security work
  • Automation of SOC planning and security-related tasks
  • Continuous learning, continuous improvement
  • Increased collaboration with other teams that are already doing agile

The question then becomes, how? The answer is simple: Hack-The-Methodology. Given that there is a noticeable lack of information on how security teams can operate in an agile way, it becomes necessary to do what security teams do best: hack! In my experience working with small internal security operation teams is the best place to start. You see, agile is founded on four fundamental values:

  • Individuals and Interactions Over Processes and Tools
  • Working Software Over Comprehensive Documentation
  • Customer Collaboration Over Contract Negotiation
  • Responding to Change Over Following a Plan

The same agile values can be applied to how we do security. Security teams may not necessarily write software; yet, their products come in the forms of concrete results (a security bug report for an application, a vulnerability analysis for an internal subnet, or remediation of a high-risk network vulnerability) that make an organization secure. Although security teams may not have traditional customers, their customer is the company they help protect.

Once the core values of agile are understood, the following steps can be taken to start hacking agile:

  • Plan and organize work using Kanban or SCRUM boards and implement stand up and sprint planning meetings. For this, it is necessary to adapt agile concepts typically used by software development teams to the needs of security teams.
  • Centralize documentation using tools that enable collaboration and change control. In other words, establish a simple documentation strategy.
  • Source control all the things.
  • Increase collaboration with other teams (development, IT).

Hacking Agile

Kanban Boards, Stands Ups and Sprint Planning Meetings

The first step is to plan to work in a more effective manner. This often means using a Kanban or SCRUM board that can be used to plan sprints, create misuse cases, create and assign tasks, etc. A Kanban board allows teams to collaborate on tasks and plan work. Rather than storing security-related tasks in spreadsheets and sticky notes, backlog items can live in a place where everyone can collaborate, allowing management to get a picture of what the security team is working on. In my experience, it is best to use the same tool that the development team uses (if the development team is working in an agile manner). That way, if security reports a bug to a group of developers, security can participate in development sprint planning meetings, answer questions about the bug, and assist in the prioritization of remediation tasks.

Security needs are in a constant state of change. What seems like a non-issue becomes a high impact risks the following week, making it difficult to make accurate time estimations when planning security work once a quarter ala waterfall. To solve this issue with agile, it is necessary to implement sprint planning meetings. During sprint planning, all security tasks in the backlog are groomed, discussed, and prioritized by the team. Moreover, sprint planning meetings should also catalyze continuous improvement in what is called a retrospective. The following questions are answered in a retrospective:

  • What worked in the previous sprint (i.e., is the team is getting better at estimating effort for assigned tasks)?
  • What can be improved (i.e., regarding the process, team organization, etc.)?
  • What are the lessons learned from the previous sprint?
  • What must the team stop doing to increase collaboration, effectiveness, and value generated by the team?

How Can Security Teams Make Use of User Stories, Epics, Personas, etc.?

User stories, epics, features, and tasks are simply “work item” types that agile teams use to organize and plan work. Each type typically serves a particular purpose, often based on the expected level of effort required for each task. For instance, developers use User Stories to keep track of application features they can complete within one sprint (2-4 weeks). The way that you leverage these constructs for security depends on the needs of your team and company. What matters is that you define and document how each type is to be used so that they are utilized in a consistently. For instance, the table below describes how a small SOC team may use each type of task type:

agile IR

Some team organization tools allow the creation of templates. For instance, in Azure DevOps, it is possible to create a template for tasks related to the remediation of identified vulnerabilities. This allows you to standardize certain types of tasks. The screenshot below shows how a user story is created and a template for a “vulnerability remediation” is applied to the story, requiring security to fill out specific data pertaining to the vulnerability that must be patched.

agile IR

How would this look like in real life? The graphic below shows a possible workflow for the investigation of recurrent events reported by a SIEM tool:

agile IR

In the above case, the security team can establish different workflows for different sets of alarms, depending on the criticality of the events. This is particularly useful in situations where SIEM tools do not provide their users with investigation tracking capabilities.

A similar workflow can take place for remediation tasks of vulnerabilities detected by monthly network scans:

agile vuln management

This allows your team to address security issues faster and organize their time more effectively, while at the same time increasing the visibility of the work done by everyone in your team.

Collaborating and Keeping Track of Documentation Changes

Does your company have documentation in five different tools? Are members of the IT team sequestering knowledge by keeping documentation for things like procedures to follow for creating SIEM alert rules in their laptops, only because they don’t know where to best store it? These types of problems that limit collaboration must be solved when going agile. Spend the time and effort to make sure that there are designated places where documentation files should live in, whether that is a tool like Atlassian Confluence or folders in shared drives. Furthermore, develop metadocumentation; that is, create a document that points team members to the different places where documents of different types should be stored. More importantly, make sure that the guidelines for where documents should live in make sense and are understood by everyone in your team.

Ideally, you’d want to store documentation somewhere that allows you to keep track of changes over time with minimal amounts of effort. Why? Security standards such as ISO 27001 require that there is some sort of change control on procedures and policies. Continuing with the topic of centralizing collaboration, you could use Wiki tools available in platforms like Github, GitLab, and Confluence, which treat every document as a markdown file source controlled using Git behind the scenes. Additionally, a centralized documentation strategy can increase opportunities for collaboration between team members, which is one of the main reasons why organizations adopt agile.

Source Control All the Things!

Just as a lack of a centralized documentation strategy can negatively impact collaboration in your team, so can having code for scripts and security tools lost in hard drives all over the company. To cultivate a culture of collaboration, continuous improvement, and continuous learning, it becomes necessary to have a strategy regarding the storage of source code. Ideally, if your team consists of developers, security analysis, and IT members, you’d want to have everyone push code to the same tool.

Things become more interesting when you start source controlling not only things like PowerShell scripts, but also security configuration files for security appliances. You can even automate the deployment of changes in the configuration of files of firewalls, access points, and WAFs by leveraging Continuous Integration and Continuous Deployment (CI/CD) tools that are embedded in many agile collaborations tools. That way, you can have a script that pushes updated configuration files to different devices when a new change is pushed and approved via a Pull Requests process.

Increasing Collaboration with Teams of Developer Teams

One of the common frustrations that I continue to hear from developers is the disconnect that exists between developers and security teams. On the one hand, security teams are finding vulnerabilities in applications written by internal development teams and demanding quick fixes. On the other hand, developers’ backlogs are so saturated with urgent tasks and technical debt that a SQL injection in the application used to book meeting rooms may not seem like that big of a deal. As a consequence, bugs are never fixed, and the frustrations between teams only increase. In my experience, this disconnect is only worsened when security teams do not understand or adopt agile.

Cross-team collaboration is fundamental for any organization wishing to become agile. For that, it is necessary to break down barriers that create conflict and separation between teams. I am not suggesting that all teams (security, development, operations) fuse into one monolithic unit which conducts sprint planning meetings around a fire. The key word here is “collaboration.” The more teams collaborate with each other, the more agile (that is to say, focused on generating value) they can become. How can that be accomplished?

1) Invite members of development teams to security sprint planning meetings (and vice-versa). 2) If a security bug needs to be fixed by development, assure to add it as an item in the backlog for the development team. There is nothing worse than a security bug that gets forgotten or ignored by development just because they did not have time to address it in the current sprint. Add every bug as a backlog item, so it stares at you every sprint planning meeting. 3) When possible, use tools that allow cross-team collaboration. For instance, Azure DevOps allows you to create multiple teams under one “project.” This means that each team has visibility into each other backlog. That way, security can get an idea as to what development is working on and vice versa.

Wrap up

Is it possible for security to become agile? Absolutely! However, this requires a lot of creativity, communication, and collaboration between team members. In my own experience, the following is necessary to keep in mind when you go through a transformation process (waterfall to agile):

  • As you are implementing agile, many of the tasks that are planned your first few sprints are likely not to be completed in time. This is not necessarily an issue with agile itself. Agile is an iterative process, and the goal is to improve in every sprint. It is to be expected that, at first, your workload estimations are going to be way off. Do not be discouraged and adjust your methodology to fit the needs of your team in every sprint.
  • Make sure to communicate changes as often as possible to reduce frustration among team members as they implement a new process.
  • Regardless of the tool, remember not to allow the tools to drive your process. Only allow the needs of your team and goals focused on increasing value to the way.
  • Do not attempt to jump from waterfall to prescriptive methodologies based on agile such as SCRUM. Before doing SCRUM, it is crucial to learn to be agile and lean.