Enterprises learn the downsides of DevSecOps metrics
DevSecOps metrics can be helpful, but they can also steer developer and security collaboration in an unproductive direction if they are over-emphasized, practitioners said.
Measuring the performance of developers in securing code is an important part of getting started with DevSecOps, but too much focus on developer metrics can make it harder to mitigate business risk, according to enterprise practitioners.
DevSecOps refers to the practice of incorporating application security into early stages of the software development process, in which developers play a critical role. The practice has become common among enterprise organizations concerned with an ever-rising tide of cyberattacks and high-profile security vulnerabilities, especially over the last two years. Given the pace of software releases demanded by Agile and DevOps and the scale of cloud-native distributed computing, modern applications require security to be built in from the beginning to have any hope of staying ahead of threats, many experts believe.
As enterprise organizations such as Komatsu Australia, a manufacturing company headquartered in New South Wales, make this transition to DevSecOps, they rely on metrics such as mean time to detect (MTTD) and mean time to resolve (MTTR) vulnerabilities and security incidents. These metrics can help establish a foundation for the Agile practice of continuous improvement in the early stages of DevSecOps.
Eric Cheng
"We obviously need to start with a baseline and see where we're at now, and then what makes sense for us to improve," said Eric Cheng, digital architect at Komatsu. "The metrics will tell you, 'Are you heading in the right direction?'"
That baseline measurement will incorporate the Open Web Application Security Project (OWASP) standards organization's DevSecOps Maturity Model, which establishes a three-level process for adopting DevSecOps practices in several areas of app development and IT ops.
"They're a well-known body," Cheng said of OWASP. "For us, where we're at in terms of the journey, I think it's a good starting point."
For specific developer feedback, Komatsu uses static application security testing and software composition analysis dashboards from cybersecurity company Snyk. In addition to MTTD and MTTR, Cheng shows developers trends in a Snyk metric called exposure window for each application, a measure of the total elapsed time between when an issue was detected and when it was resolved.
"We look at, 'OK, you've got some issues starting to creep up in terms of the exposure window,' or 'You've fixed these issues in a short period of time, keep up the great work,'" Cheng said.
This isn't the only tool in Komatsu's DevSecOps toolbox -- developer training, in part via Snyk professional services; ongoing collaboration with cybersecurity managers at the company in the design phase of apps; and security automation within CI/CD pipelines are also important components, Cheng said. But metrics and benchmarks will act as a general guide for a multi-year undertaking to shift security leftat Komatsu.
'It drove the wrong behavior'
For teams further down the maturity path, DevSecOps metrics initially served a similar purpose. But ultimately, IT organizations must look beyond metrics-driven feedback to motivate developers to be security-conscious.
The limits on the effectiveness of DevSecOps metrics as a developer incentive were reflected in the results of a May 2022 DevSecOps survey of 5,000 IT decision makers conducted by DevOps platform vendor GitLab. Security was considered a performance metric for developers by 57% of respondents' organizations, but 56% said it was still difficult to get developers to prioritize fixing code vulnerabilities, and 59% said app vulnerabilities were still most likely to be found by security teams in production.
For retailer Target Corp., an early emphasis on DevSecOps metrics eventually led to unintended consequences, said Jennifer Czaplewski, senior director of cybersecurity, in a presentation at a DevOps Connect event colocated with this year's RSA Conference.
Target calculates a product intelligence (PI) score for each of its 7,000 applications. This score takes into account DevSecOps metrics such as the percentage of endpoint vulnerabilities resolved in accordance with the company's MTTR policies, developer use of internal security services, and whether an application or service has been the cause of a security event.
In 2019, Target's first DevSecOps dashboard showed whether each app's PI score was in the top 10% of Target's apps and services -- or the bottom 20%.
We want to make sure that we're not driving teams to perfection. We're not looking for perfect scores, we're looking for good security health.
Jennifer CzaplewskiSenior director of cybersecurity, Target
"It drove a lot of friendly competition, but our teams got so good at understanding how to secure their applications that more than 10% of all teams had a perfect score, so if you had a perfect score, you still might not crack the top 10%," Czaplewski said in her presentation. "People were frustrated, and it drove the wrong behavior."
Teams are now asked to meet a minimum PI Score, but the comparative information is no longer part of Target's DevSecOps interface, Czaplewski said.
"We want to make sure that we're not driving teams to perfection," Czaplewski said. "We're not looking for perfect scores, we're looking for good security health."
From DevSecOps metrics to a secure culture
Target still expects security awareness among software engineers but has begun to cultivate what it calls Security Ninjas, developers turned security subject matter experts embedded within app dev teams. These experts, which make up approximately 5% of Target's 5,000-developer workforce, act as translators between security and developers. Security Ninjas are also tasked with building two or three threat models each year to create a strategic view of business risk, rather than focusing on specific DevSecOps metrics.
"We've learned over the last three or four years that we really want to pivot to what we should be doing to build a secure culture on a team, not what you should know that hopefully then turns into something," Czaplewski said in her presentation. "If you're a Security Ninja, the first responsibility is to at least have some basic security knowledge ... but also take action to identify and solve security problems."
Ideally, a DevSecOps transformation should bring with it a culture of continuous learning that helps IT pros identify and prioritize strategic risks, even amid rapid changes to cloud-native app security, said Robert Slaughter, CEO at IT defense contractor Defense Unicorns.
Metrics can be useful, but they can make it easy to lose focus, Slaughter said.
"What we stress is daily habits and focus on the outcomes," he said. "Asking, 'Are you becoming more secure?' and instilling that into the individual becomes a much better way to drive the right outcome versus any specific metric that can be hidden, obfuscated or tucked under a rug."
A secure culture is the ideal for Komatsu's Cheng, but it will take time.
"Business teams and the developers they work with -- and they could be vendors -- have their own priorities," he said. "They've got this app that they need to develop, and they need to release it by a certain date, and they've had this mindset and way of working for a long time."
That's starting to change, Cheng said, beginning with product teams embracing tools such as Snyk and asking the vendors they work with to apply security scans to their products as well.
In the meantime, amid a widespread developer and cybersecurity skills shortage, many organizations have begun to use IT automation via self-service DevOps platforms to shore up app security without slowing developers down. Komatsu's use of automated testing tools from Snyk is part of such an effort to automate DevSecOps workflows on behalf of developers.
Educating developers about how security awareness can save them time fixing vulnerabilities in production is an important first step in the DevSecOps process, but "the second part of it is making it as seamless and as painless as possible," Cheng said.
"It just becomes part of the development environment, whether it's Eclipse or Visual Studio, and we can also incorporate [security] in our CI/CD," he said. "For them, it's now just another tool that they've incorporated."
Beth Pariseau, senior news writer for TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Eric Cheng
