Platform engineers presented at KubeCon about how new tools have helped them better hide infrastructure complexity from developers. But a paradox of choice remains.
DETROIT -- The DevOps engineer is dead; long live the platform engineer.
That's the consensus emerging among established enterprises, and with it, the understanding that despite the early ideals of DevOps, developers can't and won't deal with the many nuances of distributed IT infrastructure. Instead, more and more enterprises are embracing platform engineering -- developers using self-service platforms created and curated by operations specialists on their behalf. It's not a new concept, but it's generated more buzz in the IT industry over the last year.
Now, even among early adopters of the platform engineering concept, there's a growing sense that the so-called "golden path" to production could use some more polishing.
"We've had developer teams on the golden path for a few years now. But what we've noticed is that it's a leaky abstraction," Anusha Ragunathan, principal software engineer at Intuit, said during a panel session for press and analysts at this week's KubeCon + CloudNativeCon North America. "Developers still have to know a little bit about Kubernetes and the cloud provider that you're running on. We've been working on developing a cleaner abstraction for the entire process."
Intuit decided to use the Kustomize KRM Functions project for that new abstraction, Ragunathan said during a separate breakout session with Kevin Downey, Intuit staff software engineer. The company chose that project after proof-of-concept testing Helm chart templates, Crossplane's API-based provisioning automation, and the KubeVela software delivery platform project.
KRM Functions won out based on Intuit platform engineering teams' experience using Kustomize, although Helm was also a strong contender, Downey said during the presentation. Crossplane's integrations with Argo CD GitOps workflows and KubeVela as an early sandbox-stage project weren't mature enough for Intuit, he said.
Another KubeCon presenter's platform team did choose Crossplane when it wanted to apply the provisioning automation it built with Kubebuilder to resources outside Kubernetes clusters.
Crossplane "has taken this concept and made it generic" -- applicable to resources outside Kubernetes -- said Jonny Langefeld, staff software engineer at autonomous vehicle company Cruise, during the keynote at KubeCon. "Rather than implementing a Kubebuilder controller yourself, you can just use Crossplane to configure your infrastructure and only expose custom fields [to users]."
Regardless of how they get there, it has become clear that these companies' application developers want less and less to do with the "full stack" behind their deployments, even if they can use self-service to trigger those deployments and can respond to app production issues through a developer portal, the presenters said.
"A Java developer shouldn't have to worry about checking ingress objects [or] version migration issues," Ragunathan said during the panel session. "We're working on a new application specification where we can take the intent of the application developer and transform that into whatever form the back-end platform needs it to be. ... The developer doesn't need to know."
End user panelists at KubeCon discussed the rise of platform engineering. Left to right: Madhusudan C.S., Robinhood; Anusha Ragunathan, Intuit; Natali Vlatko, Wayfair; Krzysztof Adamski, ING; Ricardo Torres, Boeing; panel moderator Taylor Dolezal, CNCF.
Death of the DevOps dream?
"The developer doesn't need to know" is a far cry from the early DevOps mantra of "you build it, you run it" and the purist ideal of obliterating the separation between developer and ops roles. During a press conference here this week, one industry analyst asked Cloud Native Computing Foundation (CNCF) officials what happened to cause that shift.
"Why, after all these years we've all been talking about it, has DevOps not taken off more than it has?" asked Charlotte Dunlap, research director at analysis firm GlobalData.
"If we are hoping that every company would [use] Google-inspired infrastructure for all [applications] -- that is not what has taken off," replied Priyanka Sharma, executive director at CNCF. "Companies have made big strides in becoming technology-first, and that may not have happened with everyone knowing everything in the whole software development and delivery lifecycle. ... I would disagree that it hasn't taken off; it just looks different than we would have imagined."
The answer to this question depends on how DevOps is defined, which has long been up for debate. But by the strictest definition, where one cross-functional team handles building, deploying and all aspects of operating applications in production, DevOps hasn't been widely adopted in the real world, said Ricardo Torres, chief engineer of open source and cloud native at Boeing and a CNCF Governing Board member.
"Very few teams within my company are going to do that," he said. "But using Google's playbook, we have SREs that are building and responsible for platforms and work hand in hand with our development teams. ... We've decreased friction between developers and operators, but there was always going to be a separation."
There's more "wiggle room" in the definition of DevOps for another KubeCon panelist, Natali Vlatko, global lead of the open source program office at online furniture retailer Wayfair. But the platform engineering approach seems to be gaining popularity; in some ways, her company has followed that trend, she said.
"We do want developers to own and operate their own applications and have some of that [operational] knowledge. But we also don't expect every single member of every single team is going to be on call," Vlatko said.
CNCF Executive Director Priyanka Sharma speaks to press and analysts at a KubeCon press conference.
Platform engineers take on the paradox of choice
Platform engineering isn't a bad thing, Dunlap clarified in a separate interview after the panel discussion. But she said her conversations with some enterprise clients have left her with the sense that digital transformation in general has not yet progressed far enough for many enterprises to succeed.
"My starting point for any conversation is, 'What is your DevOps strategy?' And there are still a lot of companies that really should have one that don't," she said.
When such strategies are in place, Dunlap said she still sees implementation failures. She chalks those up to "ongoing operational provisioning complexities in the technologies we're talking about here all week, such as service mesh and observability tools."
We can take the intent of the application developer and transform that into whatever form the back-end platform needs it to be. ... The developer doesn't need to know.
Anusha Ragunathan Principal software engineer, Intuit
At Cruise and Intuit, increased abstraction between developers and the back-end infrastructure -- along with automation tools, including Crossplane -- have reduced toil for platform teams and application developers, presenters said.
But Dunlap's warning about complexity was also echoed in some KubeCon presentations. The breakout session by Intuit engineers focused on the paradox of choice faced by its platform team in picking a new abstraction tool.
"If you were Neo in The Matrix, you were given two pills, a blue pill and a red pill. Easy choice," Ragunathan said during the breakout session. "We were given a million pills. ... There were too many options to actually go figure out a solution for this problem."
Arriving at an answer required not just a proof-of-concept process but also methodical quantitative analysis on a matrix of weighted criteria that assigned numerical ratings to the four finalist tools Intuit evaluated. In the end, Kustomize won out, but not by a huge margin, Downey said.
KubeCon panelists were asked what they felt is missing from the CNCF ecosystem and what they needed help with. The operational complexity issue resurfaced, especially given the growing number of CNCF projects.
"The quality of projects is a spectrum. Kubernetes is solid. Envoy is solid. But on the other end of the spectrum, we're trying to build frameworks internally to scale test to make sure that they actually fit into our use case," said panelist Madhusudan C.S., a software engineer technical lead at financial services firm Robinhood in Menlo Park, Calif. "There's an opportunity here [for CNCF to create] some sort of framework across the board [for projects] to validate that their software scales [before they] release it."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
