Fresh crop of IT automation tools targets platform engineers
As enterprise IT shifts from full-stack DevOps to platform engineering, IT automation tools reinvented for the Kubernetes era can reduce toil for operators.
These teams of IT operations specialists -- increasingly the norm among enterprises delivering microservices applications to cloud-native infrastructure -- create self-service interfaces that hide the complexities of DevOps platforms from application developers. But as they take on the burden of that complexity themselves, platform engineers are also turning to new IT automation tools that can ease the load.
Simply letting go of the notion of the "full-stack developer" and allowing developers and operators to stay in their own lanes can help platform engineers reduce toil, according to a presentation by Anusha Ragunathan, principal software engineer at Intuit, at the recent KubeCon + CloudNativeCon North America conference.
"We talk a lot about the velocity of developers, but using a [self-service] abstraction layer also lets the platform team move much faster," Ragunathan said during the presentation. "They don't have to worry about spending time making sure developers are properly managing Kubernetes or rolling out [something like] a new service mesh when they have to expose the developers directly to it."
We talk a lot about the velocity of developers, but using a [self-service] abstraction layer also lets the platform team move much faster.
Anusha RagunathanPrincipal software engineer, Intuit
A Danish bank's platform engineering team found that the open source Backstage tool it uses to create a self-service interface for developers also lightens the IT governance load with built-in service ownership tracking.
"We were recently tasked by our auditors to do software classification and asset management, and we have that in Backstage," said Kasper Nissen, lead platform architect at Lunar, in an interview at KubeCon. "From the auditor's perspective, we have everything categorized as, 'this ownership is defined in Backstage as this team in this domain, with this criticality attached to it.'"
Pulumi targets platform engineers
As these discussions continued post-KubeCon, infrastructure-as-code specialist Pulumi rolled out a new Deployment API SaaS offering in tech preview this week, targeting platform engineers with GitOps as a service.
Deployment API is based on Pulumi's Automation API, a REST API interface for the vendor's infrastructure-as-code libraries written in popular programming languages such as Python and Java. With Deployment API, Pulumi's managed service is triggered by a code push to GitHub repositories, automatically creating and previewing back-end infrastructure stacks based on Kubernetes and cloud infrastructure-as-a-service resources on AWS and Microsoft Azure.
"It's sort of like GitHub Actions, but for infrastructure," said Joe Duffy, founder and CEO at Pulumi, in an interview this week.
Platform engineers and developers can also use GitHub Actions itself to create similar workflows, but Pulumi's Deployment API replaces the manual setup for that kind of process, Duffy said.
"You've got to put together a few dozen lines of YAML, you have to go and give GitHub your cloud credentials -- it takes a bit of setup for a very clear common workflow," he said. With the Deployment API service, "you don't have to go re-create the wheel on things that are pretty standard configurations."
The Deployment API can be triggered via command-line interface or REST API calls for other code repositories, and Git-push-to-deploy support is planned for Azure DevOps, GitLab and Bitbucket. Pulumi also integrates with about a dozen CI/CD pipeline tools, including CircleCI and Jenkins.
One engineer whose SRE team uses Pulumi's Automation API said he must wait until that push-to-deploy feature is built in for Azure DevOps to take full advantage of the new Deployment API, but the concept is appealing.
Anusha Ragunathan, principal software engineer at Intuit, gives a presentation about platform engineering at KubeCon.
"It could allow us to scale better by offloading the actual running of deployment code, because we pay by the parallel job for Azure DevOps," said Dan Swartz, principal software engineer at Altana AI, a supply chain automation company in New York, in an interview this week. "There's also [another scenario] when we just want to update configuration settings or Pulumi YAML files … that's where this is going to knock it out of the park, because [right now] we don't have that sort of nice, clean workflow [for that], where you commit code and then it launches a preview for you."
Pulumi emerged with HashiCorp's Terraform infrastructure-as-code tool in its competitive sights, adding a general programming language interface on infrastructure-as-code tools, rather than requiring developers to learn YAML or a domain-specific language such as HashiCorp's HCL. Now, according to one analyst, the Deployment API amounts to a foil for another HashiCorp product, the Waypoint deployment automation service launched in public beta on the HashiCorp Cloud Platform in early October.
"They both have the same ambition of abstracting away the actual infrastructure components from the developer and allowing the operator to create the templates and automate the process," said Torsten Volk, an analyst at Enterprise Management Associates, in an interview this week. But rather than use YAML or HCL, "the key with Pulumi is that developers can just have it in their IDE, create a pull request and that gets merged, and the Automation and Deployment API take care of all the rest."
Kubernetes management tools weave into cloud-native platforms
Platform engineers at KubeCon also shared how they improved IT automation using open source utilities built around Kubernetes, the base IT infrastructure layer that underpins most DevOps platforms. Several presenters and attendees this year referenced Crossplane, a Cloud Native Computing Foundation (CNCF) project that uses the Kubernetes API to orchestrate resources outside of Kubernetes clusters.
Lunar's Nissen said his platform engineering team is looking into Crossplane but also plans to use Cluster API to standardize cross-cloud Kubernetes deployments.
"Managed Kubernetes is really annoying to be running on three different clouds, because they want their own flavor of the Kubernetes cluster in each cloud," he said. "We really want to have a way to align across jobs just to minimize the operational burden … [to have] one description or specification saying, 'This is how we want our clusters, with these core components, and it's the same across clouds.'"
A small team of three full-time and one part-time platform engineers that serve 45 developers at Kognic, a Swedish SaaS provider, used another CNCF project, cert-manager, to eliminate recurring manual cycles of SSL certificate updates and rolling Kubernetes cluster restarts, according to a KubeCon presentation.
"Whenever we were renewing our certificates … every 60 days … we first had to use a Shell script to create the new certificate … then manually edit the secret in Kubernetes to contain the new certificate," said Jessica Andersson, product area lead for engineering enablement at Kognic, during the presentation. "That's all great, except the Kubernetes deployments don't pick up the changed secret unless they are redeployed, so we had to do a rolling restart of all the deployments. It was a pain."
Cert-manager eliminated that entire process, Andersson said.
While platform engineering and a focus on developer experience have begun to catch on for the Kubernetes generation, some early adopters of PaaS see history repeating, and hope it can catch up to past platform movements quickly.
"We previously had the Open Service Broker API framework [with Cloud Foundry], and now we have things like Crossplane driving toward accepted frameworks and standards that plug into the Kubernetes ecosystem," said Greg Otto, executive director of cloud services at cable provider Comcast, in an interview at KubeCon. "But it's got to be more than just buzzword bingo. … For us, all of our decision-making is based on, 'How do the benefits of this accrue to developer experience and business outcomes?' I will trade complexity for simplicity for my platform team, but only if it comes with a big reward and payback for developer experience."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
