What is DevOps? The ultimate guide

How does DevOps work?

DevOps is a methodology meant to improve work throughout the software development lifecycle. You can visualize a DevOps process as an infinite loop, comprising these steps: plan, code, build, test, release, deploy, operate, monitor and -- through feedback -- plan, which resets the loop.

Ideally, DevOps means that an IT team writes software that perfectly meets user requirements, deploys without any wasted time and runs optimally on the first try. Organizations use a combination of culture and technology to pursue this goal.

To align software to expectations, developers and stakeholders communicate about the project, and developers work on small updates that go live independently of each other.

To avoid wait times, IT teams use CI/CD pipelines and other automation to move code from one step of development and deployment to another. Teams review changes immediately and can enforce policies to ensure releases meet standards.

It's easy to write software quickly; writing software that works is another story. To deploy good code to production, DevOps adherents use containers or other methods to make the software behave the same way from development through testing and into production. They deploy changes individually so that problems are traceable. Teams rely on configuration management for consistent deployment and hosting environments. Problems they discover in live operations lead to code improvements, often through a blameless post-mortem investigation and continuous feedback channels.

Developers might support the live software, which puts the onus on them to address runtime considerations. IT operations administrators might be involved in the software design meetings, offering guidance on how to use resources efficiently and securely. Anyone can contribute to blameless post-mortems. The more these specialists collaborate and share skills, the more they can foster a DevOps culture. 

What problems does DevOps solve?

Each company faces its own challenges, but common problems include releases that take too long, software that doesn't meet expectations and IT that limits business growth.

Without wait times, manual processes and lengthy reviews, a DevOps project moves faster from requirements to live software. Shorter cycle times can keep requirements from shifting so that the product delivers what customers want.

DevOps solves communication and priority problems between IT specializations. To build viable software, development teams must understand the production environment and test their code in realistic conditions. A traditional structure puts development and operations teams in silos. This means developers are satisfied when their code delivers functionality -- and if the release breaks in production, it's up to the operations team to fix the problems.

With a DevOps culture, developers don't resort to the "It worked on my machine" response when a problem arises. Changes rolled out to production are small and reversible. Plus, the whole team understands the changes, which greatly simplifies incident management.

With a faster process from idea to live software, companies can capitalize on market opportunities. In this way, DevOps provides a competitive advantage for businesses.

The evolution of DevOps

Patrick Debois, a software development consultant, is credited with creating the term DevOps in 2009 by naming a conference DevOps Days. DevOps addressed a shortcoming of the Agile software development methodology: Iterative, rapid code development did not necessarily lead to iterative, rapid code deployment.

Concurrent with the Agile push deeper into operations, IT administrators chafed against sometimes laborious and overly complex change management steps in the ITIL framework. ITIL champions stable, reliable and predictable IT, while Agile advocates for collaboration and change. DevOps struck a chord with people on both sides. In fact, organizations can do both ITIL and DevOps, especially if they embrace the cloud.

The concept of DevOps was then popularized with the book The Phoenix Project in 2013. The Phoenix Project uses a fictional narrative to illustrate endemic problems and help IT managers understand the concepts and benefits of collaboration and shared technologies.

Early proponents of DevOps included these key players:

• Debois and collaborator Andrew Clay Shafer;
• The Phoenix Project authors Gene Kim, Kevin Behr and George Spafford;
• Paul Hammond and John Allspaw, influential early adopters at Flickr;
• continuous delivery pioneers Jez Humble and Dave Farley;
• containerization advocate John Willis; and
• State of DevOps Report organizers, including Alanna Brown and Nicole Forsgren.

As DevOps became popular, organizations formalized DevOps approaches. Retailer Target originated the DevOps Dojo training method, for example. Vendors touted DevOps-enabling capabilities in tools, from collaboration chatbots to CI/CD suites built into cloud services. "DevOps engineer" soon emerged as a job title.

DevOps continues to evolve, as artificial intelligence surfaces to aid in everything from code creation to incident management. AI for DevOps (or AIOps) means smarter automation and even shorter wait times, even seamless translations from business need to technological offering -- but plenty of barriers remain before this becomes reality.

While DevOps has achieved mainstream status, not all adopters are full DevOps converts. Many rely on a DevOps approach for revenue-generating IT projects, where they see a return on investment in the leading-edge tooling and skills. For many internal IT services that are relatively stable and mature, DevOps does not offer significant benefits.

Methodologies, principles and strategies

DevOps is associated with Agile software development because Agile practitioners promoted DevOps as a way to extend the methodology into production. The approach has even been labeled a counterculture to the IT service management practices championed in ITIL. DevOps does not have an official framework.

To hone their strategies, organizations should understand the related contexts of DevOps, Agile and Waterfall development, site reliability engineering (SRE) and SysOps, and even the variations within DevOps.

DevOps vs. Waterfall development. Waterfall development comprises a series of steps and gates in a linear progression to production. Its phases are requirements, analysis, design, coding and implementation, testing, operation and deployment, and maintenance. In Waterfall teams, development tests new code in an isolated environment for quality assurance (QA) and -- if requirements are met -- releases the code to operations for use in production. IT operations deploys multiple releases at once, with extensive controls. Support is operations' responsibility. Waterfall approaches engender long waits between software releases. Because development and operations teams work separately, developers are not always aware of operational roadblocks that prevent code from working as anticipated.

The DevOps model aligns development, QA and IT operations efforts with fewer gates and more continuous workflow. For example, some of the operations' team responsibilities shift left in the app delivery pipeline to the development team. IT operations provides feedback for code improvements. Rather than gated steps, DevOps relies on continuous development, continuous integration, continuous delivery and continuous monitoring processes.

DevOps vs. Agile development. Agile is a software development approach defined in the Agile Manifesto. Agile teams focus on incremental and rapid cycles of code creation and delivery, referred to as sprints. Each sprint iterates upon the last, which makes the software highly flexible and adaptable to changing requirements. It is possible for the original vision of a project to be lost through this cycle.

DevOps arose from Agile's success at improving development speed, and the realization that disconnects between development and operations teams -- as well as between IT and the business side of the organization -- significantly hindered the Agile software's delivery to users.

In an Agile-only workflow, development and operations teams have separate objectives and leadership. When an organization uses DevOps and Agile together, both development and operations teams manage code throughout the software development lifecycle. While Agile work is often formalized with a framework, such as Scrum, DevOps does not have a framework.

DevOps vs. SRE. Site reliability engineering arose concurrently with Agile and DevOps. Started in the early 2000s at Google, it is essentially a programming- and automation-focused approach to the software development lifecycle. Problems should be solved in a way that prevents them from occurring again. Rote tasks should be minimized.

The SRE toolbox closely matches that for DevOps. Both disciplines aim for continuous improvement. SRE and DevOps engineers seek to abolish silos between development and operations. While DevOps also can extend to business stakeholders, SRE typically stays within the confines of IT processes. 

DevOps vs. SysOps. SysOps typically denotes that an IT administrator or IT team manages production deployment and support for a large distributed application, such as a SaaS product. As with DevOps adopters, SysOps teams should be versed in cloud computing and automation, as well as other technologies that enable applications to perform well at a large scale. SysOps teams troubleshoot IT outages and incidents, monitor for performance problems, enforce security rules and optimize operations.

They also focus on high availability, fault tolerance, security and performance just like other IT admins. While SysOps professionals are likely to use some development tools and understand development processes, their work is not as enmeshed with development as in a DevOps job. However, SysOps roles can exist within DevOps and SRE organizations.

DevSecOps vs. BizDevOps vs. GitOps. Some organizations broaden the scope of DevOps to include other roles or departments. In DevSecOps, security planning, scans, testing and reviews occur continuously throughout the DevOps loop. BizDevOps focuses on connecting executives, application owners and other business stakeholders to the technical team, which develops, tests and supports the software. While more collaboration is arguably better than less, these collaborators must share effective, timely and precise input.

Another variation on DevOps, or a different faction of the same movement, is GitOps. Named for its focus on the eponymous repository and version control technology, GitOps espouses declarative source control over application and infrastructure code. Everything about the software -- from feature requirements to the deployment environment -- comes from a single source of truth.

DevOps benefits and challenges

DevOps benefits include the following:

• fewer silos and increased communications between IT groups;
• faster time to market for software;
• rapid improvement based on feedback;
• less downtime;
• improvement to the entire software delivery pipeline through builds, validations and deployment;
• less menial work, thanks to automation;
• streamlined development processes through increased responsibility and code ownership in development; and
• broader roles and skills.

However, DevOps challenges abound:

• organizational and IT departmental changes, including new skills and job roles;
• expensive tools and platforms, including training and support to use them effectively;
• development and IT tool proliferation;
• unnecessary, fragile or unsafe automation;
• scaling DevOps across multiple projects and teams;
• riskier deployment due to a fail-fast mentality and job generalization vs. specialization;
• regulatory compliance, especially when role separation is required; and
• new bottlenecks.

DevOps adoption

DevOps transformations don't happen overnight. Many companies start with a pilot project -- a simple application where they can get a feel for new practices and tools. For large-scale DevOps adoption, try moving in stages. 

Initially, DevOps can mean a commitment from development and IT operations teams to understand the concerns and technological boundaries that exist at each stage of the software project. Agree upon KPIs to improve, such as shorter cycle times or fewer bugs in production. Lay the groundwork for continuous processes by communicating across job roles.

Evaluate the existing tools for development and IT operations. Identify shortcomings, such as a step that's always handled manually or a tool without APIs to connect with other tools. Consider standardizing on one DevOps pipeline for the whole company. With one pipeline, team members can move from one project to another without reskilling. Security specialists can harden the pipeline, and license management is eased. The tradeoff with this approach is that DevOps teams give up the freedom to use what works best for them. 

The organization now has a DevOps mindset in place, metrics to track for success and capable tools. Focus on best practices, knowledge sharing and skills development to continue improving. Optimize tooling and technologies, identifying roadblocks and gaps that affect your KPIs.

Organizations can use the DevOps maturity model as a guide to adoption:

• Initial. Teams are siloed; work is reactive and done with ad hoc tool and process choices.
• Defined. A pilot project defines a DevOps approach, basic processes and tools. It is a proof of concept.
• Managed. The organization scales up DevOps adoption with lessons learned from the pilot. The pilot's results are repeatable with different staff members and project types.
• Measured. With processes and tools in place, the teams share knowledge and refine practices. Automation and tool connectivity increase, and standards are enforced through policies.
• Optimized. Continuous improvement occurs. DevOps might evolve into different tool sets or processes to fit use cases. For example, customer-facing apps have a higher release frequency, and financial management apps follow DevSecOps practices.

DevOps tools

DevOps is a mindset, not a tool set. But it's hard to do anything in an IT team without the right tools. In general, DevOps practitioners rely on a CI/CD pipeline, containers and cloud hosting. Tools can be open source, proprietary or supported distributions of open source technology. 

Code repositories. Version-controlled source code repositories enable multiple developers to work on code. Developers check code out and in, and they can revert to a previous version of code if needed. These tools keep a record of modifications made to the source code. Without tracking, developers may struggle to follow which changes are recent and which versions of the code are available to end users.

In a CI/CD pipeline, a code change committed in the version-control repository automatically triggers next steps, such as a static code analysis or build and unit tests. Tools for source code management include Git and GitHub.

Artifact repositories. Source code is compiled into an artifact for testing. Artifact repositories enable version-controlled, object-based outputs. Artifact management is a good practice for the same reasons as version-controlled source code management. Examples of artifact repositories include JFrog Artifactory and Nexus Repository.

CI/CD pipeline engines. CI/CD enables DevOps teams to frequently validate and deliver applications to the end user through automation during the development lifecycle. The continuous integration tool initializes processes so that developers can create, test and validate code in a shared repository as often as needed without manual work. Continuous delivery extends these automatic steps through production-level tests and configuration setups for release management. Continuous deployment goes a step further, invoking tests, configuration and provisioning, as well as monitoring and potential rollback capabilities. Common tools for CI, CD or both include Jenkins, GitLab and CircleCI. 

Containers. Containers are isolated runtimes for software on a shared OS. Containers provide abstraction that enables code to work the same on different underlying infrastructure from development to testing and staging, and then to production. Docker is the most well-known containerization software, while Microsoft offers specific Windows container options. Container orchestrators -- such as Kubernetes and commercial Kubernetes distributions Red Hat OpenShift and Amazon Elastic Kubernetes Service -- deploy, scale and maintain containers automatically.

Configuration management. Configuration management systems enable IT to provision and configure software, middleware and infrastructure based on a script or template. The DevOps team can set up deployment environments for software code releases and enforce policies on servers, containers and VMs through a configuration management tool. Changes to the deployment environment can be version controlled and tested, so DevOps teams can manage infrastructure as code. Configuration management tools include Puppet and Chef.

Cloud environments. DevOps organizations often concurrently adopt cloud infrastructure because they can automate its deployment, scaling and other management tasks. AWS and Microsoft Azure are among the most used cloud providers. Many cloud vendors also offer CI/CD services.

Monitoring. Additionally, monitoring tools enable DevOps professionals to observe the performance and security of code releases on systems, networks and infrastructure. They can combine monitoring with analytics tools that provide operational intelligence. DevOps teams use these tools together to analyze how changes to code affect the overall environment. Choices are wide-ranging, but include New Relic One, Dynatrace, Prometheus, Datadog and Splunk.

Cloud-based DevOps pipelines. Public cloud providers offer native DevOps tool sets to use with workloads on their platforms. An incomplete list includes AWS CodePipeline and CloudFormation, Azure DevOps and Pipelines, and Google Cloud Deployment Manager. Cloud adopters have the option to use these pre-integrated services or run third-party tools. For example, an organization can use HashiCorp Terraform or CloudFormation to make infrastructure-as-code templates for its AWS workloads.

As-a-service models. Lastly, DevOps as a service is a delivery model for a set of tools that facilitates collaboration between an organization's software development team and the IT operations team. In this delivery model, the provider assembles a suite of tools and handles the integrations to seamlessly cover the overall process of code creation, delivery and maintenance.

DevOps skills

DevOps is often said to be more of a philosophy or collaborative IT culture, rather than a strictly defined job description or skill set. Because the area is so broad, DevOps positions suit IT generalists better than specialists.

The role of DevOps engineer does not fall along one career track. Professionals can enter into the position from a variety of backgrounds. For example, a software developer can gain skills in operations, such as configuration of the hosting infrastructure, to become a DevOps engineer. Similarly, a systems administrator with coding, scripting and testing knowledge can become a DevOps engineer.

Many DevOps job listings call for container, cloud and CI/CD knowledge, as well as soft skills. A DevOps engineer might also need to change processes and solve organizational problems to achieve business outcomes.

Other titles often found in DevOps organizations include the following:

• infrastructure developer;
• site reliability engineer;
• build and release engineer;
• full-stack developer;
• automation specialist; and
• CI/CD platform engineer.

Most entry-level DevOps jobs require a degree in computer science or a related field that covers coding, QA testing and IT infrastructure components. Higher-level positions may require advanced degrees in systems architecture and software design. People on this career path should also expand their knowledge via DevOps books, and connect with other members of the community through blogs and conferences.

There are no industrywide recognized DevOps certifications as there are for formalized frameworks such as ITIL. IBM Red Hat offers DevOps certification, while DevOps Institute has vendor-neutral options for levels from foundational knowledge to expert.