DevOps

"DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.^[1] It has developed in response to the emerging understanding of the interdependence and importance of both the development and operations disciplines in meeting an organization's goal of rapidly producing software products and services.^{[2][3][4][5] [6]}

The role of a DevOps professional has similarities with the Chief Engineer at the Toyota Production System^[7]. They have responsibilities on the project success, but no formal authority over different teams involved. This requires strong technical knowledge, to convince managers of the needs. This is possible by the endorsement of the Chief Engineer by the company executives.

History

Predecessors

Many of the ideas (and people) involved in devops came from the Enterprise Systems Management and Agile Infrastructure^[8]/Agile Operations movements.

Devops Days

The term "devops" was coined by Patrick Debois from Belgium^[9] and spread initially through a conference in Ghent called Devops Days held in late 2009.^[10] Since then, there have been Devops Days conferences held in India, the US, Brazil, Australia , Germany and Sweden.^[11]

Devops Methodologies

Illustration showing Devops as the intersection of development (software engineering), technology operations and quality assurance (QA)

When new development methodologies (such as agile software development) are adopted in a traditional organization with separate departments for Dev, IT operations and QA, development and deployment activities, that previously did not need deep cross-departmental integration with IT support or QA, now require intimate multi-departmental collaboration. Devops concerns more than just software deployments, however. It is a set of processes and methods for thinking about communication and collaboration between departments.^[12] Companies with very frequent releases may require a Devops awareness or orientation. Flickr developed Devops capability to support a business requirement of ten deployments per day;^[13] this daily deployment cycle would be much higher at organizations producing multi-focus or multi-function applications. This is referred to as continuous deployment^[14] or continuous delivery ^[15] and is frequently associated with the lean startup methodology.^[16] Working groups, professional associations and blogs have sprouted on the topic since 2009.^{[17][18][19][20]}

The use of Devops integration can have profound results in product delivery, quality testing, feature development and maintenance releases (including the once special but now ubiquitous "hot fix"). Organizations without Devops capabilities can see problems emerge from the “gap” of information shared between development and operations. This occurs as operations request greater reliability and security, developers ask for faster infrastructure responsiveness, while business users ask for more application enhancements and releases made available faster.^{[citation needed]}

The adoption of Devops is being driven by factors such as:

1. Use of agile and other development processes and methodologies
2. Demand for an increased rate of production releases from application and business unit stakeholders
3. Wide availability of virtualized ^[21] and cloud infrastructure from internal and external providers
4. Increased usage of data center automation ^[22] and configuration management tools
5. It has also been suggested a side-effect of the dominant, traditional US management style ("the Sloan model" vs "the Toyoda model")^[23] guarantees the development of silos of automation, thus creating "the Devops gap" that then requires Devops capability to address.

Devops is frequently described as a more collaborative and productive relationship between development teams and operations teams. This improved relationship and collaboration increases efficiency and reduces the production risk associated with frequent changes. There are emerging ROI measurements ^[24] and potential metrics for Devops.

Devops Impact on Application Releases

In many firms, application release events are high stress and high-risk activities involving multiple teams. In a Devops organization, application releases are lower risk for the following reasons:

Illustration showing effect of Agile methodology's dramatic increase in frequency of release events -- often measured in days or weeks -- in contrast to large, infrequent releases, often measured in quarters or years, using traditional development methodologies.

Reduced change scope

Adoption of agile or iterative development, in contrast to the traditional waterfall model, means each release has less change but occurs more often. This creates a smooth rate of progressive application change vs. the large impact effect of rare deployments–each of which contains a large number of changes.

Increased release coordination

The use of a strong release coordinator to bridge the expertise and communications gap between development and operations; employment of collaboration tools such as spreadsheets, conference calls, instant messaging and corporate portals (wikis, SharePoint) to ensure thorough understanding of the change and full cooperation of all involved.

Automation

Comprehensive deployment automation ensures repeatability of deployment tasks and reduces deployment errors.

Adoption of Devops Methodologies

Many organizations divide Development and System Administration into different departments. While Development departments are usually driven by user needs for frequent delivery of new features, Operations departments focus more on availability, stability of IT services and IT cost efficiency. These two contradicting goals create a "gap" between Development and Operations, which slows down IT's delivery of business value.

• Developers are often not concerned about the impact of their code on Operations. They deliver their code without involving Operations into architectural decisions or code reviews.
• Developers lack communicating configuration or environment changes necessary to run the updated code base.
  • Developers apply configuration changes manually to their workstations and do not document each necessary step. Often, coming up with the necessary configuration parameters for software involves experimentation with various parameters. After reaching a working state it is often difficult to identify the minimal steps to reach the working state.
  • Developers tend to use a tool set optimized for rapid development: Fast feedback on code changes, low memory consumption of runtime environment, etc. This tool set is very different from the target runtime environment in Operations where stability and performance trump flexibility requirements.
  • As Developers work on desktop computers they tend to use Operating Systems optimized for desktop use. The runtime environment is usually running a server operating system.
  • In development, the systems run locally on the developers workstation. In Operations, the system is often distributed amongst various servers like web server, application server, database server, etc.
• Development is driven by functional requirements usually directly related to business needs
• Operations is driven by non-functional requirements like availability, stability, performance, etc.
  • Operations tries to minimize risk for delivering on non-functional requirements by avoiding change
  • If frequent change is avoided, but the amount of necessary change stays constant, every change will be bigger
  • Bigger changes involve more risk, as more areas are affected by any change
• In trying to avoid change, Operations slows down the flow of new features to production and therefore slowing down Development's ability to deliver features
• Operations might not be fully aware of the application's internals making it hard to correctly define the runtime environment and update procedures
• Development might not be fully aware of the runtime environment making it hard to correctly adapt the code accordingly

Needs

• More and smaller changes–mean less risk
• Giving developers more environment control
• Giving infrastructure more application-centric understanding
• Clearly articulating simple processes
• Automating as much as possible
• Collaboration between dev and ops

In summary, as companies seek to streamline the cumbersome transition between development and operations, they typically confront 3 different types of problems:

Release Management Problems

Companies with release management problems are looking for better release planning than spreadsheets. They want an easy way to understand release risks, dependencies, stage gates adherence and an ensure compliance.

Release/Deployment Coordination Problems

Teams with release/deployment coordination problems are focused on better execution of release/deployment events. They want better tracking of discrete activities, faster escalation of issues, documented process control and granular reporting.

Release/Deployment Automation Problems

Companies with release/deployment automation problems usually have existing automation but want to more flexibly manage and drive this automation - without needing to enter everything manually at the command-line. Ideally, this automation can be invoked by non-operations resources in specific non-production environments.

One way to start streamlining release process is to identify which of the above problems is the overall team's highest priority.

Release coordinator

A relatively new role in enterprise IT which is primarily tasked with coordinating deployments of enterprise software to pre-production environments. The need for the release coordinator has been driven by:

1. The need to fill the Devops "gap"
2. Increased infrastructure complexity - multiple layers and platforms which form web applications
3. Growth in rate of releases - due to agile and iterative development
4. Distributed teams - globally deployed, outsourced and hybrid development, testing and infrastructure teams

The release co-ordinator role (also referred to as a deployment or integration co-ordinator) has emerged from the release management or release engineering teams. This role is similar to an air traffic controller—performing real time co-ordination activities across diverse teams to achieve a group goal (safe landing and take-off) using shared resources (airspace, flight paths, airport runways, and terminal gates).

Release co-ordination contrasts with release management, which is often focused on planning and reporting on software changes, in order to control the release of specific application changes into production. Release engineering is concerned with the systematic and technical work related to building and deploying code into environments.

Change management is the infrastructure discipline for tracking all types of changes in the enterprise IT environment—including both application and infrastructure changes. Change management is a core part of ITIL v3.

References

1. ^ Pant, Rajiv (2009-03-17). "devops". Organizing a Technology Department. http://www.rajiv.com/blog/2009/03/17/technology-department/. 
2. ^ Samovskiy, Dmitriy (2010-03-02). "The Rise of devops". Fubaredness Is Contagious. http://somic.org/2010/03/02/the-rise-of-devops/. 
3. ^ Edwards, Damon. "What is devops?". http://dev2ops.org/blog/2010/2/22/what-is-devops.html. 
4. ^ Vambenepe, William. "Steve Ballmer gets Cloud". http://stage.vambenepe.com/archives/1393. 
5. ^ Lyman, Jay. "devops mixing dev, ops, agile, cloud, open source and business". 451 CAOS Theory. http://blogs.the451group.com/opensource/2010/03/03/devops-mixing-dev-ops-agile-cloud-open-source-and-business/. 
6. ^ Debois, Patrick. "Devops: A Software Revolution in the Making?". Cutter IT Journal. http://www.cutter.com/promotions/itj1108.html. 
7. ^ Liker, Jeffrey (2003). The Toyota Way. McGraw-Hill; 1 edition (December 17, 2003). ISBN 0071392319. 
8. ^ Nasrat, Paul. "Agile Infrastructure". InfoQ. http://www.infoq.com/presentations/agile-infrastructure. Retrieved 31 March 2011. 
9. ^ Debois, Patrick. "devops Cafe Episode 12". devops Cafe. http://devopscafe.org/show/2010/9/15/episode-12.html. Retrieved 31 March 2011. 
10. ^ Debois, Patrick. "Devops Days Ghent 2009". DevopsDays. http://www.devopsdays.org/events/2009-ghent/. Retrieved 31 March 2011. 
11. ^ Debois, Patrick. "Devops Days". Devops Days. http://www.devopsdays.org/. Retrieved 31 March 2011. 
12. ^ Turnbull, James. "What Devops means to me…". http://www.kartar.net/2010/02/what-devops-means-to-me/. 
13. ^ "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr". http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr. 
14. ^ "SAM SIG: Applied Lean Startup Ideas: Continuous Deployment at kaChing". SVForum. http://www.svforum.org/index.cfm?fuseaction=Calendar.eventDetail&eventID=13703. 
15. ^ Humble, Jez. "Why Enterprises Must Adopt Devops to Enable Continuous Delivery". Cutter IT Journal. http://www.cutter.com/promotions/itj1108.html. 
16. ^ "Applied Lean Startup Ideas: Continuous Deployment at kaChing". http://www.slideshare.net/pascallouis/applied-lean-startup-ideas-continuous-deployment-at-kaching. 
17. ^ "Devops Group". LinkedIn. http://www.linkedin.com/groups?mostPopular=&gid=2825397. 
18. ^ "Devops Days 2009 Conference". http://www.devopsdays.org/ghent09/programme/. 
19. ^ Edwards, Damon. "Devops Meetup Recap". http://dev2ops.org/blog/2010/4/26/devops-meetup-recap.html. 
20. ^ Lyman, Jay. "Devops mixing dev, ops, agile, cloud, open source and business". 451 CAOS Theory. http://blogs.the451group.com/opensource/2010/03/03/devops-mixing-dev-ops-agile-cloud-open-source-and-business/. 
21. ^ "Virtual Infrastructure products: features comparison". Welcome to IT 2.0: Next Generation IT infrastructures. http://www.it20.info/misc/virtualizationscomparison.htm. 
22. ^ Ellard, Jennifer. "Bringing Order to Chaos through Data Center Automation". Information Management. SourceMedia, Inc.. http://www.information-management.com/infodirect/20071026/10000120-1.html. 
23. ^ Debois, Patrick. "The leaning of life - History of the Silos". http://www.jedi.be/blog/2010/06/07/the-leaning-of-life/. 
24. ^ Booth, David. "How to Measure the Effects of Development + Operations improvements, an OpenSpace conversation". http://www.zeroturnaround.com/blog/how-to-measure-the-effectiveness-of-implementing-devops.