Jenkins – The Juggernaut of Continuous Delivery

09 May 2016

Los Hermanos Jenkins

Today, continuous delivery (CD) is en-vogue. As companies adopt CD, they are faced with a choice on the tools to establish their delivery pipelines. Jenkins is often the obvious choice and best of all it is likely used by the engineering division already. This blog looks at various data points to paint a picture of the pervasiveness of Jenkins for CD.

Blog When I started at CloudBees a half a dozen years ago, CD wasn’t

in the lexicon – in fact I named “Jenkins Pipelines” feature to be “Workflow” – pipelines is much better! Asking customers about their Dev-to-Production software delivery strategy was often met with a quizzical look.

The four-minute-mile barrier of software delivery was delivering software maybe once-a-month and that barrier has long been broken. Fabled stories of Amazon, Etsy and FaceBook delivering software multiple times a day have set a new bar that every company aspires to.

Google trends shows a continuous (pun intended) rise in the interest of CD. Trends

Since data science is the hottest trend in the market now, I have used data science to provide for a relative scale.Data science

It has been clear that Jenkins is the de-facto CI leader in the market. The ZeroTurnAround survey shows that Jenkins is most adopted CI server in the market with 70% of the market share.

ZeroTurnaround

(source: Zeroturnaround)

Active (ones that report active usage) Jenkins installations count is 127k and growing. Active Installations

(source: Jenkins stats site)

My guess is that non-reporting installations must be 2-4 times the active installations. Typically, a Jenkins installation supports anywhere between 5-100 people (dev, qa etc) – conservatively this could mean anywhere from 1M+ people using Jenkins (counting 10 developers per active installation).

Let’s look at data from Jenkins Survey in 2013 and 2015 to see some trends.

Jenkins for Operations In the 2 years between the surveys, the

usage of Jenkins for operations activities has grown by 9%.

Jenkins for  Operations

(source: Jenkins 2013 survey) Jenkins Jobs (source: Jenkins 2015 survey)

Jenkins for Deployment The story for deployment is even better with

60% of Jenkins users using the tool for deployment. Note: this number is likely higher as in some cases activities under “release, operations” overlap into the deployment zone. About 70% of those doing deployment are either doing continuous delivery (manual flag to deploy to production) or continuous deployment (auto-deploy).

Deployment

(source: Jenkins 2015 survey infographic)

If we take the 127k active installation number and apply it here (70% of 127k)- we have about 88k installations using Jenkins for deployment. Is Jenkins the #1 CD tool in the market? Looks like it! Can other tools make this claim – doubt it!

83% of deployments happen multiple times a week! Talk about breaking the 4 minutes mile barrier! Delivering is no longer stressful!

Multiple deployments

Building modern pipelines with Docker

Last year, the community churned out a number of plugins to help build pipelines with Docker. It is no surprise to now that Jenkins with Docker is a common use case. In other story, Jenkins is the most popular Docker image on Docker Hub with 5M+ (yes million) pulls.

Increase in the number of plugins

A key reason for Jenkins’ success has been the number of plugins that connect into existing tools. After all, if an organisation brings in a tool to do CD – it has to work with existing tools in the market.

Jenkins has about 1100+ plugins that connect to almost any tool out there. The number of installations of these plugins is north of 4.5M (yes million – it isn’t a typo). Usage

No wonder, the actual usage of Jenkins within an organisation keeps increasing with 89% respondents saying that their usage in Jenkins increased.

Actual usage

Mission Critical Jenkins

Given that an organisations’ entire software delivery chain is dependent on Jenkins, it is no surprise Jenkins is considered mission critical. Between 2013 and today the growth is about 9%. 92% of respondents say that Jenkins is mission critical – not a surprise, given that at-least 50% of developers in organisation are using Jenkins. A colleague asked a question in talk recently “What would happen if Jenkins goes down?”, only to be met with nervous laughter.

Mission critical

A testament to Jenkins is that 89% of teams are satisfied with Jenkins. Satisfaction

This satisfaction shows up in numerous awards!

Awards

Increase in the number of jobs – Jenkins jobs

In the last 3 years, the number of jobs created in Jenkins has grown almost 3 times from ~2.25M to 7M. Number of jobs

Increase in the number of jobs – human jobs

With the increase in Jenkins usage, its usage in CD, there is no surprise that people in the “DevOps” role using Jenkins has more than doubled to 48%. Number of human jobs

This is somewhat borne in the job postings in the market as well. There seems to be a growth in the specialised role that is focused on continuous delivery. There is a corresponding growth in Jenkins jobs – doubling in the last 2 years!

Job Role

Job Trend

To make sense of the data, I added data scientist job in there for fun 😃 and was more than presently surprised to see that there are 3x jobs in continuous delivery. Job trend 2

So what does this mean for you?

The Jenkins community has worked in the last two years to build the Jenkins Pipeline feature that brings native support for Pipelines. So if you are considering walking the continuous delivery path, ask your engineers to look at Jenkins Pipelines – most likely they are already using Jenkins and Pipelines is just a plugin away. Join the 79% of Jenkins users who are going to move to pipelines in the next 12 months.

Pipeline plugin

I have given an overview in an earlier blog and more recently the Jenkins community did a virtual conference that goes into details here. These should be good starts.

Summary

Unicorn companies have proven to the world that delivering software continuously is possible. Regular companies are now well on the path of delivering software continuously. A significant number of Jenkins users are now using it to deliver software continuously. Given, Jenkins’ large reach, it is not a stretch to say that it is the #1 tool in the market to do CD.

Opinions here are all mine and do not reflect that of my employers.

Reference

Escape the Iron Triangle – ship better software faster and cheaper with Jenkins Pipelines

01 May 2016

continuos software delivery

Delivering software continuously, in other words “keeping software in an always shippable state” is a challenge for companies. Recently released Jenkins 2.0, is focussed on solving this problem through Pipelines. This blog gives a quick overview of key benefits of Jenkins Pipelines.

Executive Summary

Software delivery in any organisation is a complex process that spans tools, is hand-stitched and fragile. In-efficiencies and failures in the process have a make or break effect on an applications success in the market place.

Jenkins Pipelines automates the process through code, connecting into the gargantuan number of tools of used by companies for software delivery. Once modelled, teams get insights into their delivery performance – to optimise their delivery times. Pipelines survive infrastructure failures and thus teams can deliver applications on-time, every time.

A month early due to a stream lined process is a month’s revenue while a month late in delivery due to infra failures is a month that competition can steal your customers!

Why Continuous Delivery (CD)?

Accelerating the pace of software delivery mandates code in an always shippable state and Continuous Delivery is the process that takes software to the always shippable state.

“Thou shalt choose between good, fast or cheap” said the Iron Triangle of software delivery. Continuous Delivery upends that case to help teams deliver good software, faster and cheaper .

Why Jenkins?

Software goes through various phases (build, test, deploy) and interacts with a multitude of tools (junit, sonar, nexus etc) on its way to production. Jenkins is the orchestrator that drives the entire pipeline and interacts with each of these tools. Jenkins’ strength is that it has 1200 (and counting) plugins that let you interact with any of the tools that are in your organisation.

Jenkins Pipelines

Jenkins 2.0 has introduced a key area called Pipelines. With Pipelines, organisations can define their delivery pipeline through a DSL (Pipeline-as-code). Pipelines, thus, can be versioned, checked into source and easily shared within an organisation.

Pipelines model YOUR delivery process rather than straight-jacketing you in an “opinionated” process

Typical delivery value stream within organisations have one commonality – atypicality. Most of the delivery processes are dissimilar unlike the canonical “build, test, deploy” examples that you see in examples. Here is an example from an earlier blog from Viktor Farcic on a delivery pipeline.

Delivery pipeline

The Pipeline DSL helps you capture complex process requirements through code – thus you can try-catch on deployment failures, loop through deployments, run tests in parallel. This is immensely powerful – the DSL brings the power of a programming language (groovy) to do so. At the same time, the DSL is simple enough to capture simple cases easily without having to write groovy code. DSL encourages the DRY principle – capture common patterns in functions, keep them in a global library so that new applications can build on these functions. Pipeline-as-code

Pipelines survive infrastructure failures

A number of applications have pipelines that run into multiple days. A pipeline typically runs on Jenkins executors (slaves) and it will continue running even in case of a master failure. If you use the CloudBees Jenkins Platform, you can checkpoint your pipelines so if an executor fails, you can pick from the last check-pointed location instead of re-running the pipeline.

You can imagine the productivity benefits on an existing pipeline where if a master crashed (itself or due to infrastructure failure) on day 4 of day 7 and it didn’t have to restart from day 1 – phenomenal. This can mean the difference between shipping on-time or delays in months!

Analyse and optimise your value stream process

Optimising the value stream process is the next logical step after modelling your delivery process, Pipeline Stage View helps you analyse the process delivery across multiple runs. You can see which stages consume the most time, which stages are blocked on manual user input and so on. Thus, you can quickly hone in on a problematic phase and optimise it.

Pipeline stage view

Developers can quickly get an insight into how far is their code into the pipeline. Teams waiting on artifact delivery on the code can also see where in the pipeline is the code.

Summary

Jenkins Pipeline brings in native support for pipelines into Jenkins. It is aimed at an audience that wants to continuously deliver software and is well worth a spin.

Resources:

Jenkins 2.0: Makes it easy for organisations to deliver software continuously

28 Apr 2016

I have spent the last 3 years of my life helping Jenkins claim its place upfront and central in the Continuous Delivery market. The journey first started by providing first class support for CD pipelines through Jenkins Pipelines and has culminated in the first major release of Jenkins in 10 years. I am incredibly privileged to have helped Kohsuke Kawaguchi in this journey. The following is a blog post that I put out on cloudbees.com for the release.

Jenkins 2.0 is generally available now. It is the first major release of Jenkins in 10 years after about 700 weekly releases. It is indeed a major milestone in the Jenkins project. The primary focus of the release is Pipelines – a means to helping organisations deliver better software faster.

Why should I care?

A fair enough question so I will wear a different “I” hat and answer why Jenkins 2.0 is relevant to you.

Role: Engineering Manager

Key Issue: My team delivers an application from development to production – what does Jenkins 2.0 bring to the table?

Pipelines: An easy to use DSL to model, view and optimise your application delivery value stream that can significantly improve the pace of delivery for your applications.

Jenkins 2.0 natively supports the notion of pipelines and helps to model your delivery pipeline through a DSL that is checked into the source code repository (Jenkinsfile). This file can then be version controlled and shared with other teams.

If your team was using multiple (existing) Jenkins plugins to model your pipeline, Pipelines greatly simplifies expressing the delivery pipeline, maintaining and managing it.

Pipeline stage view helps your team analyse the performance of the delivery pipeline and hone in on problematic stages in the delivery chain. Pipeline Stage View

Pipelines support GitHub and BitBucket organisations and pull requests. Point Jenkins to a GitHub organisation that has a Jenkinsfile checked in, Jenkins will automatically create a job, run and manage it – very useful when onboarding new teams.

Role: Shared Services Engineering Manager, VP of Engineering, Tools Group Manager

Key Issues:

  • My team is responsible for minimizing the impact of downtimes for various teams that depend on the tools supplied by my team.
  • My team is responsible for fostering best practices across engineering groups delivering applications.

Pipelines survive infrastructure outages; common delivery patterns can be stored in global libraries that are shared across various teams.

Pipelines are durable and resumable. Thus, if you support an engineering team whose application delivery spans multiple days, you can support them with the confidence to know that pipelines has inbuilt resilience to survive infrastructure failure. The pipeline job continues running even if the master fails. If you use CloudBees Jenkins Platform – your job can resume from the last checkpoint on executor failures.

Pipelines follow the DRY principle – your team can capture common deliverability patterns (“deploy to Tomcat”) into functions that can be invoked by teams that you support. Thus, your team can share best practices within the organisation.

Role: An existing Jenkins user

Key Issue: Any usability improvements?

Besides Pipelines, Jenkins 2.0 brings in a number of improvements for usability. Jenkins 1.x configuration page becomes overloaded with configuration choices introduced by plugins. Improvements to the configuration management has been an area of focus for Jenkins 2.0.

Config UI

A new startup wizard helps newbies get to a running installation quickly with a pre-configured set of plugins as well.

Role: Jenkins administrator Key Issue: Will an upgrade break an existing installation?

Jenkins 2.0 is backward compatible and a drop-in replacement. So upgrade without worries.

So go ahead – bring in Jenkins 2.0 and earn good karma within your organisation

Resources: