CloudBees and Codeship – Deep Cloud and Continuous Delivery Expertise at Your Doorstep

This is a repost of my blog on CloudBees.com

Tuesday was a big day for CloudBees, Codeship and the greater DevOps market. We are excited that Codeship is now part of the CloudBees family!

Our mission at CloudBees has always been about helping customers deliver valuable software faster by providing them with a great continuous delivery solution. Like us, Codeship has been focused on the same mission. With Codeship, CloudBees can address an entire spectrum of use cases from turnkey SaaS for teams to departmental CI/CD on-premise and to run-your-own-SaaS for enterprises on the cloud and on-premise.

Bringing DevOps practices to your team is a journey, and together with Codeship, CloudBees can offer your team the right tools for your development practices exactly when and where you need them in your journey. Thus, this move brings more flexibility to your engineering organization.

One vendor for all use cases and sizes of customers

In our recent years, we have been laser-focused on delivering a superior CD experience for SMB and large enterprise customers, alike, on the cloud and on-premise where customers want the flexibility to offer their run-your-own-SaaS to their developers. We also continued to come across customers who want to just do CI/CD and want someone else to manage their CI/CD solution. On the other hand, Codeship consistently comes across customers who want the flexibility and scale to drive their own SaaS.

Together, we can serve both sets of customers extremely well.

CloudBees and Codeship share customers where some teams are leveraging both the Codeship SaaS solution and the CloudBees platform to provide the tooling that works best for each team. Now they are able to get all that from a single vendor.

About CloudBees and Codeship synergies

Codeship was born in the cloud, and offers a rich SaaS platform to build, test and deploy your cloud native applications. Turn-key deployments to targets like AWS, Azure and Google Cloud are mixed with highly-customizable and flexible custom deployments using container technologies like Docker and Kubernetes.

“We were up and running within a day and sped up our development by orders of magnitude.”
– Ryan Fisch, Placester, Codeship Customer

Codeship’s strong focus on cloud native and the container ecosystem helps your teams get containerized applications to production, all while using containers to speed up your build processes and make your development lifecycle more efficient.

Container technology has been a focus for CloudBees in the last three years, with CloudBees Jenkins Enterprise running on Mesos and now on Kubernetes to deliver a truly scalable enterprise CI/CD solution to customers. Thus, we’re excited to offer that expertise and maturity to our users wherever they are – cloud or on-premise.

What does this mean for the DevOps Market and for our customers?

DevOps practices, and in particular CI/CD, are really something organizations of all sizes, development practices or technologies are aspiring to introduce to their teams. The problems that engineering organizations set out to solve with CI/CD are universal, like automating delivery workflows and reducing risk of defects and regressions, although the scale and complexity of them might vary.

CloudBees and Codeship combined are able to offer the full spectrum of solutions, from a SaaS-based CI/CD focused on best practices, to a large-scale enterprise run-your-own-SaaS CI/CD solution provided by the CloudBees product lines. CloudBees offers you the ability to grow and expand in the ever-changing world of DevOps. Whether your organization is running on classic architecture or fully buying into cloud native patterns, like a highly-available Kubernetes cluster, teams can pick the right CI/CD tooling for them.

This news also indicates the growing maturity and importance of the DevOps market, where customers are demanding enterprise-level features, support, scale and maturity from their vendors. CloudBees with Codeship strengthens CloudBees position as the number one vendor of choice for customers with products and support that customers have come to love.

Peering into the near future

There are exciting plans ahead for what the combined portfolio will be able to offer. One that I can share is that we have just launched a product called CloudBees DevOptics Deliver that helps customers understand and quantify the benefits of CI/CD by providing unprecedented insight into the software value stream. We will be looking forward to bringing the benefits of CloudBees DevOptics Deliver to Codeship customers.

Watch this space,
Harpreet

The dummies guide to Kubernetes ecosystem

27 Mar 2018

I will do a series a blog as I get my head wrapped around Kubernetes.

A recent proclamation in the technology trends is that Kubernetes has won the container wars. It implies that Kubernetes is the new operating system for the cloud.

What is Kubernetes (k8s)?

One of the key misconceptions is that k8s is an orchestration engine for docker containers. This is how I thought about it. That is true but it is much more.

Container orchestration is a key capability of the platform but there is much more. I think of k8s as the container orchestrator and the ecosystem services around it. Here are the key services that a company needs to think about as they bring in k8s:

  • Container scheduler, orchestration and runtime: the runtime are the services like network, filesystem that containers need to use to function; the scheduler schedules containers to run in a cluster; the orchestrator manages the SLA for the system.
  • API and management UI: every service in k8s is available through an API service and the management UX uses the API to provide the UX
  • Registry (docker registry): The service registry is where you look up docker containers that are provisioned by the runtime
  • Service discovery: all the services provided in k8s are discoverable through this service
  • Security and Governance policies: secret management, rbac, image protection and the overall resource policies fall here
  • Monitoring: if you have imagined a criss-cross of services, containers and messages flying around – you are right about it. A system like this needs a monitoring system.

So if you are bringing k8s in-house or evaluating a cloud solution, you need to think about the above categories of service and make sure that your provider is providing an acceptable service (for you) on each of the categories.

Jenkins – The Juggernaut of Continuous Delivery

09 May 2016

Los Hermanos Jenkins

Today, continuous delivery (CD) is en-vogue. As companies adopt CD, they are faced with a choice on the tools to establish their delivery pipelines. Jenkins is often the obvious choice and best of all it is likely used by the engineering division already. This blog looks at various data points to paint a picture of the pervasiveness of Jenkins for CD.

Blog When I started at CloudBees a half a dozen years ago, CD wasn’t

in the lexicon – in fact I named “Jenkins Pipelines” feature to be “Workflow” – pipelines is much better! Asking customers about their Dev-to-Production software delivery strategy was often met with a quizzical look.

The four-minute-mile barrier of software delivery was delivering software maybe once-a-month and that barrier has long been broken. Fabled stories of Amazon, Etsy and FaceBook delivering software multiple times a day have set a new bar that every company aspires to.

Google trends shows a continuous (pun intended) rise in the interest of CD. Trends

Since data science is the hottest trend in the market now, I have used data science to provide for a relative scale.Data science

It has been clear that Jenkins is the de-facto CI leader in the market. The ZeroTurnAround survey shows that Jenkins is most adopted CI server in the market with 70% of the market share.

ZeroTurnaround

(source: Zeroturnaround)

Active (ones that report active usage) Jenkins installations count is 127k and growing. Active Installations

(source: Jenkins stats site)

My guess is that non-reporting installations must be 2-4 times the active installations. Typically, a Jenkins installation supports anywhere between 5-100 people (dev, qa etc) – conservatively this could mean anywhere from 1M+ people using Jenkins (counting 10 developers per active installation).

Let’s look at data from Jenkins Survey in 2013 and 2015 to see some trends.

Jenkins for Operations In the 2 years between the surveys, the

usage of Jenkins for operations activities has grown by 9%.

Jenkins for  Operations

(source: Jenkins 2013 survey) Jenkins Jobs (source: Jenkins 2015 survey)

Jenkins for Deployment The story for deployment is even better with

60% of Jenkins users using the tool for deployment. Note: this number is likely higher as in some cases activities under “release, operations” overlap into the deployment zone. About 70% of those doing deployment are either doing continuous delivery (manual flag to deploy to production) or continuous deployment (auto-deploy).

Deployment

(source: Jenkins 2015 survey infographic)

If we take the 127k active installation number and apply it here (70% of 127k)- we have about 88k installations using Jenkins for deployment. Is Jenkins the #1 CD tool in the market? Looks like it! Can other tools make this claim – doubt it!

83% of deployments happen multiple times a week! Talk about breaking the 4 minutes mile barrier! Delivering is no longer stressful!

Multiple deployments

Building modern pipelines with Docker

Last year, the community churned out a number of plugins to help build pipelines with Docker. It is no surprise to now that Jenkins with Docker is a common use case. In other story, Jenkins is the most popular Docker image on Docker Hub with 5M+ (yes million) pulls.

Increase in the number of plugins

A key reason for Jenkins’ success has been the number of plugins that connect into existing tools. After all, if an organisation brings in a tool to do CD – it has to work with existing tools in the market.

Jenkins has about 1100+ plugins that connect to almost any tool out there. The number of installations of these plugins is north of 4.5M (yes million – it isn’t a typo). Usage

No wonder, the actual usage of Jenkins within an organisation keeps increasing with 89% respondents saying that their usage in Jenkins increased.

Actual usage

Mission Critical Jenkins

Given that an organisations’ entire software delivery chain is dependent on Jenkins, it is no surprise Jenkins is considered mission critical. Between 2013 and today the growth is about 9%. 92% of respondents say that Jenkins is mission critical – not a surprise, given that at-least 50% of developers in organisation are using Jenkins. A colleague asked a question in talk recently “What would happen if Jenkins goes down?”, only to be met with nervous laughter.

Mission critical

A testament to Jenkins is that 89% of teams are satisfied with Jenkins. Satisfaction

This satisfaction shows up in numerous awards!

Awards

Increase in the number of jobs – Jenkins jobs

In the last 3 years, the number of jobs created in Jenkins has grown almost 3 times from ~2.25M to 7M. Number of jobs

Increase in the number of jobs – human jobs

With the increase in Jenkins usage, its usage in CD, there is no surprise that people in the “DevOps” role using Jenkins has more than doubled to 48%. Number of human jobs

This is somewhat borne in the job postings in the market as well. There seems to be a growth in the specialised role that is focused on continuous delivery. There is a corresponding growth in Jenkins jobs – doubling in the last 2 years!

Job Role

Job Trend

To make sense of the data, I added data scientist job in there for fun 😃 and was more than presently surprised to see that there are 3x jobs in continuous delivery. Job trend 2

So what does this mean for you?

The Jenkins community has worked in the last two years to build the Jenkins Pipeline feature that brings native support for Pipelines. So if you are considering walking the continuous delivery path, ask your engineers to look at Jenkins Pipelines – most likely they are already using Jenkins and Pipelines is just a plugin away. Join the 79% of Jenkins users who are going to move to pipelines in the next 12 months.

Pipeline plugin

I have given an overview in an earlier blog and more recently the Jenkins community did a virtual conference that goes into details here. These should be good starts.

Summary

Unicorn companies have proven to the world that delivering software continuously is possible. Regular companies are now well on the path of delivering software continuously. A significant number of Jenkins users are now using it to deliver software continuously. Given, Jenkins’ large reach, it is not a stretch to say that it is the #1 tool in the market to do CD.

Opinions here are all mine and do not reflect that of my employers.

Reference

Escape the Iron Triangle – ship better software faster and cheaper with Jenkins Pipelines

01 May 2016

continuos software delivery

Delivering software continuously, in other words “keeping software in an always shippable state” is a challenge for companies. Recently released Jenkins 2.0, is focussed on solving this problem through Pipelines. This blog gives a quick overview of key benefits of Jenkins Pipelines.

Executive Summary

Software delivery in any organisation is a complex process that spans tools, is hand-stitched and fragile. In-efficiencies and failures in the process have a make or break effect on an applications success in the market place.

Jenkins Pipelines automates the process through code, connecting into the gargantuan number of tools of used by companies for software delivery. Once modelled, teams get insights into their delivery performance – to optimise their delivery times. Pipelines survive infrastructure failures and thus teams can deliver applications on-time, every time.

A month early due to a stream lined process is a month’s revenue while a month late in delivery due to infra failures is a month that competition can steal your customers!

Why Continuous Delivery (CD)?

Accelerating the pace of software delivery mandates code in an always shippable state and Continuous Delivery is the process that takes software to the always shippable state.

“Thou shalt choose between good, fast or cheap” said the Iron Triangle of software delivery. Continuous Delivery upends that case to help teams deliver good software, faster and cheaper .

Why Jenkins?

Software goes through various phases (build, test, deploy) and interacts with a multitude of tools (junit, sonar, nexus etc) on its way to production. Jenkins is the orchestrator that drives the entire pipeline and interacts with each of these tools. Jenkins’ strength is that it has 1200 (and counting) plugins that let you interact with any of the tools that are in your organisation.

Jenkins Pipelines

Jenkins 2.0 has introduced a key area called Pipelines. With Pipelines, organisations can define their delivery pipeline through a DSL (Pipeline-as-code). Pipelines, thus, can be versioned, checked into source and easily shared within an organisation.

Pipelines model YOUR delivery process rather than straight-jacketing you in an “opinionated” process

Typical delivery value stream within organisations have one commonality – atypicality. Most of the delivery processes are dissimilar unlike the canonical “build, test, deploy” examples that you see in examples. Here is an example from an earlier blog from Viktor Farcic on a delivery pipeline.

Delivery pipeline

The Pipeline DSL helps you capture complex process requirements through code – thus you can try-catch on deployment failures, loop through deployments, run tests in parallel. This is immensely powerful – the DSL brings the power of a programming language (groovy) to do so. At the same time, the DSL is simple enough to capture simple cases easily without having to write groovy code. DSL encourages the DRY principle – capture common patterns in functions, keep them in a global library so that new applications can build on these functions. Pipeline-as-code

Pipelines survive infrastructure failures

A number of applications have pipelines that run into multiple days. A pipeline typically runs on Jenkins executors (slaves) and it will continue running even in case of a master failure. If you use the CloudBees Jenkins Platform, you can checkpoint your pipelines so if an executor fails, you can pick from the last check-pointed location instead of re-running the pipeline.

You can imagine the productivity benefits on an existing pipeline where if a master crashed (itself or due to infrastructure failure) on day 4 of day 7 and it didn’t have to restart from day 1 – phenomenal. This can mean the difference between shipping on-time or delays in months!

Analyse and optimise your value stream process

Optimising the value stream process is the next logical step after modelling your delivery process, Pipeline Stage View helps you analyse the process delivery across multiple runs. You can see which stages consume the most time, which stages are blocked on manual user input and so on. Thus, you can quickly hone in on a problematic phase and optimise it.

Pipeline stage view

Developers can quickly get an insight into how far is their code into the pipeline. Teams waiting on artifact delivery on the code can also see where in the pipeline is the code.

Summary

Jenkins Pipeline brings in native support for pipelines into Jenkins. It is aimed at an audience that wants to continuously deliver software and is well worth a spin.

Resources:

Jenkins 2.0: Makes it easy for organisations to deliver software continuously

28 Apr 2016

I have spent the last 3 years of my life helping Jenkins claim its place upfront and central in the Continuous Delivery market. The journey first started by providing first class support for CD pipelines through Jenkins Pipelines and has culminated in the first major release of Jenkins in 10 years. I am incredibly privileged to have helped Kohsuke Kawaguchi in this journey. The following is a blog post that I put out on cloudbees.com for the release.

Jenkins 2.0 is generally available now. It is the first major release of Jenkins in 10 years after about 700 weekly releases. It is indeed a major milestone in the Jenkins project. The primary focus of the release is Pipelines – a means to helping organisations deliver better software faster.

Why should I care?

A fair enough question so I will wear a different “I” hat and answer why Jenkins 2.0 is relevant to you.

Role: Engineering Manager

Key Issue: My team delivers an application from development to production – what does Jenkins 2.0 bring to the table?

Pipelines: An easy to use DSL to model, view and optimise your application delivery value stream that can significantly improve the pace of delivery for your applications.

Jenkins 2.0 natively supports the notion of pipelines and helps to model your delivery pipeline through a DSL that is checked into the source code repository (Jenkinsfile). This file can then be version controlled and shared with other teams.

If your team was using multiple (existing) Jenkins plugins to model your pipeline, Pipelines greatly simplifies expressing the delivery pipeline, maintaining and managing it.

Pipeline stage view helps your team analyse the performance of the delivery pipeline and hone in on problematic stages in the delivery chain. Pipeline Stage View

Pipelines support GitHub and BitBucket organisations and pull requests. Point Jenkins to a GitHub organisation that has a Jenkinsfile checked in, Jenkins will automatically create a job, run and manage it – very useful when onboarding new teams.

Role: Shared Services Engineering Manager, VP of Engineering, Tools Group Manager

Key Issues:

  • My team is responsible for minimizing the impact of downtimes for various teams that depend on the tools supplied by my team.
  • My team is responsible for fostering best practices across engineering groups delivering applications.

Pipelines survive infrastructure outages; common delivery patterns can be stored in global libraries that are shared across various teams.

Pipelines are durable and resumable. Thus, if you support an engineering team whose application delivery spans multiple days, you can support them with the confidence to know that pipelines has inbuilt resilience to survive infrastructure failure. The pipeline job continues running even if the master fails. If you use CloudBees Jenkins Platform – your job can resume from the last checkpoint on executor failures.

Pipelines follow the DRY principle – your team can capture common deliverability patterns (“deploy to Tomcat”) into functions that can be invoked by teams that you support. Thus, your team can share best practices within the organisation.

Role: An existing Jenkins user

Key Issue: Any usability improvements?

Besides Pipelines, Jenkins 2.0 brings in a number of improvements for usability. Jenkins 1.x configuration page becomes overloaded with configuration choices introduced by plugins. Improvements to the configuration management has been an area of focus for Jenkins 2.0.

Config UI

A new startup wizard helps newbies get to a running installation quickly with a pre-configured set of plugins as well.

Role: Jenkins administrator Key Issue: Will an upgrade break an existing installation?

Jenkins 2.0 is backward compatible and a drop-in replacement. So upgrade without worries.

So go ahead – bring in Jenkins 2.0 and earn good karma within your organisation

Resources: