Best Practices for Implementing CI/CD Pipelines in Kubernetes - Reactive Ops

Tell us more


Best Practices for Implementing CI/CD Pipelines in Kubernetes

Best Practices for Implementing CI/CD Pipelines in Kubernetes

At ReactiveOps we take an opinionated approach on how CI/CD should be done when Docker and Kubernetes are involved. In the following sections, I will outline the guiding principles of our opinions, and then lay out a typical CI/CD workflow that we use. I’ll also share some code that will enable you to give it a shot if you want. None of these principles are brand new or earth-shattering, but the combination makes for a repeatable, sustainable, and safe CI/CD pipeline that many of our customers love.

Guiding Principles

Containers Should Be Immutable

Docker allows you to rewrite a tag when you push a container. A common example of this is the latest tag. This is a dangerous practice because it presents a possible scenario where you don’t know what code is running in your container. We instead take the approach that all tags are immutable, so we tie them to an immutable, unique value that can be found in our codebase — the commit ID. This directly connects a container to the code that it was built from and erases any doubt that we know exactly what is inside that container.

Release The Container That Was Tested

In line with the first principle, the immutable container that we have deployed to a staging/dev/QA environment and then tested (hopefully) should be the same exact container that gets deployed to production. This eliminates any concern that something might change when deployed to production. We handle this by triggering a release to production using a git tag and releasing the container with the commit ID of the tag.

Keep Secrets And Config Out Of The Container

Since a container should be considered an immutable artifact of our build process, configuration should never be “baked in.” Configuration should be stored in a configmap in the Kubernetes cluster, and secrets should be stored, you guessed it, in Kubernetes secrets. This allows the deployment of the container to be configured per environment without making changes to the container itself.

Deploy With Helm

At ReactiveOps, we use Helm to template all of the yaml that gets deployed to Kubernetes. This allows the easy configuration of multiple environments and their config. In addition, it makes it easier to keep track of logical groupings or “releases.”

Use A Git-Based Workflow

All CI/CD pipelines should be triggered by Git operations. This falls in line with a normal developer workflow, and makes the pipeline more accessible to developers. In addition, it works well with the previous concept of immutability being tied to a commit ID.

Build Inside Docker

We strongly discourage the use of what a former co-worker referred to as “artisanal hand-crafted build machines” that contain a random assortment of dependencies and tools. If you build inside the container, you can control all dependencies and tools in your code and never have to worry about which “builder” is running your pipeline. Docker’s multistage builds make it easy to do this while also keeping your deployed runtime image as small as possible.

Take Advantage Of Caching

Docker images can be cached and re-used. We recommend the use of this feature as much as possible in order to keep build times to a minimum. The cache images should be pushed to a repository or the caching features of your CI/CD system should be used to maximize efficiency.

Example Pipeline

I’m going to lay out step-by-step a normal pipeline that we would set up for a customer. It takes into account all of the principles in the previous section, and will provide a path from code to production “In X easy steps!”

The process goes like this:

  1. Make a feature branch
  2. Push that branch and open a PR — this gets built and deployed to an ephemeral namespace that you can use to test the changes. Iterate until it’s ready to go
  3. Merge the feature branch to master — this triggers a build and a deployment to a staging environment
  4. Once the staging environment reaches a state that is ready for production, tag a release
  5. The tag gets deployed to production using the same container that was built for staging, but using production configuration

Looking at the process, we can see several “loops.” These are feedback loops that can be used to improve quality, test code, etc. The goal is to make these small and fast so that we can release code that is tested and manageable.

Go Forth And Make Pipelines

If you like the way we do CI/CD at ReactiveOps, you can have your very own pipelines just like ours! We use a set of bash scripts that do all of this for us. We just have to create some config files and we get a setup very similar to the one in this article. The repo is called rok8s-scripts (pronounced “rocket scripts”), and as always, PRs are accepted.

Other Developer Hub Posts

| Rob Scott

How we manage Kubernetes RBAC and IAM Roles on GKE

Read more

| Rob Scott

rbac-lookup: Reverse Lookup for Kubernetes Authorization

Read more

| ReactiveOps

The Benefits of Running Kubernetes on Google Container Engine

Read more

| Eric Hole

kops 102 - An Inside Look at Deploying Kubernetes on...

Read more