Canary Deployment: Pros, Cons, and the Kubernetes Connection
What is Canary Deployment?
Canary deployment is a risk mitigation strategy for software releases. Every new software release represents a risk. Canary deployments can limit the damage caused by new releases that contain bugs or errors, by quickly and safely rolling back to a previous working version.
A canary version of a software product is a new version that includes specific new features, upgrades, or configuration changes. It includes all the necessary application code and dependencies, and is released to the production environment. However, it is only used by a small number of users.
After DevOps teams monitor usage and verify the stability of the new release, it is released to all users. If the version is not well accepted or issues are discovered, the system is rolled back to the current working version.
This is part of our series of articles about GitOps.
In this article:
- How Canary Deployments Work
- Canary Deployments vs. Blue Green Deployments
- Canary Deployment: Pros and Cons
- Can You Do Kubernetes Canary Deployments?
- Planning Your Canary Deployment Strategy
- Canary Deployment with Ocean CD
How Canary Deployments Work
In a canary deployment, you run two versions of an application at the same time. The old version is called the "stable version" and the new version is called the "canary". There are two ways to distribute updates: rolling deployment and parallel deployment.
Rolling Deployments
A rolling deployment introduces changes in steps. Several production servers start running the canary version while the others continue to run the stable version. Only those users accessing the canary servers can see the update.
The DevOps team can monitor the behavior of the upgraded system on the canary servers, check for errors and performance issues, and hear user feedback:
- If the canary version looks good, they will proceed to deploy the canary version to additional servers, until all servers are running the latest version.
- If there are major issues or users are not happy with the change, the team rolls back the upgraded servers to their original state.
Parallel Deployments
In a parallel canary deployment, instead of upgrading servers in stages, the DevOps team creates a whole new cloned environment and deploys the canary version on it. This requires duplicating the hardware environment on which the current production version is running.
Once the canary version is running, it is shown to a small fraction of users. This happens using routers, load balancers, reverse proxies, or other business logic. Gradually, the DevOps team migrates more users from the stable version to the canary version, while monitoring the canary. This process continues until:
- All users join the canary with no major issues
- A problem is detected, in which case traffic is routed back to the stable version
When deployment is complete, the duplicate environment can be shut down, until it is needed for another canary test.
Canary Deployments vs. Blue Green Deployments
Canary deployments can be confused with blue/green deployments. Both use load balancers or parallel production environments, managed by feature flags, to reduce the risk of software issues. But each has its own nuances.
In a blue/green deployment, the current production environment and the new release run in parallel. The Blue server is the current version and the Green server acts as a hot backup, deployed with a new software release. Then, using routers, feature flags, or traffic management schemes, traffic gradually transitions from the Blue to the Green server until 100% of all traffic goes to Green. After the cutover is complete and the new environment is verified, the team upgrades the Blue server to the new version, and it now becomes the hot backup.
In a canary deployment, the DevOps team is testing a specific new feature or system change, to see if it has a positive or negative effect. Here too, the current and new versions run in parallel. However, the switchover is handled differently—in a canary deployment, only a small number of users are exposed to the new version for an extended period of time. After the change is validated, it is rolled out to all users.
Related content: Read our guide to canary deployment vs blue green (coming soon)
Canary Deployment: Pros and Cons
Benefits of Canary Deployment
Here are a few key benefits of the canary deployment approach:
- Phased rollout reduces user exposure to negative operational issues. Some users will still experience problems, but the canary model keeps this number low, and because rollback happens quickly, it minimizes the negative experience for individual users.
- Easier rollback in case of problems, compared to traditional approaches where a rollback was complex, time consuming and risky.
- Fast rollout is an option—a DevOps team can design a canary test to last only a few minutes or a few hours. This enables rapid updates, supports high development velocity, and reduces cycle length.
- Improves customer trust because users are exposed to new features on a regular basis, and at the same time see that the software vendor is closely watching and taking care of problems as they occur.
- Improved morale of developer teams, because deployment of new features becomes easier and is more likely to be successful.
Challenges of Canary Deployment
Canary deployment is very beneficial but can raise a few challenges:
- Added complexity—teams must be ready to manage additional code, services, and components during the time of the canary release. It can also be complex to manage multiple API versions and database schemas.
- Software deployed on customer devices—if software is deployed remotely to customer devices or remote customer sites, canaries are more difficult to manage, and require more time to ensure customers have updated and evaluated the software.
- Automation is a must—manual canary deployments are error prone and time consuming. An automated process is key to performing canary deployments on a regular basis, but it can take time to build canary deployment into an existing CI/CD pipeline.
- Observability—canary deployments requires visibility of user behavior, system and application issues, to measure the relevant metrics and determine if the canary test is a success. In many environments, this level of visibility is not available yet.
Can You Do Kubernetes Canary Deployments?
There are several ways to perform canary deployments in Kubernetes. A common way to achieve it is using feature flags. The practice of managing software development using feature flags is known as feature management.
Feature flags give DevOps teams more control over your deployment and usually don’t require modifying configuration files, carrying out full blue-green deployments, or performing rollbacks. Feature management allows teams to securely deploy canaries to Kubernetes without starting multiple production instances of their applications.
To achieve a canary deployment in Kubernetes, a team can develop a new feature, mark it with a feature flag, and deploy the new version of the application to a fraction of the production environment. By default, users view the current stable version without the new feature. The team can designate a “canary group” of users and only they will see the new feature, marked with the feature flag.
If the canary test fails, the team can simply toggle the feature flag to "off". There is no need for a full rollback or redistribution. Conversely, if the canary test is successful, the feature flag remains “on”, and the new version can be rolled out to all users.
Planning Your Canary Deployment Strategy
There are a few things teams should consider when planning a canary deployment:
- How many users will be transitioned to the canary and in what stages?
- How long will the canary run? The team may have to wait days or weeks for enough clients to upgrade and evaluate the new features.
- Which metrics will be logged to analyze progress? These can include application performance, errors, and client satisfaction.
- How will the team determine if the canary is successful? This defines the threshold of the evaluated metrics that will determine success.
Here are a few options for scaling up the canary version:
- Starting with 1% of traffic and then scaling up to 10%, finally to 100%. This is known as 1-10-100, with two steps to fully rolling out the canary.
- A more bold approach is 25-50-100, in which users are exposed to the new feature much more quickly. Of course teams can experiment with any other values.
- Another option is to break down the rollout into more steps—for example, 10%, then 20%, then 50%, then 100%—a three step rollout. This is known as 10-20-50-100.
A final consideration is how to select users that should participate in the canary test. It is common to select users randomly, but there are a few other possibilities. Teams can expose a canary version to geographical regions one at a time. They can expose the canary to “early adopter” customers who have signed up to experimental features. Another option is to expose the canary first to internal users and employees.
