Canary deployment is a risk mitigation strategy for software releases. Every new software release represents a risk. Canary deployments can limit the damage caused by new releases that contain bugs or errors, by quickly and safely rolling back to a previous working version.
A canary version of a software product is a new version that includes specific new features, upgrades, or configuration changes. It includes all the necessary application code and dependencies, and is released to the production environment. However, it is only used by a small number of users.
After DevOps teams monitor usage and verify the stability of the new release, it is released to all users. If the version is not well accepted or issues are discovered, the system is rolled back to the current working version.
This is part of our series of articles about GitOps.
In this article:
In a canary deployment, you run two versions of an application at the same time. The old version is called the "stable version" and the new version is called the "canary". There are two ways to distribute updates: rolling deployment and parallel deployment.
A rolling deployment introduces changes in steps. Several production servers start running the canary version while the others continue to run the stable version. Only those users accessing the canary servers can see the update.
The DevOps team can monitor the behavior of the upgraded system on the canary servers, check for errors and performance issues, and hear user feedback:
In a parallel canary deployment, instead of upgrading servers in stages, the DevOps team creates a whole new cloned environment and deploys the canary version on it. This requires duplicating the hardware environment on which the current production version is running.
Once the canary version is running, it is shown to a small fraction of users. This happens using routers, load balancers, reverse proxies, or other business logic. Gradually, the DevOps team migrates more users from the stable version to the canary version, while monitoring the canary. This process continues until:
When deployment is complete, the duplicate environment can be shut down, until it is needed for another canary test.
Canary deployments can be confused with blue/green deployments. Both use load balancers or parallel production environments, managed by feature flags, to reduce the risk of software issues. But each has its own nuances.
In a blue/green deployment, the current production environment and the new release run in parallel. The Blue server is the current version and the Green server acts as a hot backup, deployed with a new software release. Then, using routers, feature flags, or traffic management schemes, traffic gradually transitions from the Blue to the Green server until 100% of all traffic goes to Green. After the cutover is complete and the new environment is verified, the team upgrades the Blue server to the new version, and it now becomes the hot backup.
In a canary deployment, the DevOps team is testing a specific new feature or system change, to see if it has a positive or negative effect. Here too, the current and new versions run in parallel. However, the switchover is handled differently—in a canary deployment, only a small number of users are exposed to the new version for an extended period of time. After the change is validated, it is rolled out to all users.
Related content: Read our guide to canary deployment vs blue green (coming soon)
Here are a few key benefits of the canary deployment approach:
Canary deployment is very beneficial but can raise a few challenges:
There are several ways to perform canary deployments in Kubernetes. A common way to achieve it is using feature flags. The practice of managing software development using feature flags is known as feature management.
Feature flags give DevOps teams more control over your deployment and usually don’t require modifying configuration files, carrying out full blue-green deployments, or performing rollbacks. Feature management allows teams to securely deploy canaries to Kubernetes without starting multiple production instances of their applications.
To achieve a canary deployment in Kubernetes, a team can develop a new feature, mark it with a feature flag, and deploy the new version of the application to a fraction of the production environment. By default, users view the current stable version without the new feature. The team can designate a “canary group” of users and only they will see the new feature, marked with the feature flag.
If the canary test fails, the team can simply toggle the feature flag to "off". There is no need for a full rollback or redistribution. Conversely, if the canary test is successful, the feature flag remains “on”, and the new version can be rolled out to all users.
There are a few things teams should consider when planning a canary deployment:
Here are a few options for scaling up the canary version:
A final consideration is how to select users that should participate in the canary test. It is common to select users randomly, but there are a few other possibilities. Teams can expose a canary version to geographical regions one at a time. They can expose the canary to “early adopter” customers who have signed up to experimental features. Another option is to expose the canary first to internal users and employees.
Complete access
for up to 20 instances