Kubernetes and Jenkins – Going from continuous integration to continuous optimization

In modern organizations, continuous integration and continuous deployment (CI/CD) not only streamlines the process of code delivery and software deployment, it also plays a significant role in auditing and optimization. For Kubernetes workloads this translates into checking deployment configurations and pod definitions to ensure they are scoped correctly for both CPU and memory allocation. Over-provisioning at this step can lead to resource and infrastructure waste at scale, potentially eliminating any efficiency Kubernetes originally brought to the table. 

To address this challenge, Ocean by Spot complements CI/CD with “continuous optimization” (CO). Ocean ensures that the ongoing resources a cluster needs are handled in the most efficient manner, simplifying Kubernetes infrastructure management while reducing costs by optimizing cluster scaling and pod sizing.

Ensuring that pod sizing is in line with actual pod needs is accomplished via continuous monitoring. Ocean’s analysis provides rightsizing recommendations based on historical usage patterns as well as intelligent projection of future workload needs.

Combined with a CI/CD Pipeline such as Jenkins, Ocean’s recommendations  for right-sizing Kubernetes pods, can be seamlessly implemented to modify and optimize deployment requirements. This way companies can move fast while ensuring infrastructure costs stay as low as possible.

Let’s get started. 

But first some prerequisites

Before we can proceed with the implementation, we will need to make sure that the following prerequisites are in place:

1. Our Spot Ocean cluster has a Metrics Server set up and has collected enough information about our cluster utilization (typically 4 days of data). More information on how to set Ocean rightsizing
2. We have generated an authentication token to use the Spot API. Instructions are described here: Spotinst API Token
3. In order to fetch the information about the rightsizing recommendations we will need to have the Spot Python SDK installed. We will set a stage in our Jenkins pipeline so we have it installed.

For illustration purposes, our Jenkins pipeline will have four stages, as shown in the following screenshot.

Stage 3 and 4 are what actually gets the sizing recommendations and implement them in our Ocean cluster.

In order to set up the pipeline as described above, the Jenkinsfile that makes up our pipeline can be accessed here.

Setting up our Jenkins pipeline for continuous optimization

Step 1: Setting up our Container Templates for Python and kubectl

Use Container Templates to generate the appropriate docker images that are equipped to achieve the different tasks we need them to do.

For example, once we fetch our python script from the Git repository we will need a container equipped with python to execute it. Same goes for when we will have to interact with our Kubernetes deployment using ‘kubectl’ and a docker Container Template helps us build and run our own docker image.

podTemplate(containers: [
    containerTemplate(name: 'docker',
    image: 'docker',
    command: 'cat',
    ttyEnabled: true),
    containerTemplate(name: 'python',
    image: 'python',
    command: 'cat',
    ttyEnabled: true),
    containerTemplate(name: 'kubectl',
    image: 'lachlanevenson/k8s-kubectl:v1.8.8',
    command: 'cat',
    ttyEnabled: true)
]

Step 2: Setting the pipeline stage 1 and 2 – fetching our app and Dockerfile from Git

For the first two stages in our pipeline we need to get our Git repository which contains our sample app and Dockerfile, using the Jenkins Git plugin.

In Stage 2 we can build our docker image using the app and Dockerfile that we downloaded in Stage 1.

The purpose of these 2 stages is just to demonstrate a generic Continuous Delivery Pipeline.

Step 3:  Setting the Pipeline Stage Stage 3 – Ocean rightsizing 

The purpose of this step is to download and run our script, which sequentially uses Spot.io SDK to get the rightsizing recommendations of our Ocean cluster.

Finally the recommendations are transformed to a kubectl output, that ultimately we can feed to the next stage in the Pipeline.

stage('Get Ocean rightsizing') {
    container('python') {
        try {
            git url: 'https://github.com/nironkoren/spotinst-sdk-python.git',
            branch: 'master'
            sh 'pip3 install -U requests PyYaml'
            sh 'wget https://raw.githubusercontent.com/nironkoren/flask-http/master/get_sizing.py'
            // Generating a 'kubectl' cmd based on rightsizing recommendations
            cmd_output = sh(script: ('python ./get_sizing.py -d <DEPLOYMENT> -r <RATIO> ' +
            '-t <API_TOKEN> ' +
            '-a <ACCOUNT_ID> -o <OCEAN_ID> -n <NAMESPACE>'), returnStdout: true).trim()
            env.CMD_OUT = cmd_output
        }
        catch (exc) {
            println "Failed to get suggestions from Ocean"
            throw(exc)
        }
    }
}

As shown above, in order to execute our script we have to provide some parameters, and their functions are as follows:

Authentication to the SDK client can be done in three ways, we can do that using the ‘auth_token’ and ‘account_id’ parameters mentioned above, or we can use one of the alternative methods such as Spotinst Profile or using an Environment Variables. see here for more details.

The parameters for ‘deployment’, ‘namespace’, and ‘ocean_id’ are used to get the data for our desired deployment in a cluster.

Setting conditions for controlled rightsizing 

The ‘ratio’ parameter specifies the ratio between our current deployment size and the recommended size that we are willing to act upon. For example, if our deployment is set with a resource request of 4GB of memory but our rightsizing recommends to use 1GB of memory, we can set the ‘-r’ flag to 4, which means if our deployment is oversized by a factor of four, only then will it be resized. Alternatively if we were to set our ‘ratio’ parameter to 2 or 3, it will not proceed to right-size our deployment.

Step 4 –  Setting the Pipeline Stage 4 – Deploy Recommendations

The final stage is when we implement the rightsizing suggestions in our deployment using ‘kubectl’.

In the example below, Jenkins was installed using Helm on top of our Kubernetes deployment and was therefore configured with the appropriate Kubernetes plugin, so we could specify the following arguments to send a command back to our Kubernetes deployment.

‘CMD_OUT’, the environment variable we created in the previous step, contains the generated kubectl command with the rightsizing recommendations as a string, it is therefore used to deploy them in this stage. 

kubectl parameters:

• Token – “Bearer token for authentication to the API server”
• Certificate Authority – “Path to a cert file for the certificate authority”
• Server – “The address and port of the Kubernetes API server”

Source: https://kubernetes.io/docs/reference/kubectl/kubectl/#options

stage('K8S Deploy') {
    container('kubectl') {
        if (CMD_OUT) {
            try {
            sh (script: ('${CMD_OUT} ' +
                        '--token $(cat /var/run/secrets/kubernetes.io/serviceaccount/token) ' +
                        '--certificate-authority  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt ' +
                        '--server https://kubernetes.default.svc.cluster.local'))
            }
            catch (exc) {
                println "Failed to set deployment resources"
                throw(exc)
            }
                    }
        else {
            println "No adjustment needed"
        }
    }
}

Summary

By using this check-up automation in our ongoing Jenkins pipeline we were able to monitor if our cluster is up to snuff with the rightsizing recommendations, ensure that we use vCPU and Memory as efficiently as possible, and whenever necessary, seamlessly implement any modifications to the deployment requirements.

The next blog in this series, Automatic rightsizing with the Kubernetes Admission Controller, will expand on the fundamentals here to work with any CI/CD tooling.