UPDATED! Elastigroup Jenkins Plugin

Jenkins supports the “master/slave“ mode, where the workload of building projects are delegated to multiple “slave” nodes, allowing a single Jenkins installation to host a large number of projects, or to provide different environments needed for builds/tests. This document describes this mode and how to use it.

We are happy to introduce our new Elastigroup Jenkins Plugin

This plug-in helps you to do more with your Jenkins setup, by allowing you to configure Jenkins to automatically scale a designated Amazon EC2 VMs as Slaves up or down depending on the number jobs to be completed.
Spotinst Plug-in enables you to get these resources with 100% uptime and 80% discount.

How It Works

The Spotinst Jenkins plug-in (1) automatically scales instances up & down based on the number of jobs in its queue. Nodes are (2) provisioned across multiple instance types and AZs to optimize savings while still guaranteeing availability. The nodes that are provisioned run a startup script (3) to connect as Slave nodes to the Master and immediately start running jobs.

How To Get Started on AWS

1. Generate a Spotinst API Access Token

Login into your Spotinst Console and then go to Settings -> Personal Access Tokens -> Generate

Once you’ve generated the API Access Token, Copy it, and save it for later use inside the Jenkins Console.

2. Create an Elastigroup with a Proper Startup Script

Create an Elastigroup with your preferred Region, AMI, and Instance Types. In the General tab under Advanced set the Capacity Unit to vCPU.

Add the following startup script:

#!/bin/bash
install_deps() {
  log_info "Installing dependencies"
  # Install deps.
  packages=$1
  for package in $packages; do
    installed=$(which $package)
    not_found=$(echo $(expr index "$installed" "no $package in"))
    if [ -z $installed ] && [ "$not_found" == "0" ]; then
      log_info "Installing $package"
      if [ -f /etc/redhat-release ] || [ -f /etc/system-release ]; then
        yum install -y $package
      elif [ -f /etc/arch-release ]; then
        pacman install -y $package
      elif [ -f /etc/gentoo-release ]; then
        emerge install -y $package
      elif [ -f /etc/SuSE-release ]; then
        zypp install -y $package
      elif [ -f /etc/debian_version ]; then
        apt-get install -y $package
      fi
      log_info "$package successfully installed"
    fi
  done
}
EC2_INSTANCE_ID="$(curl http://169.254.169.254/latest/meta-data/instance-id)"
JENKINS_MASTER_IP="IP:PORT"
# Install Java If not already installed
install_deps "jre"
# Get The Jenkins Slave JAR file
curl http://${JENKINS_MASTER_IP}/jnlpJars/slave.jar --output /tmp/slave.jar
# Run the Jenkins slave JAR
java -jar /tmp/slave.jar -jnlpCredentials user:password/token -jnlpUrl http://${JENKINS_MASTER_IP}/computer/${EC2_INSTANCE_ID}/slave-agent.jnlp &

The jnlpCredentials flag is used for authenticating to Jenkins, pass the username and password or token (such as the GitHub access token if GitHub is the hosting service which is being used for the authentication process).

Note: For optimal performance we recommend using the Amazon Standard AMI (CentOS based).

3. Change the default slave connection port

The Slave – Master connection is based on JNLP protocol. By default, the Slaves try to connect on a random JNLP port. Therefore, the firewall rules need to be reconfigured in order to allow all ports to be open to ensure successful communications from Slave to Master.
To configure a fixed JNLP port for the Jenkins Slaves, navigate to Manage Jenkins >> Global Security>>Agents and set a static TCP port for JNLP agents. Then, configure the network to be available exclusively for this port.

4. Install the Jenkins Plugin and Add Your Spotinst API Token and Elastigroup ID

Login to the Jenkins console, install the Spotinst Plugin from the available Plugins list.
After installing the plugin, Restart Jenkins.
Navigate to Manage Jenkins >> Configure System, scroll down to the Spotinst section and add the API Token generated in Step 1, along with an appropriate Account ID (will be used as a global Account ID in case no Account ID is specified for every cloud added in the next step).

Then click on Validate Token to ensure that the token is valid.

Once the Spotinst Token is set, scroll down towards the bottom to the “Cloud” section. Click on Add a new cloud and select the cloud provider connected to the Spotinst account being used (you can more than one cloud, each specifying it’s own Elastigroup and Account IDs).

There should now be more fields to choose from. For more information on each field hover over the information button on the right side of each field. Specify the Elastigroup ID for the Elastigroup created in Step 2, the appropriate Account ID associated with that Elastigroup and Idle Minutes Before Termination to determine how long Elastigroup should wait before terminating an idle instance.

Configuration Notes

• As noted in Step 4, Jenkins must be restarted after installing the Spotinst plugin.
• The connection between the Jenkins’ Slaves and Master is vital, make sure that this connection is working properly.
• Executors per instance: By default, the number of executors per Slave (the number of parallel jobs that a node can run) is based in the number of vCpu of the instance. You can override this configuration by setting the Instance type weight. For each instance type that you define in the Elastigroup, add the desired number of executors.

That’s all! From now on, the Jenkins Master will automatically launch new instances through the Spotinst API, and will terminate them as they get unused.

Best Regards,
The Spotinst Team.