Elastigroup & Nomad

Nomad – Micro Services deployments have never been easier

Nomad is a tool for managing a cluster of machines and run applications on them. Nomad abstracts away machines and the location of applications and instead enables users to declare what they want to run and Nomad handles where they should run and how to run them.

Spotinst & Nomad

We are excited to introduce a native support for Nomad clusters.

Containers in general and specifically Nomad are a natural fit for the diverse pool of resources that Spotinst Elastigroup thrives on. Elastigroup enable you to provision capacity across multiple Instance Types and Availability Zones, which helps improve your application’s availability while reducing operating costs using the EC2 Spot Market. Combining the extensible and flexible container placement system provided by Nomad with Elastigroup allows you to efficiently deploy containerized workloads and easily manage clusters at any scale for a fraction of the cost.

How does it work?

Each instance within the Elastigroup automatically registers itself to the cluster as a client node and will be managed by Nomad and monitored by Spotinst.

Upon a prediction of a Spot interruption, Elastigroup will drain the node from its running jobs and replace it with another, new instance. Additionally, Elastigroup provides automatic scaling policies to meet the changing demand of services provided through Nomad.

Nomad Server vs. Client Mode

Nomad relies on an agent on every machine in the cluster. The agent can run either in server or client mode. Each region must have at least one server, yet a cluster of 3 or 5 servers is recommended. Due to the fact that the cluster configuration is held by the server instances, A single server deployment is highly discouraged as data loss is inevitable in a failure scenario.

All other agents run in client mode. A client is a very lightweight process that registers the host machine, performs heart beating, and runs any tasks that are assigned to it by the servers.

Scope

This integration is well-suited for Client-Server deployment.
Nomad clients should run on Spotinst’s Elastigroup using spot instances, while Nomad servers should be kept on On-Demand instances.

See it in action

Create your Nomad Server(s)

1. Install Nomad (or use a custom AMI that has it installed)
   a. For instructions, use the following documentation
2. Install Docker (or use a custom AMI that has it installed)
   a. For instructions, use the following documentation
3. Associate an Elastic IP with the instance
   a. For instructions, use the following documentation
4. Allow inbound traffic for Spotinst service (Used to control the instances deregistration and draining prior to termination)
   a. Allow access from the following sources using Nomad HTTP protocol (default: TCP/4646)
   Spotinst IP: 34.226.61.134/32

5. Create your server config file and name it server.hcl

   a. Minimal configuration: (Make sure to provide your EC2 host Elastic IP where required)

# Increase log verbosity
log_level = "DEBUG"

# Setup data dir
data_dir = "/tmp/server1"

advertise {
  # Defaults to the node's hostname. If the hostname resolves to a loopback
  # address you must manually configure advertise addresses.
  http = "<ElasticIPHere>"
  rpc  = "<ElasticIPHere>"
  serf = "<ElasticIPHere>:5648" # non-default ports may be specified
}

# Enable the server
server {
    enabled = true

    # Self-elect, should be 3 or 5 for production
    bootstrap_expect = 1
}

   b. Full agent configuration documentation can be found here

6. Start your Nomad agent server using the following command
   a. sudo nomad agent -config server.hcl &

7. Configure your Nomad agent to start upon instance restart
   a. One way to do that is using Crontab, read about it here

Create your Nomad Client(s)

1. Configure your Nomad client settings
   1.1. Create your client config file and name it client.hcl
    1.1.1. Minimal configuration: (Make sure to provide your Elastic IP where required)

# Setup data dir
data_dir = "/tmp/client"

# Enable the client
client {
   enabled = true
   servers = ["<NomadServerElasticIP>"]
}

1.1.2 Data Folder should be created
1.1.3 Full agent configuration documentation can be found here
1.2 You can either Upload your client config file to S3 or create a custom AMI containing the data folder and the client config file.
2. Configure your Elastigroup
2.1 Log on to your Spotinst console
2.2 Create a new Elastigroup
2.3 If you are not using a custom AMI that has Nomad & Docker installed – Set the user data script to install both Docker & Nomad
2.4 Download the Nomad client configuration from S3 & launch the agent
2.4.1 Add the following to your user data:
2.4.1.1 If your client config file is part of your AMI, Use the following User Data

export SPOTINSTINSTID=`curl http://169.254.169.254/latest/meta-data/instance-id`
  sudo nomad agent -config client.hcl -node $SPOTINSTINSTID &

2.4.1.2 If your client config file is hosted on S3, Use the following User-Data

sudo mkdir /tmp/client
sudo curl <NomadClientConfigFileURL> > client.hcl
export SPOTINSTINSTID=`curl http://169.254.169.254/latest/meta-data/instance-id`
sudo nomad agent -config client.hcl -node $SPOTINSTINSTID &

2.5 Select Nomad integration in the Spotinst integrations section
2.5.1 Configure the master server FQDN/public IP
2.5.2 Configure the master server port
2.5.2.1 Click ‘Test Connection’ To check connectivity

Cheers,