Running an AWS ECS Service with Persistent Storage Using Docker Plugins and Spot Elastigroup

Amit Bar Oz
September 27, 2018
6 min read
Amazon Web Services Containers Docker Spot Instances
ECS blog post banner

Amazon recently announced that ECS now supports Docker volumes and volume plugins. In this blog post, we will cover the configuration needed to use Docker volume plugins on top of ECS to run containers with persistent/shared storage, whilst utilizing Spot Elastigroup to manage the ECS cluster container instances.

We will also cover the use of the Cloudstor:AWS plugin. It is possible to use several other Docker volume plugins in a similar matter such as REX-Rey, Portworx, etc. However, the installation and usage might be slightly different. The Cloudstor:AWS plugin runs on your ECS container instance and attaches EBS and/or EFS volumes to your tasks (containers).

Spot’s Elastigroup Managed Container Service (MCS) allows you to run an ECS cluster on top of Elastigroup, which is an entity allowing you to run Spot Instances on a production-grade level. MCS will take care of replacing your instances seamlessly without any downtime. On top of that, MCS provides Serverless Experience Autoscaling and makes sure that you will always have the most cost-effective infrastructure for your workloads without the need to manage or scale your EC2 Instances.

The MCS AutoScaler allows you to create an ECS cluster with multiple container instance types and sizes, out of the box. It uses smart Tetris Scaling that is driven by container resources requirements rather than nodes utilization. On top of that, it allows you to configure headroom capacity which will make sure that capacity is available for new workloads at any time (in this way you don’t have to wait for a new instance to launch when creating a new service/task).

 

Let’s Start by Creating an Elastigroup MCS with a Running Docker Volume Plugin

1. Create an IAM policy for Docker volume plugins.
“Action”: [
“ec2:CreateTags”,
“ec2:AttachVolume”,
“ec2:DetachVolume”,
“ec2:CreateVolume”,
“ec2:DeleteVolume”,
“ec2:DescribeVolumes”,
“ec2:DescribeVolumeStatus”,
“ec2:CreateSnapshot”,
“ec2:DeleteSnapshot”,
“ec2:DescribeSnapshots”
],
“Effect”: “Allow”,
“Resource”: “*”

 

2. Create an IAM role. Under “Permissions policies”, attach “AmazonEC2ContainerServiceforEC2Role” and the policy created in step 1.

Please note:

If you already have an IAM role for your ECS container instances, make sure to add the permissions policies from step 1 to it.

 

3. Create a new MCS Cluster by importing an existing ECS cluster or by using the Spotinst CFN template in the Elastigroup Creation Wizard. Make sure to select the IAM role that you created before as your container instances IAM role.

 

4. Update your Elastigroup user data script. To support the Docker volume plugin installation, we need to update our ECS agent version and install the plugin on the container instances.
#!/bin/bash
echo ECS_CLUSTER=<ECS-CLUSTER-NAME> >> /etc/ecs/ecs.config

# Updating the ECS agent
sudo yum update -y ecs-init

# Use this command if you want to support EBS as well as EFS
docker plugin install --alias cloudstor:aws --grantall-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS EFS_ID_REGULAR=<YOUR_EFS_ID> EFS_ID_MAXIO=<YOUR_MAXIO_EFS_ID> AWS_REGION=<REGION_NAME EFS_SUPPORTED=1 DEBUG=1

# Use this command if you only want to support EBS
docker plugin install --alias cloudstor:aws --grant-all-permissions docker4x/cloudstor:18.03.0-ce-aws1 CLOUD_PLATFORM=AWS AWS_REGION=<REGION_NAME> EFS_SUPPORTED=0 DEBUG=1

 

Creating a Task Definition

Create a new task definition and configure it as you please. Next, we will focus on configuring a persistent volume with a Docker volume plugin. Click on the “Add volume” button to add and configure the volume.

In this section, we will configure the volume that will be created for the task definition. There are several things that need to be configured:

  • Volume name.
  • Check the “Specify a volume driver”.
  • Set Driver to the desired Docker volume plugin. In this case, “cloudstor:aws”
  • The scope of this volume.
    • Task – for every task that will run from this task definition, a volume will be provisioned. This volume will be destroyed when the task stops.
    • Shared – creates a persistent volume.
  • Enable auto-provisioning – if the volume doesn’t exist before running the task, it will be created by the plugin. When shared is marked, it is important to mark this checkbox.
  • Driver options: These are flags that are sent to the plugin regarding the volume to create.
    Key Value
    size The size of the volume.
    backing Use “shared” for EFS volumes, “relocatable” for EBS volumes.
    ebstype If you use EBS volumes, you can specify gp2, io1, st1, sc1
    iops Required if ebstype specified is io1, which enables provisioned IOPS. Needs to be in the appropriate range as required by EBS.

When finished, click the Add button to complete the volume setup. With the volume configured, we now have to attach it to the container(s) where this task will run.

Add/edit a container, go to the Storage and Logging section:

Create Mount points for the volumes created earlier for the container:

For example, this is the JSON of a task definition for a Nginx container with an EBS volume attached to it:

{
    "requiresCompatibilities": [
        "EC2"
    ],
    "containerDefinitions": [
        {
            "name": "nginx",
            "image": "nginx",
            "memory": "500",
            "essential": true,
            "portMappings": [
                {
                    "hostPort": "8080",
                    "containerPort": "80",
                    "protocol": "tcp"
                }
            ],
            "environment": null,
            "mountPoints": [
                {
                    "sourceVolume": "nginx-vol",
                    "containerPath": "/data",
                    "readOnly": ""
                }
            ],
            "volumesFrom": null,
            "hostname": null,
            "user": null,
            "workingDirectory": null,
            "extraHosts": null,
            "logConfiguration": null,
            "ulimits": null,
            "dockerLabels": null,
            "repositoryCredentials": {
                "credentialsParameter": ""
            }
        }
    ],
    "volumes": [
        {
            "dockerVolumeConfiguration": {
                "driver": "cloudstor:aws",
                "scope": "shared",
                "driverOpts": {
                    "size": "10",
                    "backing": "relocatable"
                },
                "labels": null,
                "autoprovision": true
            },
            "name": "nginx-vol"
        }
    ],
    "networkMode": null,
    "memory": null,
    "cpu": null,
    "placementConstraints": [],
    "family": "nginx"
}

Now, all we have to do is to create an ECS service using this task definition and we will be good to go.

 

 Now Let’s Explore Using Docker Volume Plugins with ECS

Now that we know how to create a task definition and a service with a persistent volume, let’s discuss how the different volumes serve us and some limitations around this method.

When using EBS volumes across multiple availability zones, we have an issue because the plugin works in a way that if a task in a service is terminated, a new task will be scheduled and the EBS volume will be re-attached to it. When working across multiple availability zones, if the task is rescheduled on a container host in another availability zone, the plugin will take a snapshot, migrate it, and create a new EBS volume from this snapshot. This procedure does take a few minutes and will take longer as the volume gets bigger. When using EBS volumes, it is highly recommended to set a constraint that will allow this task (either from the task definition or the creation of the service) to spawn only on one availability zone.

You can set task placement constraints from the Task Definition, Service, or Task creation.

When doing that, you have to use Cluster Query Language. In the following example, we will configure the current task to run on Availability Zone “us-west-2b”.

When using unique persistent volumes, you’ll have to create a task definition and service for every container that you want to run. The reason for that is at this point, you cannot configure a task definition with a “shared” Docker volume that will be created for every service/task run from this task definition.

Since an EBS volume can only be attached to one task at a time, it is important to think about your architecture thoroughly before using this method.

When using EFS volumes, since EFS is a region based service, they are easily shared across all of the tasks created from the task definition.

Conclusion

Using the Cloudstor plugin provides an easy and effective way to deploy persistent storage with AWS ECS cluster. When integrating it with Spot’s Elastigroup MCS, you will achieve maximum cost efficiency and simplified management of ECS container instances.