Data Persistence & Spot Instances, Solved.

We are excited to introduce our latest development that makes it possible to maintain data persistence with Amazon EC2 Spot Instances.

As always, we have been working closely with our customers to deliver this feature that brings immediate value to match your use cases, thus helping to lead to maximum utilization and efficient usage of AWS resources.

Overview

The concept behind keeping data safe with Spot is very simple. Usually, your data should be stored on EBS volumes, and you would most likely want to keep the data attached even in a case where the EC2 Instance is being replaced or terminated.

Every EC2 Spot Instance has a root volume which attached as /dev/xvda , Other data volumes (that contains application files and data) are attached to the EC2 Instance as /dev/xvd{b...z} .

This diagram shows a root volume(blue) as /dev/xvda and other volumes(purple) attached as other devices /dev/xvd{B...Z}.

Maintaining data persistence means that the EBS relations(volume IDs & Device Mapping) must stay the same across any instance. Effectively, it means that the EBS volume should be ‘migrated’ into a new instance so it can be accessible using the same volume mapping and the same stored data.

To make it possible, we are delighted to introduce our brand new “HOT EBS MIGRATION” feature.

Using the Hot EBS Migration feature, you can momentarily run workloads on Spot and ensure that your data stays safe and will be accessible in a case of any Spot interruption.

HOT EBS MIGRATION – How It Works

1. Create or have your EBS Volume ready in the relevant Availability Zone.

Make sure that the Volume is ready and Available

Your EBS Volume should contain a filesystem. If it’s not done already, please create a filesystem and then continue to the next step.

2. Create an Elastigroup Cluster and specify the Volume ID under “compute” tab in the “Hot EBS Migration Section”

If needed, add the relevant statement to the /etc/fstab in order to mount the volume automatically at the instance launch.

3. Once all is set, hit the create button and wait for the instance to be ready.

See It In Action

Now, let’s connect to the instance and look for the relevant device, e.g /dev/xvdb

That’s great, my instance is up and running, the EBS is attached as /dev/xvdb and already mounted at /mnt. I can also see that my previous files are available in that EBS.

Now – In the case of any Spot replacement or a failure, Elastigroup will seamlessly re-attach the EBS volume to a new instance.

This diagram shows what happens when a Spot is being replaced.
1. A new instance is being launched
2. The EBS Volume is being automatically de-registered from the old EC2 Instance
3. The EBS Volume is being automatically attached to the new EC2 Instance and mapped to the previous device name and mount point.

Instance Replacement Process

Upon Instance replacement, Spotinst’s prediction algorithm identifies that a Spot should be reimbursed, and immediately launches a new EC2 Instance. This process happens before the Existing Spot Instance gets its shutdown signal from AWS, meaning that the EBS replacement is happening in near real time, and leaves no place for data loss or application interruption.

That’s all, you are just 2 steps away from creating a data persistence workload on Spot.

Coming Soon

Support in Multi-AZs Deployments using continuous snapshotting of your EBS Volumes across multiple Availability Zones.

Best Regards,

The Spotinst Team.