Tutorial: Run Hadoop Cluster Nodes on Stateful Spot Instances

Earlier this month we announced support for “Stateful Spot” instances in Spotinst Elastigroups. This feature allows you to provision Spot Instances and automatically recover the full state of the instance including the private IP. When a recovery occurs we will automatically create a clone of the previous instance and it will appear as if the instance was brought down for a restart. Read more here: https://spot.io/blog/2017/04/new-service-stateful-spot-service/

Create the Hadoop Cluster

First, let’s create a new Hadoop Cluster. In this tutorial, I will be using a pre-built cloud formation template to provision the master node and a couple of data nodes. You can find this project here on GitHub https://github.com/varunrao/HDP-deployment

1. Open the CloudFormation template and paste this into the CloudFormation editor in your AWS console. The script can be found here: https://github.com/varunrao/HDP-deployment/blob/master/HDP-cloudformation-ambari-2.0.json
2. Modify the Cloud formation script for the region you are using. In my case, I had to change the AMI and region to match us-west-2. If you are using us-east-1 you will not need to make any changes.
   
3. Click the “create stack” button to deploy your CloudFormation script
   
4. Specify the details for the cluster as you see fit. Make sure to enter the correct keypair or this will throw an error.
5. Deploy the new stack and after a few minutes, you should see the status change to “CREATE_COMPLETE” as you can see below.
6. Go to “Outputs” to find the path for the management console.
7. 
8. From the console enter admin/admin as the login and password
   
9. You will now see your new cluster, if you click on hosts you will find your data nodes below.

Create new Data Nodes on Stateful Spot Instances

Now that we have created our new Hadoop cluster, let’s go into the Spotinst console and import the data node auto scaling group and add a stateful spot instance.

1. From the Elastigroup menu click “Create” then select “Import” and choose “Auto Scaling Group” from the drop-down menu.
   
2. Choose the region you deployed the Hadoop cluster to and select the ASG name as you can see below and click “Import”.
   
3. Select “Product” as “Linux” from the menu and select some spot instance types to match the cluster on-demand instance. In this case, I chose m3.medium and m3.large from the Spot type menu. Notice that there are six markets available for spots to be provisioned from.
   
4. Under “Stateful Configuration” select the options as you can see below. This will ensure that a replacement spot will be a clone of the instance in case a spot instance needs to be terminated.
   
5. Launch the new Elastigroup and wait for the new instances to be added to the Hadoop data node cluster.

Testing

Now that we have created a new Hadoop cluster let’s try to break it. From the EC2 console terminate one of the instances that were launched by your Spotinst Elastigroup.

When you go back to your Ambari console notice that one of the instances is in error state. This is the instance that was terminated during our test. Since we have enabled the Stateful features of our Elastigroup the Instance data volume, root volume, and private IP will be recovered automatically on a new EC2 instance. Spotinst will shut down the server, take a final snapshot of the volumes, create an AMI from this snapshot and launch a brand new instance with the same data, and private IP address.

The data node is now down since the instance was terminated via the EC2 console.

After a few minutes Spotinst Elastigroup will automatically recover the previously terminated spot instance into a new one maintaining the data and the private IP.

That’s it! you now have a new Stateful Spot instance added to your Hadoop cluster. In case the instance needs to be replaced it will automatically be shut down, cloned and replaced with a new spot with the same state and private IP as the previous instance.

Conclusion

We hope you enjoyed this tutorial on how to provision stateful spot instances for your Hadoop clusters. You can easily take advantage of Spot instances for your Hadoop cluster nodes with excellent cost savings and availability.

-The Spotinst Team