Spotad Runs Complex EMR Clusters on EC2 Spot With Confidence

Spotad Keen on Spot Instances, but Environments Don’t Seem Immediately Compatible

Like any forward-thinking business, Spotad always kept one eye on their rising EC2 costs. Naturally, as the company went from success to success, these costs began to increase and Tal Maizels, CTO at Spotad, was determined to find a way to manage these rising costs. At first glance, spot instances seemed like a great possibility. After all, the proposition of having the exact same instances as on-demand or reserved instances for around 80% less than on-demand and on a pay-as-you-use model is a very appealing one.

The issue, however, is that the complexities of Spotad’s environments meant that, whilst the savings and flexibility offered by spot instances were appealing, they were impractical to implement due to the increased demand it would place on their developers. Environments such as EMR Clusters, which make up a significant part of Spotad’s AWS usage, would require the Spotad developer teams to develop a complex and intricate spot instance management tool on top of also performing their regular duties.

Such an immense task was not practical, and could still result in termination issues and imperfections. What Spotad really needed was a pre-built platform that could help them utilize spot instances without the additional demands it would place on the Spotad developers.

Spot Allows Spotad to Run EMR Clusters and More on Spot, Hassle-Free

Amazon Elastic MapReduce (EMR) is a web service that simplifies big data processing, providing a managed Hadoop framework that makes it easy, fast, and cost-effective for you to distribute and process vast amounts of your data across dynamically scalable Amazon EC2 instances.

Elastigroup by Spot makes intelligent and reliable use of Amazon EC2 spot instances in your existing Amazon EMR cluster or your custom Hadoop environment. In addition to seamless management, the Spot algorithm will choose the best spot instances in terms of price and availability. Spot also has a built-in auto-scale feature that scales task and core nodes up or down to achieve the best cost efficiency.

Elastigroup makes an efficient and sophisticated use in the Spot Market by launching spot instances as much as possible to lower the cluster costs and improve the ROI. Spot ensures that your capacity won’t drop from your user-set minimum by using Elastigroup’s prediction algorithm in the Spot Market. When Spot scales down resources, it scales first from “risky” spot capacity, and always scales up from the most available Spot Markets in AWS