Financial Operations (FinOps)
Autoscaling applications
Amazon Web Services
Big data
Container infrastructure
Microsoft Azure
Reserved capacity management
Cloud services for MSPs
Google Cloud Platform
Spot Overview
About NetApp
Elastigroup
Ocean
Ocean CD
Spot Storage
Eco
CloudCheckr
Cloud Analyzer
Spot Security
Case Studies
Resource Center
Documentation
News
Service Status
CloudOps hub
Our Story
The Spot Team
Contact Us
Our Values
Careers
Just recently we have released a breakthrough feature that allows AWS Elastic Beanstalk users to run their environments on Spot. and now, we are so happy to announce our unique Autoscaler for Elastic Map Reduce: EMR ClusterScaler.
EMR ClusterScaler enables you to make an elastic use of Task Nodes within your existing EMR Clusters. It lets you define target, minimum and maximum capacity for Task Nodes, select multiple instance types and sizes, and of course, CloudWatch rules for Scale UP and Down EC2 instances based on your business rules or load.
How it works?
EMR ClusterScaler is defined per EMR Job-Flow. In fact, it enables you to add 1..N task nodes, from multiple instance-types and lets you define CloudWatch rules for scaling up/down Task nodes.
For example, you can connect EMR ClusterScaler to an existing EMR cluster which runs 1 master node, 20 Core-Nodes, and 5 Task-Nodes.
- Selecting your target EMR Job-Flow Id
- Defining the amount of Task Nodes, the instance types, and sizes
- Define the CloudWatch rules for Scale Up and Scale Down
- It can be one of your EMR Metrics such as RemainingMapTasksPerSlot
- It can be per instance metric, for example- CPU Usage, Network bandwidth etc..
Once everything is configured, EMR ClusterScaler launches the desired amount of Task Nodes into your Job-Flow Id. and will automatically scale based on the metrics provided.
ClusterScaler makes an efficient and sophisticated use in the Spot Market, by launching Spot Instances as much as possible, to lower the cluster costs, and improve the ROI.
Spotinst ensures that your capacity won’t drop from your user-set minimum, by making a use in our in-house prediction algorithm in the Spot market. When Spotinst scales down resources, it scales first from “risky” Spot capacity, and always scales up from the most available Spot slots in AWS.
So, Autoscaling for Map Reduce application was around the corner for a long time. We are so happy that we have the chance to help cloud customers to make more efficient use with their resources.
As always, we value your feedback!
Try EMR ClusterScaler today.