Deploy AWS spot instances with Elastigroup

AWS spot instances represent AWS’s excess capacity, available for any surge in customer demand. To offset the loss of idle infrastructure, AWS offers this excess capacity at a massive discount to drive usage. With up to 90% in savings when compared to on-demand pricing, spot instance pricing is an affordable option for organizations looking to significantly reduce their cloud bills.

One of the challenges with spot instances, however, is that they can disappear at a moment’s notice. Without automation in place, it’s impossible for AWS users to predict when these instances might shut off and find a suitable replacement in time to minimize interruptions to resources and services. This is where Elastigroup from Spot by NetApp comes in.

Understanding Elastigroup by Spot

Elastigroup is a cloud infrastructure automation service in which the user can provision, manage, and scale compute instances to support any elastic application or load-balanced workload, on top of spare and reserved cloud capacity, without compromising availability.

With an enterprise-grade SLA (Service Level Agreement), even mission-critical and production workloads can benefit from up to 90% cost savings.

Elastigroup provides every feature from a standard cluster management platform but raises the bar with predictive approaches to instance selection, auto-recovery, and autoscaling, along with comprehensive dashboards and advanced automation.

How to deploy spot instances with Elastigroup

Pre-requisites

1. Create a Spot account

2. Connect your Spot account to your AWS account

3. Deploy/use an existing AWS auto scaling group

Note: We have the following infrastructure in place on our AWS account for this Proof of Concept (PoC):

• An AWS auto scaling group scaled across two availability zones
• 2 AWS EC2 on-demand instances running under region us-west-2

Get Started

1. Log in to the Spot Console and select Elastigroup (top-left menu)

2. Go to Groups -> select Import -> Auto Scaling Group

3. Import configuration by selecting the appropriate region and auto scaling group and hit Next.

Spot creates the Elastigroup based on the ASG (Auto Scaling Group) parameters imported from AWS. A summary of the parameters appears with the Review tab open. You can review the list of parameters.

4. Hit Create and Elastigroup will import the auto scaling group with existing configuration and deploy the desired infrastructure on AWS.

Once the import has been completed successfully, you can:

• Route to the Instances section to identify newly created spot instances now running under your AWS account managed via Elastigroup.
• Route to the Costs section to get a breakdown of imported clusters by:
  • Instance Lifecycle
  • Cost by Pricing Option
  • Daily Cluster Cost
• Route to your AWS account to find additional infrastructure spun up by Spot now being managed by Elastigroup.

5. Scale down the auto scaling group on AWS

Once Elastigroup takes over to deploy instances and run them on spot instances, you can scale down the AWS auto scaling group to zero.

Going forward Elastigroup would be the deployment arm for running spot instances on AWS with more features and capabilities than a traditional auto scaling group, such as:

• Delivering fully optimized and automated infrastructure scaling
• Delivering availability and scalability for scale-out applications, at up to 90% lower cost

To scale down, navigate Auto Scaling Groups under AWS. Select the appropriate ASG and go to Actions, Edit, then Update Group Size all to zero.

Reverting to AWS Auto Scaling Group

To revert infrastructure deployment to AWS auto scaling groups, follow the steps below:

• AWS -> Auto Scaling groups -> Select appropriate ASG -> Actions –> Edit –> Update Group Size all to desired level
• Spot.io -> Elastigroup -> Groups -> Select appropriate Group -> Actions -> Delete Group

Once completed, your desired infrastructure would be again consuming AWS auto scaling to launch EC2 on-demand instances as defined.

Additional Information

Before selecting the Create option, Elastigroup presents several options as follows:

1. Export the same configuration using JSON (JavaScript Object Notation), CloudFormation, and Terraform

2. Review and edit the imported configuration with the desired state before selecting the Create option for the following:

• General:
  • Downscale Imported ASG – Elastigroup will automatically scale down the existing auto scaling group on AWS as per the desired target.
  • Downscale will only happen once Spot launches new instances on your AWS account that are running healthy.

• Compute:
  • Edit the existing configuration with the desired compute parameters, such as changing the VPC (Virtual Private Cloud), Operating System, Subnets, instance types, and spot types.
  • Review compute also allows users to edit/add launch specifications, User Data Startup Script, and/or Shutdown Script.
  • Enable persistent stateful resources such as root volumes, data volumes, and IP addresses.
  • Integrate with Chef, ECS (Elastic Container Service), Beanstalk, Docker, Gitlab, Kubernetes, and more.

• Networking:
  • Elastigroup import allows attaching the imported group with load balancers or Spot ITF (Intelligent Traffic Routing).

• Predictive rebalancing:
  • Define the capacity workload to Min, Max, and Target
  • Define Optimization Strategies, such as:
    • Failback to On-Demand Instances
    • Utilize Commitment Plans
    • Cluster Orientation
    • Spot Market Scoring
    • Preferred Availability Zones
    • Preferred Spot Types
    • Draining Timeout (seconds)

• Scaling – Scaling lets you define the following:
  • Metric List
  • Target Scaling Policies
  • Simple Scaling Policies
  • Termination Policies
  • Scheduling

When you are finished reviewing and making modifications to the configuration, click Create. Spot will create your Elastigroup.

Additional resources on Elastigroup

• Understanding Spot Instances
• A Technical Introduction to Elastigroup
• Elastigroup Documentation
• Elastigroup APIs
• Terraform