Introducing Spot Storage: Cloud-Native Storage Optimization

Since becoming part of NetApp, we have increasingly been asked by customers how Spot’s optimization technology could be extended to their cloud storage resources to help them use storage more efficiently, reducing costs and eliminating waste. Those conversations made it clear that there is a critical need, and a huge opportunity, for us to leverage NetApp’s expertise in storage and data management and apply that to giving cloud operations teams the ability to see and address inefficiencies in not only their use of compute resources, but also of storage resources, automatically and continuously.

That led us to work on and announce our plans for “Spot Storage”, a fully managed service that continuously optimizes cloud storage using capacity and performance auto-scaling engines and waste management technology for object, block and file storage. Based on historical usage and modeling of future application storage needs, Spot Storage will automatically remove wasted, unused storage and optimize the underlying storage type, disk size, and filesystem to achieve maximum performance and efficiency at the lowest possible cost.

We’re excited to announce that we’re rolling out the first Spot Storage capabilities, available today to AWS customers in the Spot console, to give them immediate, actionable insights to reduce costs and improve efficiency for storage.  And we’re even more excited about what’s coming next as we deliver the full vision of Spot Storage. Read on to learn about our plans for Spot Storage and about what we’ve released today for you to try and use.

Put Storage Optimization on Autopilot

DevOps teams continually tune cloud infrastructure to meet application SLOs.  Storage is a key part of this equation to balance application speed and overall costs. Tuning storage is usually a manual and tedious process often left to optimize only after compute needs are handled first. Tuning throughput, IOPS and capacity is some of the most time-consuming and sensitive work that most infrastructure teams face.

With Spot Storage, DevOps teams now have a simple and reliable way to optimize storage without needing to spend time on manual research, configuration and monitoring.  Spot Storage enables teams to adapt quickly to changing application needs and new storage types, seamlessly.

Spot Storage is powered by capacity and performance auto-scaling engines that will continuously monitor storage usage and performance and act accordingly. No manual intervention is required from engineering teams in order to reach and maintain the most optimal and cost-efficient storage environment.

Waste Management for the Cloud

Manual tuning for optimization is not the only concern that cloud teams are facing, another crucial issue is the ability to track, manage and remove wasted cloud resources. Among our customers, the area where we’ve seen that customers often suffer the most from not having solutions for waste management is in storage.

“We have thousands of volumes in our AWS accounts and are excited about leveraging Spot Storage to reach highly optimized costs in our storage environment.” 

–Marcelo Almeida, Principal Engineer SRE, Talkdesk

It’s extremely easy to lose track of your configured storage resources and, as a direct result, lose track of all the wasted resources that have been allocated. Using Spot Storage, customer will be able to leverage its built-in auto-scaling engines and recommendations to be able to gain advanced waste management across all storage types.

Here are just a few examples of what customers will be able to do as a result:

• Volumes 
  • Automatically scale down type/performance parameters of idle volumes.
  • Get real-time reports on unused volumes (volumes attached to stopped instances, unattached volumes, and more).
  • Enable a ‘recycle bin’ for volumes to automatically take a dedicated snapshot before the volume’s deletion and set retention rules for that snapshot.
• Snapshots 
  • Identify old snapshots that are eligible for removal based on the customer’s preferences and requirements.
• File Systems 
  • Provide real-time reports on unused file systems and automatically scale down supported capacity, all based on capacity usage, for supporting filesystems.
• Object Storage 
  • Automatically migrate objects from one storage class to another based on analysis of usage and access frequency.

Save Time and Money with Storage Optimization on Autopilot

Capacity auto-scaling will perform scaling activities, including expanding and shrinking capacity, for both managed file systems and block storage devices according to application requirements and needs. The capacity engine ensures that applications are always using precisely the amount of storage they require, with as little waste as possible, by adding, removing and resizing volumes as necessary.

Working alongside capacity auto-scaling, Spot Storage will provide Performance Auto-scaling, which will scale disk performance by continuously monitoring and updating the underlying media type (e.g. IOP1, IOP2, GP2, GP3, etc.) and various configurable parameters based on predicted application usage. The performance engine will tune the volume’s IOPS definition and alter storage types to achieve efficient application runtime SLOs.

Both the capacity and performance engines will coordinate changes with the filesystem to ensure consistency and application availability.

As storage needs change based on schedule, growth or load, the performance auto-scaler will weigh cost vs. performance and make necessary changes.  Changes can include direct parameter configuration such as IOPS or a full switch of the underlying storage from magnetic disk to SSD and back again, when necessary.  If during the weekend provisioned IOPS are not needed, Spot Storage will recognize this and make the necessary changes, saving up to 90% of the original cost.

“We are looking forward to using Spot Storage to automate the boring auto-scaling of storage and save money! This will reduce our unplanned activities 24×7.”

–Stephan Kaufmann, DevOps Chapter Lead, SHARE NOW/Car2Go

Some examples of what you’ll be able to do with Spot Storage:

• Define an auto-scaling policy that monitors a volume’s IOPS utilization and automatically decreases its IOPS definition by 1500 IOPS when the utilization drops below 40%. Or vice-versa, increase the IOPS definition seamlessly when utilization increases.
• Define an auto-scaling policy that increases the file system’s capacity by 20% when it reaches 75% usage by either adding an additional volume or by increasing the capacity of an existing one. In supported file systems, the scaler will be able to decrease the capacity automatically when the usage drops below a specific threshold.

Reducing Storage Costs with Spot Storage – What You Can do Now

Starting today, you can use the new Spot Storage dashboard directly in the Spot console to view cost optimization recommendations for your managed storage resources. The dashboard covers all storage components across your AWS account and generates real-time actions you can take to address wasted or misconfigured resources. Implementing those suggestions provides high potential cost savings that you can act on immediately. To achieve a simplified management experience, the dashboard aggregates the storage resources by their type and physical location, summarizes the call-to-actions (including the affected volumes’ details), and calculates the potential monthly savings for each recommendation and in total.

Looking Ahead:  What’s Coming Next for Spot Storage

Spot Storage will soon include support for Spot Volumes and SpotFS, our managed volumes and file system, as well as the release of both capacity and performance auto-scaling.

Spot Storage, together with the rest of Spot and NetApp portfolio, brings customers the next generation of cloud compute management (optimizing compute instances, network traffic, and now, storage layers as well). It demonstrates how combining NetApp’s experience and expertise in storage and data management with Spot’s technology and expertise in cloud optimization is delivering easier and better ways for customers to get more out of their cloud investment.

To be notified of new Spot Storage capabilities and try them out, register for our upcoming private previews using the Spot Storage dashboard in the Spot console.