Spot + Data Mechanics: Delivering fully automated, optimized Spark on Kubernetes

After working with hundreds of customers over the last few years, we know one thing for sure at Spot by NetApp—when applications have infrastructure that is automatically optimized to meet their needs, they can scale and thrive in the cloud. Spot by NetApp recently announced the launch of service that lets customers run their Apache Spark jobs with a simple API on k8s, with fully managed infrastructure under the hood, bringing cloud infrastructure automation and optimization to big data applications, a key workload that organizations are running in the cloud. This expansion of the Ocean product family was met with excitement and strong interest from many of our customers, and now Spot is accelerating its roadmap with the acquisition of Data Mechanics, a fully-managed offering of cloud native Apache Spark.  

The Data Mechanics solution makes running Apache Spark, a go-to open-source framework for big data, AI and ML-based applications, easy and accessible on Kubernetes. With Data Mechanics capabilities integrated into the Ocean product family in Spot Wave, big data application owners will be able to easily run fully automated, optimized Spark on fully automated, optimized cloud infrastructure. While Data Mechanics technologies and team will be integrated into Spot Wave, Data Mechanics customers will continue to receive full support, and will now significantly benefit from the expanded offering of Wave.

Big data in a Kubernetes-native world 

At a time when enterprise data growth is exploding, companies are looking for ways to simplify and speed up development and deployment of analytics, artificial intelligence and machine learning applications. The power and flexibility that Kubernetes offers is ideal for Spark and other big data applications, which are processing more and more data in the cloud every day and uncovering valuable insights that enable and enhance automation, efficiency, customer experience, cost optimization and many more strategic business goals across an organization. However, deploying and operating these platforms requires a rare combination of cloud infrastructure, platform and big data software expertise that puts a heavy load on big data infrastructure and platform teams.  

Operating big data infrastructure on your own is complex and inefficient, and running applications like Spark with Kubernetes adds an additional layer of complexity that even experienced DevOps teams can struggle with. Despite these challenges, companies are going all-in on Kubernetes hoping to build modern cloud environments that are fast, scale big and enable their development teams to be more agile.   

Designed specifically to help these big data applications run on Kubernetes, Spot Wave automatically provisions, scales and optimizes cloud infrastructure, relieving platform and DevOps teams from the need to manually manage cloud infrastructure on a day-to-day basis. With Data Mechanics, Spot Wave can take a huge a leap forward. In addition to technology that understands Spark workloads, Data Mechanics brings significant expertise in operating cloud-native Spark. The application-aware capabilities of Data Mechanics technologies combined with Spot’s recognized expertise in cloud infrastructure and Kubernetes will offer customers a modern, efficient, and scalable way to run Spark in their cloud.