Proudly serving Africa’s unbanked: JUMO achieves pipeline excellence and infrastructure efficiency with Spot by NetApp - Featured Image

Executive summary

Mobile fintech vendor JUMO uses the full range of Spot by NetApp’s DevOps offering on their AWS Kubernetes environment to fulfill their mission: Delivering superb low-cost financial services to Africa’s unbanked individuals and small businesses.
Since 2020, Spot has enabled JUMO to save 50-70% off their main platform’s Kubernetes compute costs – thanks to consistent access to AWS spot instances.
In 2022, JUMO rolled out Spot’s latest solution, Ocean CD, to automate their Kubernetes deployment pipeline: gradual deployment, continuous verification and automatic rollbacks.
As a result, engineering teams were greatly offloaded, and production errors were dramatically reduced, improving developer and DevOps productivity as well as service quality.

Overview

Founded in London in 2015, JUMO is a global B2B fintech vendor, providing consumers, businesses, and banks with a mobile banking-as-a-service platform. Integrating into JUMO’s platform enables banks to offer their customers loans, savings, and a wide range of financial choices to a new, often unbanked, market segment.

JUMO operates in sub-Saharan Africa, where the future of banking is mobile. In this region, 45% of adults are unbanked, meaning that they are not served by banks or other financial institutions, and mobile wallets are widely used as an alternative. In 2022, the region’s registered mobile accounts and transactions were valued at over $600B, making up 60% of the global $1T mobile money market. As these numbers grow rapidly year by year, JUMO’s technology plays an increasing role in promoting financial inclusion and sustainable economic development, especially in emerging markets.

Using Kubernetes to maintain high standards

JUMO is an AWS-native company with modernized architecture from day 1. At first, they used Docker Swarm to deploy containers. As they scaled, they moved to Kubernetes.

Being part of the fintech sector, and given their end user profile, JUMO is committed to maintain several standards on their backend cloud environment:

1. Availability: JUMO’s unbanked end users greatly depend on JUMO in their everyday lives. A service outage due to a failed deployment can cause them substantial financial damage (e.g., a failed transaction that cannot be redone). Therefore, JUMO must minimize the downtime originating from deployment errors.

2. Data security & privacy: Dealing with masses of financial & personal data, JUMO chose to comply with ISO27001:2013 as its security framework. This dictates tight governance regarding:

Infrastructure: JUMO’s Kubernetes clusters have lots of security policies. This complicates and lengthens some processes, like onboarding a third-party SaaS solution.
Third-party vendors: A solid audit trail is needed to demonstrate compliance and facilitate forensics investigations. This makes Configuration as Code a necessary feature of any third-party solution. Managing a solution via Terraform, or even a custom API, ensures that all changes are documented (unlike web UI).
Data storage: Policies and controls to protect financial user data from loss or unauthorized access must be set and followed, with great emphasis on cloud storage security and resilience.

3. Infrastructure efficiency: JUMO aspires to keep their cloud estate as lean as possible, to help maintain low end-user fees. Hence, FinOps is not just an idea but a company policy.

JUMO’s cloud management characteristics & challenges

JUMO’s Developer and Product teams manage their infrastructures independently, so most daily DevOps duties are done by developers themselves. This includes maintaining low compute spend. However, as they are not infrastructure specialists, some of their choices are not optimal.

For that reason, their efforts are supported by a systems development engineering team that implements company-wide cloud initiatives, such as the CI/CD pipeline and large-scale FinOps initiatives. This team of six supports a total of 90 developers, 80 microservices and 400 monolith pods – so they’re chronically overworked.

To offload themselves and the practicing teams, the Systems Development team wanted to implement optimization across the board. This led to JUMO’s first encounter with Spot by NetApp.

A long-standing partnership with Spot

The JUMO-Spot story is very much the story of JUMO’s infrastructure and FinOps evolution.

The introduction of Spot happened in Feb/March 2020 with Spot by NetApp’s FinOps tool, Eco. Eco was part of JUMO’s process of getting their AWS spend under control.

“We essentially used Eco to help us catch all the low hanging fruit when it came to our AWS bill – for example, using reserved capacity. We got some great results. Eco helped us achieve an overall 50% reduction in our cloud bill.”
Matthieu Antoine, staff systems development engineer, JUMO

Later in 2020, JUMO did a complete rewrite of its Kubernetes platform. At the core of this improved platform, JUMO rolled out Spot by NetApp’s serverless Kubernetes manager, Spot Ocean.

Prior to this rewrite, JUMO used the K8s cluster-autoscaler, that lacked the tight integration with AWS that JUMO was seeking. In parallel, JUMO also ran massively underprovisioned on-demand instances.

“Ocean helped us implement optimization across the board. It provided us with seamless scaling of our main platform compute. We were able to migrate most of our workloads to spot instances and therefore save greatly on our OpEx. Bringing Ocean in, our rework of Kubernetes brought the costs down by about 70%. On top of that, it allowed us to engineer for failure, bringing more resiliency to our systems.”
Matthieu Antoine, staff systems development engineer, JUMO

Having achieved continuous optimization on the infrastructure level, JUMO set out to optimize their DevOps pipeline, starting with streamlining the CI process. Here, Spot by NetApp’s VM autoscaler Elastigroup was put to work. Elastigroup’s integration with Jenkins enabled the autoscaling of JUMO’s CI workers on spot instances, so streamlining was two-fold: operational and financial.

Now JUMO was ready for the final move: Modernizing their CD process.

JUMO’s CD challenge

JUMO originally used Spinnaker to manage Kubernetes CD across their seven EKS clusters. For pipeline monitoring, they chose Datadog.

JUMO was seeking alternatives for Spinnaker to overcome its major limitations:

Spinnaker has no easy way to create new pipelines for new use cases (reusable entity model)
Spinnaker is overall very hard to manage, especially via Configuration as Code. There are 2 main “under the hood” reasons for that:
1. It uses JSON schemas for pipeline templates.
2. It uses random IDs for resources, so when you want to change a pipeline, you can only find out the resource ID through the UI. When you want to make some pipeline changes, this leaves no choice but to do these changes manually on the UI, where they are not documented.
Spinnaker is very feature-rich, with most features irrelevant to teams only deploying on Kubernetes. The recommended architecture for Spinnaker has extensive resource requirements leading to high costs, of which only a small portion is justified for Kubernetes-only teams. For a small DevOps team that is tasked with much more than just CD, Spinnaker is overkill.
Spinnaker does not offer integral canary deployment for containers, only for instances/VMs. This makes it completely irrelevant to teams with Kubernetes-only pipelines.

“It feels like operating a beast that’s too big for our needs. From an efficiency standpoint, we wanted a slimmer solution that’s a better fit to our use case,” says Antoine.

These shortcomings led to several everyday issues that any DevOps, SRE, or system engineering team would cringe at:

Failed deployments were wasting valuable engineering time. It took engineers between 40-60 minutes to investigate why a deployment failed and how to make it succeed.
Developers did not have reliable automation. Policies were not auto-enforced due to lack of confidence in how they would behave. This led to more manual steps to monitor new deployments – including acting manually upon Datadog alerts.
Spinnaker’s heavy setup surfaced serious UI issues. While modifying large pipelines, UI would often freeze for several minutes – delaying or risking the entire process.
Due to lack of configuration as code support, engineers operating their services didn’t notice that someone was changing the deployment pipeline. So they deployed normally and could not anticipate or understand the failure.
Without Configuration as Code, updating the deployment pipeline across the board was extremely laborious and error prone. A good example is Kubernetes version updates on existing clusters. Ideally, one would create new clusters with the new version for developers to migrate into once ready, without having to do anything. With Spinnaker, developers were forced to manually update the deployment pipeline themselves. This required lots of time clicking around the Spinnaker UI – especially for developers owning multiple services, each with hundreds of deployments. At each such update round, each of JUMO’s approximately 90 developers would end up with at least one or two errors. While most errors had no consequences besides wasted time, it only takes one that is of consequence to break production and cause real financial damage.

For example, at one point, these troubles resulted in staging docker images and staging values that made their way to production and broke it. For JUMO’s engineers, this was the last straw.

Technical and regulatory requirements

JUMO was seeking a solution to streamline their CD so errors and time spent would be minimal. To fit their technical, regulatory, and cultural environment, that solution had to meet several requirements:

Capabilities: Automate gradual deployments and rollbacks, and reduce the time required to detect & debug failed deployments.
Compatibility: Kubernetes-native.
Offloading potential: Reliable, scalable, simple to understand to run, highly automated.
Audit trail by Configuration as Code.
UX: JUMO only adopts a SaaS solution if they have substantial advantages over open-source alternatives. In the context of CD, they wanted developers to enjoy using it, be curious about how it works and how it helps them innovate. This was as opposed to open-source options (including Spinnaker), where UX is not the prime focus.

Spot by NetApp’s newly developed Ocean CD appeared to be the only SaaS solution that met all the above requirements.

Being very pleased with Spot by NetApp’s value to their developers (on autoscaling and cost savings), JUMO was happy to be among the first companies ever to try it out.

The design phase and onboarding process

In April 2022, JUMO joined the Ocean CD product team as a design partner. This gave them a chance to shape the product to deliver on all their success criteria.

Design phase

JUMO made several feature requests that were then developed as part of Ocean CD’s public beta version. Among those:

Reusing rollout spec files across different services, so they all roll out in the same manner & environments: This was an idea raised by a developer once they saw what Ocean CD can do.
Automated verification using Datadog metrics: JUMO were asked in advance how they would like to execute automated verification. As satisfied Datadog users, they requested that Ocean CD canary deployment strategies support Datadog integration. This helped developers gain confidence in the underlying metrics of their deployed code.
Advanced support for Kubernetes rolling updates: JUMO is using a rolling update strategy for a significant portion of workloads. The Spot product team was asked to enable it in addition to ‘canary’ strategies and together with Ocean CD value layers (e.g., verifications, UI for developers, support for multiple micro services across clusters, etc.). By that, more developers can utilize Ocean CD capabilities as part of their software delivery lifecycle.

“The Ocean CD team were open and eager for feedback, and very resourceful in addressing it, even in niche use cases. As a growing company we highly appreciated it.”
Matthieu Antoine, staff systems development engineer, JUMO

Onboarding

The Ocean CD rollout was led by JUMO’s systems development engineering team, where the organizational knowledge was established. One Dev team working on a new service was selected to test it hands-on. Developers were trained to use Ocean CD, with one “champion” overseeing it and providing feedback.

“Onboarding was easy because Ocean CD has everything regarding the deployment pipeline stored in files and Configuration as Code. Therefore, we could easily use the files across different services and projects. [The customer] took a one-hour session. Today as Ocean CD evolved, I believe it’ll take 30 minutes to onboard a new service for a new customer.”
Matthieu Antoine, staff systems development engineer, JUMO

Results

Once Ocean CD public beta was launched, JUMO went from zero to canary deployment in production in just two hourlong sessions.

Ocean CD helped JUMO automate canary deployments, monitoring and verification across all their selected service’s environments (dev, staging, and production).

Antoine explains: “CI/CD is a very complex process which varies greatly between organizations. But one piece of advice that may fit all is: Seek a solution that lets you automate things easily.”

JUMO pointed out some immediate benefits of having Ocean CD in place:

Full-cycle efficiency: The Datadog integration and clear visibility on the Ocean CD UI provided a level of confidence that JUMO’s developers had not yet experienced. This confidence was two-fold:
1. Trusting automation to manage their CD pipelines in production – from pre-detection of errors, through gradual deployment, to continuous monitoring.
2. Trusting their own ability to make quick, informed decisions if the deployed code doesn’t perform as expected.
Transparency of the deployment pipeline to the developers: JUMO developers can now be more involved in verification and not depend on the system engineering team to notify them about errors. This makes developers more independent and productive around deploying their services.
Removal of maintenance bottlenecks: “Ocean CD enables the automation of Kubernetes cluster migrations,” Antoine explains. “In the past this could take us one to two months each time, so we didn’t do it as often as we could. Ocean CD removes such bottlenecks in a way that allows us to be the best we can be in our CD environment.”

“One of Spot Ocean CD’s greatest traits is perfect integration with Kubernetes and third-party tools like Datadog. For us, this helps complete the product lifecycle.”
Matthieu Antoine, staff systems development engineer, JUMO

Business impact

Once Ocean CD is deployed across all JUMO’s development teams, they expect it to be felt everywhere: from the end-user experience to the company’s business KPIs.

“Fewer production errors mean improved availability to the end-users, as much as it depends on us. This, of course, supports end-user retention and growth. It is our way to deliver on JUMO’s greater mission: offering reliable financial services to people that so far had no access to such services at all.”
Matthieu Antoine, staff systems development engineer, JUMO

Developers using Ocean CD are practically relieved from “babysitting” their deployment pipeline: they trust that any error will automatically roll back, without them having to anxiously watch and analyze metrics on Datadog. This means, first and foremost, more time to innovate and code.

“Ocean CD will allow us to create a seamless CD process that’s as automated as can be, so developers spend as little time around it as possible. This is in line with my team’s core mission, to give our engineers the means to make their job faster and easier.”
Matthieu Antoine, staff systems development engineer, JUMO

JUMO’s engineering team also expects to save much time on orchestrating complex processes that currently require lots of “chasing people around,” like migrating Kubernetes clusters to newer versions. This, again, means more time for quality, cross-company cloud initiatives.

Try Ocean CD now

Worry-free operations

Now a user of Spot by NetApp’s full DevOps stack, JUMO enjoys a new level of infrastructure and pipeline efficiency. This will keep increasing as Spot further develops the integrations between our entire range cloud operations products.

“The Spot by NetApp stack is invisible in the best possible way. It always works as expected, integrates perfectly so we don’t mess with configuration changes, and doesn’t require thoughts or worries.”
Matthieu Antoine, staff systems development engineer, JUMO