
As the official AWS CLI tool, eksctl is an open-source CLI that has gained popularity within the Kubernetes community for easily creating Elastic Kubernetes Service (EKS) clusters. One of the benefits of EKS itself is that the Kubernetes control plane is handled entirely by AWS, freeing you from managing high availability for your control plane(s) as well as from handling all ongoing maintenance such as Kubernetes upgrades.
However, when it comes to your clusters’ worker nodes, you still need to find a solution that simplifies container infrastructure management by handling challenges like:
- Unavailable resources due to autoscaling that only looks at cluster CPU and Memory and not actual Pod requirements
- Cost visibility (showback) that allocates spend based on namespaces, resources, labels, and annotations
- Pod rightsizing based on analysis of their actual resource consumption
- Handling immediate container scaling for high priority workloads
- Reducing cloud infrastructure costs using spot instances and preemptible VMs
Fortunately, Ocean by Spot takes care of all these time-consuming, DevOps activities, with our container-driven autoscaling, cloud-native cost allocation, vertical rightsizing, a customizable buffer of spare nodes and optimized utilization of the perfect blend of spot instances, reserved capacity and on-demand resources. Our integration with eksctl allows you to streamline and optimize the entire EKS process, from initially creating your cluster to managing and optimizing it on an ongoing basis.
In this post, we will discuss how to create a Kubernetes cluster with eksctl using Ocean by Spot and explain how to easily migrate an existing unmanaged nodegroups into Ocean-managed ones so you can spend more time with other tasks instead of managing infrastructure.
Features
- Create, get, list, update and delete clusters
- Create, drain and delete nodegroups
- Mix unmanaged and Ocean-managed nodegroups for gradual migration
- Auto Scaling of Ocean-managed nodegroups
- Auto installation of the Ocean controller
- IAM management and add-on policies
Prerequisites
Configure your credentials (if you are not an Ocean user, sign up for free here)
To use environment variables, run the following commands.
$ export SPOTINST_TOKEN=<spotinst_token> $ export SPOTINST_ACCOUNT=<spotinst_account>
To use the credentials file, run the spotctl configure command.
$ spotctl configure ? Enter your access token [? for help] ********************************** ? Select your default account [Use arrows to move, ? for more help] > act-01234567 (prod) act-0abcdefg (dev)
Or, manually create an INI formatted file like this:
[default] token = <spotinst_token> account = <spotinst_account>
and place it in:
- Unix/Linux/macOS: ~/.spotinst/credentials
- Windows: %UserProfile%\.spotinst/credentials
For more information, see Getting Started.
Configure your AWS credentials
To use environment variables, run the following commands.
$ export AWS_ACCESS_KEY_ID=<aws_access_key> $ export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
To use the credentials file, run the aws configure command.
$ aws configure AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: region-code Default output format [None]: json
For more information, see Configuring the AWS CLI.
Installation
Download and extract the eksctl binary with the following command.
$ curl -sfL https://spotinst-public.s3.amazonaws.com/integrations/kubernetes/eksctl/eksctl.sh | sh
Move the extracted binary.
$ sudo mv ./bin/eksctl /usr/local/bin && rm -rf ./bin
Test that your installation was successful with the following command.
$ eksctl version
NOTE: The version should be 0.15.0. If not, check your terminal output for any installation errors, or manually download an archive of the release for your operating system from the releases page, extract eksctl, and then execute it.
Create Your EKS Cluster and Worker Nodes
Using command-line flags
Create your cluster and worker nodes with the following command. Replace the example values with your own values.
$ eksctl create cluster \ --name prod \ --nodegroup-name standard-workers \ --spot-ocean
NOTE: The `spot-ocean` command-line flag enables Ocean integration.
Using configuration files
First, create a `cluster.yaml` file to hold your cluster and worker nodes configuration.
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: prod tags: creator: liran environment: prod nodeGroups: - name: standard-workers [... nodegroup standard fields; ssh, tags, etc.] spotOcean: {}
NOTE: The `spotOcean: {}` section enables Ocean integration and uses all defaults.
If you’d like to configure your Ocean integration, create a `cluster.yaml` file with the following configuration:
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: prod tags: creator: liran environment: prod nodeGroups: - name: standard-workers [... nodegroup standard fields; ssh, tags, etc.] spotOcean: strategy: # Percentage of Spot instances that would spin up from the desired # capacity. spotPercentage: 100 # Allow Ocean to utilize any available reserved instances first before # purchasing Spot instances. utilizeReservedInstances: true # Launch On-Demand instances in case of no Spot instances available. fallbackToOnDemand: true autoScaler: # Enable the Ocean autoscaler. enabled: true # Cooldown period between scaling actions. cooldown: 300 # Spare resource capacity management enabling fast assignment of Pods # without waiting for new resources to launch. headrooms: # Number of CPUs to allocate. CPUs are denoted in millicores, where # 1000 millicores = 1 vCPU. - cpuPerUnit: 2 # Number of GPUs to allocate. gpuPerUnit: 0 # Amount of memory (MB) to allocate. memoryPerUnit: 64 # Number of units to retain as headroom, where each unit has the # defined CPU and memory. numOfUnits: 1 compute: instanceTypes: # Instance types allowed in the Ocean cluster. Cannot be configured # if the blacklist is configured. whitelist: # OR blacklist - t2.large - c5.large
Next, create your Amazon EKS cluster and worker nodes with the following command.
$ eksctl create cluster -f cluster.yaml
The output of the command will be as shown below (note that all the output from Spot is prefixed with `spot`).
[ℹ] eksctl version 0.15.0 [ℹ] using region region-code [ℹ] setting availability zones to [region-codec region-codea region-coded] [▶] spot: executing pre-creation actions [▶] spot: checking whether ocean cluster should be created [▶] spot: will create ocean cluster using nodegroup "standard-workers" [REDACTED] [ℹ] 2 sequential tasks: { create cluster control plane "prod", 2 sequential sub-tasks: { spot: create ocean cluster, create nodegroup "standard-workers" } } [REDACTED] [ℹ] building cluster stack "eksctl-prod-cluster" [ℹ] deploying stack "eksctl-prod-cluster" [REDACTED] [ℹ] building nodegroup stack "eksctl-poc-nodegroup-ocean" [ℹ] deploying stack "eksctl-poc-nodegroup-ocean" [REDACTED] [ℹ] building nodegroup stack "eksctl-poc-nodegroup-standard-workers" [ℹ] deploying stack "eksctl-prod-nodegroup-standard-workers" [REDACTED] [✔] all EKS cluster resources for "prod" have been created [REDACTED] [▶] spot: executing post-creation actions [▶] spot: installing ocean controller [REDACTED] [✔] all EKS cluster resources for "prod" have been created [✔] EKS cluster "prod" in "region-code" region is ready
Cluster provisioning usually takes between 10 and 15 minutes. When your cluster is ready, test that your kubectl configuration is correct.
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1m
NOTE: If you receive the error “aws-iam-authenticator”: executable file not found in $PATH, your kubectl isn’t configured for Amazon EKS. For more information, see Installing aws-iam-authenticator.
Finally, test the installation of the Ocean controller.
$ kubectl get deployment --namespace kube-system NAME READY UP-TO-DATE AVAILABLE AGE spotinst-kubernetes-cluster-controller 1/1 1 1 5m
Migrate Your Existing Worker Nodes
Step 1. Create a new Ocean-managed nodegroup
Using command-line flags
Create a new Ocean-managed nodegroup of worker nodes with the following command. Replace the example values with your own values.
$ eksctl create nodegroup \ --cluster <cluster-name> \ --nodegroup-name <ocean-nodegroup-name> \ --spot-ocean
NOTE: The `spot-ocean` command-line flag enables Ocean integration.
Using configuration files
First, update your `cluster.yaml` configuration file by renaming your unmanaged nodegroups and adding the Ocean configuration.
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: prod tags: creator: liran environment: prod nodeGroups: - name: ocean-standard-workers [... nodegroup standard fields; ssh, tags, etc.] spotOcean: {}
NOTE: The `spotOcean: {}` section enables Ocean integration and uses all defaults.
Next, create apply the changes to create a new Ocean-managed nodegroup with the following command:
$ eksctl create nodegroup -f cluster.yaml
Step 2. Migrate Your Workload
Safely evict all of your pods from the nodes of the unmanaged nodegroup with the following command:
$ eksctl drain nodegroup \ --cluster <cluster-name> \ --nodegroup-name <unmanaged-nodegroup-name>
Step 3. Delete the Unmanaged Nodegroup [optional]
Remove the nodes by deleting the unmanaged nodegroup with the following command:
$ eksctl delete nodegroup \ --cluster <cluster-name> \ --nodegroup-name <unmanaged-nodegroup-name>
And that’s all folks!