EKS Done Right – From Control Plane To Worker Nodes

Liran Polak
Liran Polak
Chief Architect
Mar 11th, 2020
5 Min read
Amazon Web Services
eksctl integration

As the official AWS CLI tool, eksctl is an open-source CLI that has gained popularity within the Kubernetes community for easily creating Elastic Kubernetes Service (EKS) clusters. One of the benefits of EKS itself is that the Kubernetes control plane is handled entirely by AWS, freeing you from managing high availability for your control plane(s) as well as from handling all ongoing maintenance such as Kubernetes upgrades.  

However, when it comes to your clusters’ worker nodes, you still need to find a solution that simplifies container infrastructure management by handling challenges like:

  • Unavailable resources due to autoscaling that only looks at cluster CPU and Memory and not actual Pod requirements
  • Cost visibility (showback) that allocates spend based on namespaces, resources, labels, and annotations
  • Pod rightsizing based on analysis of their actual resource consumption
  • Handling immediate container scaling for high priority workloads
  • Reducing cloud infrastructure costs using spot instances and preemptible VMs 

Fortunately, Ocean by Spot takes care of all these time-consuming, DevOps activities, with our container-driven autoscaling, cloud-native cost allocation, vertical rightsizing, a customizable buffer of spare nodes and optimized utilization of the perfect blend of spot instances, reserved capacity and on-demand resources. Our integration with eksctl allows you to streamline and optimize the entire EKS process, from initially creating your cluster to managing and optimizing it on an ongoing basis. 

In this post, we will discuss how to create a Kubernetes cluster with eksctl using Ocean by Spot and explain how to easily migrate an existing unmanaged nodegroups into Ocean-managed ones so you can spend more time with other tasks instead of managing infrastructure.

Features

  • Create, get, list, update and delete clusters
  • Create, drain and delete nodegroups
  • Mix unmanaged and Ocean-managed nodegroups for gradual migration
  • Auto Scaling of Ocean-managed nodegroups
  • Auto installation of the Ocean controller
  • IAM management and add-on policies

Prerequisites

Configure your credentials (if you are not an Ocean user, sign up for free here)

To use environment variables, run the following commands.

$ export SPOTINST_TOKEN=<spotinst_token>
$ export SPOTINST_ACCOUNT=<spotinst_account>

To use the credentials file, run the spotctl configure command.

$ spotctl configure
? Enter your access token [? for help] **********************************
? Select your default account  [Use arrows to move, ? for more help]
> act-01234567 (prod)
  act-0abcdefg (dev)

Or, manually create an INI formatted file like this:

[default]
token   = <spotinst_token>
account = <spotinst_account>

and place it in:

  • Unix/Linux/macOS: ~/.spotinst/credentials
  • Windows: %UserProfile%\.spotinst/credentials

For more information, see Getting Started.

Configure your AWS credentials

To use environment variables, run the following commands.

$ export AWS_ACCESS_KEY_ID=<aws_access_key>
$ export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

To use the credentials file, run the aws configure command.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: region-code
Default output format [None]: json

For more information, see Configuring the AWS CLI.

Installation

Download and extract the eksctl binary with the following command.

$ curl -sfL https://spotinst-public.s3.amazonaws.com/integrations/kubernetes/eksctl/eksctl.sh | sh

Move the extracted binary.

$ sudo mv ./bin/eksctl /usr/local/bin && rm -rf ./bin

Test that your installation was successful with the following command.

$ eksctl version

NOTE: The version should be 0.15.0. If not, check your terminal output for any installation errors, or manually download an archive of the release for your operating system from the releases page, extract eksctl, and then execute it.

Create Your EKS Cluster and Worker Nodes

Using command-line flags

Create your cluster and worker nodes with the following command. Replace the example values with your own values.

$ eksctl create cluster \
    --name prod \
    --nodegroup-name standard-workers \
    --spot-ocean

NOTE: The `spot-ocean` command-line flag enables Ocean integration.

Using configuration files

First, create a `cluster.yaml` file to hold your cluster and worker nodes configuration.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod
  tags:
    creator: liran
    environment: prod

nodeGroups:
  - name: standard-workers
    [... nodegroup standard fields; ssh, tags, etc.]

    spotOcean: {}

NOTE: The `spotOcean: {}` section enables Ocean integration and uses all defaults.

If you’d like to configure your Ocean integration, create a `cluster.yaml` file with the following configuration:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod
  tags:
    creator: liran
    environment: prod

nodeGroups:
  - name: standard-workers
    [... nodegroup standard fields; ssh, tags, etc.]

    spotOcean:
      strategy:
        # Percentage of Spot instances that would spin up from the desired 
        # capacity.
        spotPercentage: 100

        # Allow Ocean to utilize any available reserved instances first before 
        # purchasing Spot instances.
        utilizeReservedInstances: true

        # Launch On-Demand instances in case of no Spot instances available.
        fallbackToOnDemand: true

      autoScaler:
        # Enable the Ocean autoscaler. 
        enabled: true

        # Cooldown period between scaling actions.
        cooldown: 300

        # Spare resource capacity management enabling fast assignment of Pods 
        # without waiting for new resources to launch.
        headrooms:
            # Number of CPUs to allocate. CPUs are denoted in millicores, where 
            # 1000 millicores = 1 vCPU.
          - cpuPerUnit: 2

            # Number of GPUs to allocate.
            gpuPerUnit: 0

            # Amount of memory (MB) to allocate.
            memoryPerUnit: 64

            # Number of units to retain as headroom, where each unit has the 
            # defined CPU and memory.
            numOfUnits: 1

      compute:
        instanceTypes:
          # Instance types allowed in the Ocean cluster. Cannot be configured 
          # if the blacklist is configured.
          whitelist: # OR blacklist
            - t2.large
            - c5.large

Next, create your Amazon EKS cluster and worker nodes with the following command.

$ eksctl create cluster -f cluster.yaml

The output of the command will be as shown below (note that all the output from Spot is prefixed with `spot`).

[ℹ]  eksctl version 0.15.0
[ℹ]  using region region-code
[ℹ]  setting availability zones to [region-codec region-codea region-coded]
[▶]  spot: executing pre-creation actions
[▶]  spot: checking whether ocean cluster should be created
[▶]  spot: will create ocean cluster using nodegroup "standard-workers"

[REDACTED]

[ℹ]  2 sequential tasks: { create cluster control plane "prod", 2 sequential sub-tasks: { spot: create ocean cluster, create nodegroup "standard-workers" } }

[REDACTED]

[ℹ]  building cluster stack "eksctl-prod-cluster"
[ℹ]  deploying stack "eksctl-prod-cluster"

[REDACTED]

[ℹ]  building nodegroup stack "eksctl-poc-nodegroup-ocean"
[ℹ]  deploying stack "eksctl-poc-nodegroup-ocean"

[REDACTED]

[ℹ]  building nodegroup stack "eksctl-poc-nodegroup-standard-workers"
[ℹ]  deploying stack "eksctl-prod-nodegroup-standard-workers"

[REDACTED]

[✔]  all EKS cluster resources for "prod" have been created

[REDACTED]

[▶]  spot: executing post-creation actions
[▶]  spot: installing ocean controller

[REDACTED]

[✔]  all EKS cluster resources for "prod" have been created
[✔]  EKS cluster "prod" in "region-code" region is ready

Cluster provisioning usually takes between 10 and 15 minutes. When your cluster is ready, test that your kubectl configuration is correct.

$ kubectl get svc

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
svc/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   1m

NOTE: If you receive the error “aws-iam-authenticator”: executable file not found in $PATH, your kubectl isn’t configured for Amazon EKS. For more information, see Installing aws-iam-authenticator.

Finally, test the installation of the Ocean controller.

$ kubectl get deployment --namespace kube-system

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
spotinst-kubernetes-cluster-controller   1/1     1            1           5m

Migrate Your Existing Worker Nodes

Step 1. Create a new Ocean-managed nodegroup

Using command-line flags

Create a new Ocean-managed nodegroup of worker nodes with the following command. Replace the example values with your own values.

$ eksctl create nodegroup \
    --cluster <cluster-name> \
    --nodegroup-name <ocean-nodegroup-name> \
    --spot-ocean

NOTE: The `spot-ocean` command-line flag enables Ocean integration.

Using configuration files

First, update your `cluster.yaml` configuration file by renaming your unmanaged nodegroups and adding the Ocean configuration.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod
  tags:
    creator: liran
    environment: prod

nodeGroups:
  - name: ocean-standard-workers
    [... nodegroup standard fields; ssh, tags, etc.]

    spotOcean: {}

NOTE: The `spotOcean: {}` section enables Ocean integration and uses all defaults.

Next, create apply the changes to create a new Ocean-managed nodegroup with the following command:

$ eksctl create nodegroup -f cluster.yaml

Step 2. Migrate Your Workload

Safely evict all of your pods from the nodes of the unmanaged nodegroup with the following command:

$ eksctl drain nodegroup \
    --cluster <cluster-name> \
    --nodegroup-name <unmanaged-nodegroup-name>

Step 3. Delete the Unmanaged Nodegroup [optional]

Remove the nodes by deleting the unmanaged nodegroup with the following command:

$ eksctl delete nodegroup \
    --cluster <cluster-name> \
    --nodegroup-name <unmanaged-nodegroup-name>

And that’s all folks!

Reference: https://github.com/spotinst/weaveworks-eksctl