Unable to obtain kubeconfig of an AWS EKS cluster in Go code

Question

I have created an AWS EKS cluster. In order to obtain its kubeconfig, I usually run aws eks update-kubeconfig --name cluster-name --region us-west-2 using a shell.

However, I now wish to obtain the kubeconfig in Go without having to run anything in the shell (the goal being to create and then manipulate an EKS cluster in a Go test). I am able to describe an EKS cluster using this code:

package main

import (
    "fmt"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/eks"
)

func main() {
    sess := session.Must(session.NewSession(&aws.Config{
        Region: aws.String("us-west-2"),
    }))
    eksSvc := eks.New(sess, aws.NewConfig().WithRegion("us-west-2"))


    clusterOutput, err := eksSvc.DescribeCluster(&eks.DescribeClusterInput{
        Name: aws.String("cluster-name"),
    })
    if err != nil {
        panic(err)
    }
    fmt.Printf("%#v\n", clusterOutput)
}

After that, I currently have no idea about how could I get the kubeconfig of that cluster in order to then use it with the Go client for Kubernetes without having to use aws eks separately.

I have checked the AWS documentation, AWS CLI codebase, and eksctl codebase with no luck so far. The connection to an EKS cluster is only documented in this webpage and it uses a shell: https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection/

Any ideas?

Lobule

Answer 1

The general flow goes something like this:

1. DescribeCluster (as you have done) and extract some necessary data
2. Using the necessary data, get a token using aws-iam-authenticator's package token
3. Using that token, create a Kubernetes clientset with the help of client-go.

package main

import (
    "encoding/base64"
    "log"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"

    "github.com/aws/aws-sdk-go/service/eks"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"

    "sigs.k8s.io/aws-iam-authenticator/pkg/token"
)

func newClientset(cluster *eks.Cluster) (*kubernetes.Clientset, error) {
    log.Printf("%+v", cluster)
    gen, err := token.NewGenerator(true, false)
    if err != nil {
        return nil, err
    }
    opts := &token.GetTokenOptions{
        ClusterID: aws.StringValue(cluster.Name),
    }
    tok, err := gen.GetWithOptions(opts)
    if err != nil {
        return nil, err
    }
    ca, err := base64.StdEncoding.DecodeString(aws.StringValue(cluster.CertificateAuthority.Data))
    if err != nil {
        return nil, err
    }
    clientset, err := kubernetes.NewForConfig(
        &rest.Config{
            Host:        aws.StringValue(cluster.Endpoint),
            BearerToken: tok.Token,
            TLSClientConfig: rest.TLSClientConfig{
                CAData: ca,
            },
        },
    )
    if err != nil {
        return nil, err
    }
    return clientset, nil
}

func main() {
    name := "wonderful-outfit-1583362361"
    region := "us-east-2"
    sess := session.Must(session.NewSession(&aws.Config{
        Region: aws.String(region),
    }))
    eksSvc := eks.New(sess)

    input := &eks.DescribeClusterInput{
        Name: aws.String(name),
    }
    result, err := eksSvc.DescribeCluster(input)
    if err != nil {
        log.Fatalf("Error calling DescribeCluster: %v", err)
    }
    clientset, err := newClientset(result.Cluster)
    if err != nil {
        log.Fatalf("Error creating clientset: %v", err)
    }
    nodes, err := clientset.CoreV1().Nodes().List(metav1.ListOptions{})
    if err != nil {
        log.Fatalf("Error getting EKS nodes: %v", err)
    }
    log.Printf("There are %d nodes associated with cluster %s", len(nodes.Items), name)
}

Here's my go.mod for versions:

module github.com/swoldemi/sandbox

go 1.14

require (
    github.com/aws/aws-sdk-go v1.29.19
    k8s.io/apimachinery v0.0.0-20190612125636-6a5db36e93ad
    k8s.io/client-go v0.0.0-20190425172711-65184652c889
    sigs.k8s.io/aws-iam-authenticator v0.5.0
)

Hu

Great answer! You may also want to pass the session in the GetTokenOptions struct. See my answer. – Skeie

@AndrésMejía and @Hu Is there a way to pass on access_key_id and secret_access_key to the session.NewSession() instead of having a requirement of looking into aws.config file or any other file. – Glutenous

@SuryavanshiVirendrasingh Yes, it's possible! You can just export the 2 keys to your environment and NewSession will find them from the environment using the default order of precedence * Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY – Hu

@Hu Can you please elaborate on what do you mean by exporting the 2 keys to the environment? – Glutenous

@SuryavanshiVirendrasingh Assuming you're running your program in a shell: AWS_ACCESS_KEY_ID=access_key_id AWS_SECRET_ACCESS_KEY=secret_access_key go run youprogram.go See "How to set environment variables" on this page: docs.aws.amazon.com/cli/latest/userguide/… – Hu

@Hu Ohh, I see what you meant by exporting the keys to the environment. But I don't want to do that as this will run in multiple environments and I don't think it will be a good practice to export it every time into different environments. So, is it possible to pass on directly the credentials to session.NewSession(), which I am receiving through Kafka stream through another service. – Glutenous

@SuryavanshiVirendrasingh I don't recommend that you try and pass credentials across a network, between services, from a location that isn't STS/IAM, but here's a snippet on how to create your own provider using credentials.Value and create a session suing those keys: play.golang.org/p/kmbVrBE_UT7 – Hu

@SuryavanshiVirendrasingh sess, err := session.NewSessionWithOptions(session.Options{ Config: aws.Config{ Credentials: credentials.NewStaticCredentials(accessKey, secretKey, ""), }}, ) – Missing

wait..does nodes in the above code return kube config similar to the command line version aws update-kubeconfig? – Athallia

Answer 2

Adding to Simon's great answer, you may want to also pass the session in the GetTokenOptions struct, like so:

opts := &token.GetTokenOptions{
        ClusterID: aws.StringValue(cluster.Name),
        Session: sess,
    }
    tok, err := gen.GetWithOptions(opts)

Otherwise the gen.GetWithOptions(opts) call will try to read your AWS credentials from local sources (e.g. ~/.aws/credentials) and may fail with this error:

NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Sexless