Failed to construct REST client

Question

I'm trying to use kubernetes-alpha provider in Terraform, but I have "Failed to construct REST client" error message. I'm using tfk8s to convert my yaml file to terraform code.

I make the seme declaration for the provider than kubernetes, and my kubernetes provider work correctely

provider "kubernetes-alpha" {
  host                   = "https://${data.google_container_cluster.primary.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.primary.master_auth[0].cluster_ca_certificate)
}

provider "kubernetes" {
  host                   = "https://${data.google_container_cluster.primary.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(data.google_container_cluster.primary.master_auth[0].cluster_ca_certificate)
}

resource "kubernetes_manifest" "exemple" {
  provider = kubernetes-alpha
  manifest = {
     # result of tfk8s
  }
}

the error message

somebody can help ?

Loosen

This has been happening to me for weeks for one of my environments. Hashicorp seems clueless – Dressy

This is also happening to me after successfully applying the custom resource, and destroying that particular terraform created kubernetes cluster via terraform destroy. Oddly enough, my custom resource is also a cert manager clusterIssuer object. – Flexor

Answer 1

It's rather unfortunate kubernetes_manifest doesn't allow it and that's due to it trying to resolve types. For the type resolution the devs chose to contact the Kubernetes API server. When you have a single main.tf (or single plan/apply) with creating the Kubernetes cluster and and the same time using the cluster you get into a missing dependency once you reach kubernetes_manifest. That's because of the Terraform files not being read just-in-time but only single-pass and execute anything a provider defines.

If you have a provider's configuration depending on the previous provider's resources being created, you get into these problems. For kubernetes_manifest however there's a quick solution - just use kubectl_manifest from the gavinbunney/kubectl provider and either put the yaml to a heredoc or into separate file/folder.

Raw:

resource "kubectl_manifest" "some-name" {
  yaml_body = <<YAML
some: yaml
with: ${var.something}
YAML
}

File:

resource "kubectl_manifest" "some-name" {
  yaml_body = "${file("./file.yaml")}"
}

Multiple files in folder:

data "kubectl_path_documents" "other-name" {
  pattern = "./folder/*.yaml"
}

resource "kubectl_manifest" "some-name" {
  for_each  = toset(data.kubectl_path_documents.other-name.documents)
  yaml_body = each.value
}

There is also built-in functionality for templating in the kubectl provider:

A set of helpful data resources to process directories of yaml files and inline templating is available. (source)

so you can use ${var.something} in the file itself.

If that doesn't help, there are always these for the rescue as the yaml_body is just a string thus you can work with it as with any other string:

• file
• templatefile
• replace()

Note: Intermixing kubernetes provider and kubectl provider doesn't seem to cause issues and I've been using it for a long time. The issue seems to be happening only for kubernetes_manifest and only because Terraform wants to resolve the types (which is understandable but also a flawed chicken&egg problem), so any other resource such as namespaces, pods, the standard stuff works.

Kurtiskurtosis

Answer 2

After some digging, I found that this resource requires a running kubernetes instance and config before the terraform plan will work properly. Best stated in github here: https://github.com/hashicorp/terraform-provider-kubernetes-alpha/issues/199#issuecomment-832614387

Basically, you have to have two steps to first terraform apply your main configuration to stand up kubernetes in your cloud, and then secondly terraform apply the CRD resource once that cluster has been established.

EDIT: I'm still trying to learn good patterns/practices for managing terraform config and found this pretty helpful. How to give a .tf file as input in Terraform Apply command?. I ended up just keeping the cert manager CRD as a standard kubernetes manifest yaml that I apply per-cluster with other application helm charts.

Flexor