How to deserialize Kubernetes YAML file

Question

How can I deserialize a Kubernetes YAML file into an Go struct? I took a look into the kubectl code, but somehow I get an error for every YAML file:

no kind "Deployment" is registered for version "apps/v1beta1"

This is an MWE:

package main

import (
    "fmt"

    "k8s.io/client-go/pkg/api"
)

var service = `
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
`

func main() {
    decode := api.Codecs.UniversalDecoder().Decode
    //decode := api.Codecs.UniversalDeserializer().Decode

    obj, _, err := decode([]byte(service), nil, nil)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%#v\n", obj)
}

I am using client version 2.0.0. The glide.yaml looks like this:

package: test/stackoverflow
import:
- package: k8s.io/client-go
  version: ^2.0.0

These are the references to kubectl:

• https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/apply.go#L637
• https://github.com/kubernetes/kubernetes/blob/43ac38e29e6ecf83e78bc7c5d9f804310b051c95/pkg/kubectl/cmd/util/factory_client_access.go#L205-L213

Unfortunately, the docs are very confusing to me, so I have no idea how to tackle this problem.

Edit:

This problem also exists with other resource types:

• no kind "Service" is registered for version "v1"

Oba

Wanna give Go dep a try? hackernoon.com/… – Postglacial

This has nothing to do with dependency management. – Oba

Yup, I know (hence my answer below) but trying to encourage people to move to a more future-proof alternative ;) – Postglacial

Answer 1

You need to import _ "k8s.io/client-go/pkg/apis/extensions/install" otherwise the schema is empty, see also docs.

The complete working example is:

$ go get -u github.com/golang/dep/cmd/dep
$ dep init
$ go run main.go

With the following main.go:

package main

import (
    "fmt"

    "k8s.io/client-go/pkg/api"
    _ "k8s.io/client-go/pkg/api/install"
    _ "k8s.io/client-go/pkg/apis/extensions/install"
)

var deployment = `
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
  metadata:
    labels:
      run: my-nginx
  spec:
    containers:
    - name: my-nginx
      image: nginx
      ports:
      - containerPort: 80
`

func main() {
    // decode := api.Codecs.UniversalDecoder().Decode
    decode := api.Codecs.UniversalDeserializer().Decode

    obj, _, err := decode([]byte(deployment), nil, nil)
    if err != nil {
        fmt.Printf("%#v", err)
    }

    fmt.Printf("%#v\n", obj)
}

Note that I also imported _ "k8s.io/client-go/pkg/api/install" for you so that you can use objects in v1 such as pods or services.

EDIT: Kudos to my colleague Stefan Schimanski who proposed the initial solution.

Postglacial

Whoa. That's very magic ;-) On the other hand it's understandable, if one wants a extendable system. I guess the docs are improvable. Thanks! – Oba

Pretty close to magic indeed and I was noodling around for a while myself a bit before Stefan came to the rescue. The docs need tons of improvement alright, agreed! – Postglacial

Hi @MichaelHausenblas, I'm trying to make this work (parse a k8s yaml to use Go structs) but cannot make it work with dep (or vndr). Copying you exact same example, and running "dep init" gets: No versions of k8s.io/apimachinery met constraints: master: unable to update checked out version: : command failed: [git checkout 1168e538ea3ccf444854d1fdd5681d2d876680a7]: exit status 128 (next comment) – Torhert

release-1.6: unable to update checked out version: : command failed: [git checkout 35be0db31bd6e5635ef6366708dc2b137392f6a2]: exit status 128 release-1.7: unable to update checked out version: : command failed: [git checkout 8ab5f3d8a330c2e9baaf84e39042db8d49034ae2]: exit status 128 – Torhert

Ok I've been able to fix it importing "k8s.io/client-go/pkg/api", "k8s.io/client-go/pkg/api/install", "k8s.io/client-go/pkg/apis/apps/install" and "k8s.io/client-go/pkg/apis/extensions/install". But if I run it, I get a bunch of # github.com/amegianeg/example/vendor/k8s.io/client-go/pkg/api vendor/k8s.io/client-go/pkg/api/types.generated.go:633: too many arguments in call to r.DecodeBytes have ([]byte, bool, bool) want ([]byte, bool) – Torhert

What if there is multiple object types in the same file? I am having the exact scenario. – Albion

@MichaelHausenblas Looks like the above example is deprecated, as that import _ "k8s.io/client-go/pkg/api/install" is not working. Also clicking on docs shows Not Found. Can you please update the same. – Kailakaile

Accepted answer is outdated - there is no more k8s.io/client-go/api/install package. Please see github discussion with corrected version: github.com/kubernetes/client-go/issues/… – Essential

Answer 2

I've been using api machinery'sk8s.io/apimachinery/pkg/util/yaml to decode kubernete's deployment and service manifests.

import (
   "fmt"
   "bytes"
   appsv1 "k8s.io/api/apps/v1"
    k8Yaml "k8s.io/apimachinery/pkg/util/yaml"
)
...

d := &appsv1.Deployment{}
dec := k8Yaml.NewYAMLOrJSONDecoder(bytes.NewReader([]byte(deploymentManifest)), 1000)

if err := dec.Decode(&d); err != nil {
        return nil, err
}

fmt.Printf("%+v", d)

Plot

How do you fetch deployment.yaml/service.yaml of a particular pod – Charlet

if you have the pod object, you can check the ownerReferences to get the replicaSet that owns the pod. In the replicaset object, check the value of ownerReferences to get the deployment. – Plot

The service is trickier, you can use the labels on the pod to find a match in the services, but it's not very efficient. – Plot

Answer 3

import (
    "fmt"
    "gopkg.in/yaml.v2"
    "log"
    //corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime/serializer/json"
    "k8s.io/client-go/kubernetes/scheme"
    v1alpha1 "github.com/argoproj/argo-cd/pkg/apis/application/v1alpha1"
)

....

func ParseYaml2(yaml []byte) (v1alpha1.Application, error) {
    // Create a YAML serializer.  JSON is a subset of YAML, so is supported too.
    s := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme,
        scheme.Scheme)

    // Decode the YAML to an object.
    var app v1alpha1.Application
    _, _, err := s.Decode(yaml, nil, &app)
    if err != nil {
        panic(err)
    }
    //fmt.Printf("%#v\n", app)
    return app, err
}
---
go.mod
// https://github.com/argoproj/argo-cd/issues/4055
replace github.com/argoproj/argo-cd => github.com/argoproj/argo-cd v1.5.5

var yaml2 = []byte(`
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
...

var app v1alpha1.Application
app,err := ParseYaml2(yaml2)
// Types from https://github.com/argoproj/argo-cd/blob/master/pkg/apis/application/v1alpha1/types.go
    fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Path)
    fmt.Printf("--- t:\n%s\n\n", app.Spec.Source.Helm.ValueFiles)
----

Briny