Unmarshal JSON with some known, and some unknown field names

Question

I have the following JSON

{"a":1, "b":2, "?":1, "??":1}

I know that it has the "a" and "b" fields, but I don't know the names of other fields. So I want to unmarshal it in the following type:

type Foo struct {
  // Known fields
  A int `json:"a"`
  B int `json:"b"`
  // Unknown fields
  X map[string]interface{} `json:???` // Rest of the fields should go here.
}

How do I do that?

Whiffle

Answer 1

Unmarshal twice

One option is to unmarshal twice: once into a value of type Foo and once into a value of type map[string]interface{} and removing the keys "a" and "b":

type Foo struct {
    A int                    `json:"a"`
    B int                    `json:"b"`
    X map[string]interface{} `json:"-"` // Rest of the fields should go here.
}

func main() {
    s := `{"a":1, "b":2, "x":1, "y":1}`
    f := Foo{}
    if err := json.Unmarshal([]byte(s), &f); err != nil {
        panic(err)
    }

    if err := json.Unmarshal([]byte(s), &f.X); err != nil {
        panic(err)
    }
    delete(f.X, "a")
    delete(f.X, "b")

    fmt.Printf("%+v", f)
}

Output (try it on the Go Playground):

{A:1 B:2 X:map[x:1 y:1]}

Unmarshal once and manual handling

Another option is to unmarshal once into an map[string]interface{} and handle the Foo.A and Foo.B fields manually:

type Foo struct {
    A int                    `json:"a"`
    B int                    `json:"b"`
    X map[string]interface{} `json:"-"` // Rest of the fields should go here.
}

func main() {
    s := `{"a":1, "b":2, "x":1, "y":1}`
    f := Foo{}
    if err := json.Unmarshal([]byte(s), &f.X); err != nil {
        panic(err)
    }
    if n, ok := f.X["a"].(float64); ok {
        f.A = int(n)
    }
    if n, ok := f.X["b"].(float64); ok {
        f.B = int(n)
    }
    delete(f.X, "a")
    delete(f.X, "b")

    fmt.Printf("%+v", f)
}

Output is the same (Go Playground):

{A:1 B:2 X:map[x:1 y:1]}

Amal

Anyway to automate the handling of A and B? When handling structs with ~20 instead of 2 fields this will lead to really lengthy code. – Centripetal

Answer 2

It's not nice, but you could to it by implementing Unmarshaler:

type _Foo Foo

func (f *Foo) UnmarshalJSON(bs []byte) (err error) {
    foo := _Foo{}

    if err = json.Unmarshal(bs, &foo); err == nil {
        *f = Foo(foo)
    }

    m := make(map[string]interface{})

    if err = json.Unmarshal(bs, &m); err == nil {
        delete(m, "a")
        delete(m, "b")
        f.X = m
    }

    return err
}

The type _Foo is necessary to avoid recursion while decoding.

Jacinto

Why would using just Foo cause recursion? – Autograft

btw I used this with some good success. example here: play.golang.org/p/WLeEJIESg6 – Autograft

I also think this solution of leveraging the bson module is nice, especially if you already include it as a dependency: devel.io/2013/08/19/go-handling-arbitrary-json – Autograft

"Why would using just Foo cause recursion?" - @Autograft The json package checks if a type has UnmarshalJSON() defined, and if so it calls that implementation. So we arrive at the top of the function, then we call Unmarshal() on f, and the json package checks if Foo has UnmarshalJSON() defined, and it does, so it calls it, and so on, to infinite recursion. The purpose of _Foo is to be a type that does not implement UnmarshalJSON(), to break the cycle. – Rangy

Chris' Link to devel.io is dead by now. To save you the search on the wayback machine, it can be found here: web.archive.org/web/20161019055501/http://devel.io/2013/08/19/… – Wilde

There is a proposal to add an 'inline' struct tag into golang core github.com/golang/go/issues/6213 – Marou

Answer 3

Simplest way is to use an interface like this:

var f interface{}
s := `{"a":1, "b":2, "x":1, "y":1}`

if err := json.Unmarshal([]byte(s), &f); err != nil {
    panic(err)
}

Go Playground example

Eyesight

It unmarshals this way, but how do you access the values after that? – Doreathadoreen

You have to use type switches to access the values. Possible types will be string, int, map[string]interface{}, nil, bool, and so forth. – Oligarch

Could you provide a small example? I'm looking for information on how to do it and I can't find the way, thanks – Kaufman

How to access the elements: go.dev/play/p/FkOE_X2dmqt – Aldred

How is this different from unmarshalling into a map[string]interface{}? – Volgograd

It's different to map[string]interface{} as interface{} will handle any structure or type like an int array [1, 2, 3] for example. – Eyesight

Answer 4

I use interface to unmarshal uncertain type json.

bytes := []byte(`{"name":"Liam","gender":1, "salary": 1}`)
var p2 interface{}
json.Unmarshal(bytes, &p2)
m := p2.(map[string]interface{})
fmt.Println(m)

Changchun

Answer 5

Almost single pass, uses json.RawMessage

We can unmarshal into map[string]json.RawMessage, and then unmarshal each field separately.

JSON will be tokenized twice, but that's quite cheap.

The following helper function can be used:

func UnmarshalJsonObject(jsonStr []byte, obj interface{}, otherFields map[string]json.RawMessage) (err error) {
    objValue := reflect.ValueOf(obj).Elem()
    knownFields := map[string]reflect.Value{}
    for i := 0; i != objValue.NumField(); i++ {
        jsonName := strings.Split(objValue.Type().Field(i).Tag.Get("json"), ",")[0]
        knownFields[jsonName] = objValue.Field(i)
    }

    err = json.Unmarshal(jsonStr, &otherFields)
    if err != nil {
        return
    }

    for key, chunk := range otherFields {
        if field, found := knownFields[key]; found {
            err = json.Unmarshal(chunk, field.Addr().Interface())
            if err != nil {
                return
            }
            delete(otherFields, key)
        }
    }
    return
}

Here is the complete code on Go Playground - http://play.golang.org/p/EtkJUzMmKt

Whiffle

This only works with objects, not arrays, nor strings, all of which are valid json – Feeder

Answer 6

Single Pass With Marshmallow

We use marshmallow to solve exactly that problem. It requires no explicit coding of any kind which keeps your code cleaner and more maintainable than other solutions, but it also provides the best performance (up to x3 faster than other solutions provided here, see benchmarks and results in the repo).

type Foo struct {
    A int `json:"a"`
    B int `json:"b"`
}

func main() {
    s := `{"a":1, "b":2, "x":1, "y":1}`
    f := Foo{}
    result, err := marshmallow.Unmarshal([]byte(s), &f)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", f)      // {A:1 B:2}
    fmt.Printf("%+v\n", result) // map[a:1 b:2 x:1 y:1]
}

Playground link

Marshmallow is used internally at PerimeterX for some time and we've recently decided to open-source it. We also wrote a blog post about how it helped us trim 70% of our JSON parsing costs in production.

Demogorgon

Answer 7

Single pass, use github.com/ugorji/go/codec

When unmarshaling into a map, encoding/json empties the map, but ugorji/go/codec doesn't. It also attempts to fill existing values, so we can put pointers to foo.A, foo.B into foo.X:

package main

import (
    "fmt"
    "github.com/ugorji/go/codec"
)

type Foo struct {
    A int
    B int
    X map[string]interface{}
}

func (this *Foo) UnmarshalJSON(jsonStr []byte) (err error) {
    this.X = make(map[string]interface{})
    this.X["a"] = &this.A
    this.X["b"] = &this.B
    return codec.NewDecoderBytes(jsonStr, &codec.JsonHandle{}).Decode(&this.X)
}

func main() {
    s := `{"a":1, "b":2, "x":3, "y":[]}`
    f := &Foo{}
    err := codec.NewDecoderBytes([]byte(s), &codec.JsonHandle{}).Decode(f)
    fmt.Printf("err = %v\n", err)
    fmt.Printf("%+v\n", f)
}

Whiffle

Answer 8

Use Hashicorp's map-to-struct decoder, which keeps track of unused fields: https://godoc.org/github.com/mitchellh/mapstructure#example-Decode--Metadata

It's two-pass, but you don't have to use known field names anywhere.

func UnmarshalJson(input []byte, result interface{}) (map[string]interface{}, error) {
    // unmarshal json to a map
    foomap := make(map[string]interface{})
    json.Unmarshal(input, &foomap)

    // create a mapstructure decoder
    var md mapstructure.Metadata
    decoder, err := mapstructure.NewDecoder(
        &mapstructure.DecoderConfig{
            Metadata: &md,
            Result:   result,
        })
    if err != nil {
        return nil, err
    }

    // decode the unmarshalled map into the given struct
    if err := decoder.Decode(foomap); err != nil {
        return nil, err
    }

    // copy and return unused fields
    unused := map[string]interface{}{}
    for _, k := range md.Unused {
        unused[k] = foomap[k]
    }
    return unused, nil
}

type Foo struct {
    // Known fields
    A int
    B int
    // Unknown fields
    X map[string]interface{} // Rest of the fields should go here.
}

func main() {
    s := []byte(`{"a":1, "b":2, "?":3, "??":4}`)

    var foo Foo
    unused, err := UnmarshalJson(s, &foo)
    if err != nil {
        panic(err)
    }

    foo.X = unused
    fmt.Println(foo) // prints {1 2 map[?:3 ??:4]}
}

Lory