What's the best way to maintain un-parsed JSON fields in Go?

Question

I'd like to decode a JSON blob into a Go struct, manipulate on it, and encode it back to JSON. However, there are dynamic fields in the JSON that aren't relevant to my struct and I want to maintain them when I serialize back to JSON.

For example:

{ "name": "Joe Smith", 
  "age": 42, 
  "phone": "614-555-1212", 
  "debug": True, 
  "codeword": "wolf" }

type Person struct {
    Name string
    Age uint
    Phone string
}

var p Person
json.Unmarshal(data, &p)
// Happy birthday
p.Age++
data, _ = json.Marshal(p)

// Any way to maintain the "debug" and "codeword" fields -- which might not
// be known ahead of time?

I know one possibility is to decode everything into a map[string]interface{} but boy, do things get ugly when you do that.

Is there any way to have the best of both worlds?

Eudiometer

related question with more answers #33437230 – Desperado

Answer 1

With encoding/json there's no way to do decode a struct and save unknown fields at the same level for posterior re-encoding. What you can do is choose not to decode part of the struct by using the json.RawMessage type, along the lines of:

type Person struct {
    Name    string
    Address json.RawMessage
}

You might workaround that by implementing your own Unmarshaler that decodes the document into a map and saves the unknown keys in a field of the struct, and then have a counterpart Marshaler that puts the fields back before marshaling.

Just out of curiosity, the feature you're looking for does exist in labix.org/v2/mgo/bson, via the inline tag flag, and the goal was to solve precisely the use case you're describing.

Alula

Answer 2

Turns out I wrote my own library to do this: https://github.com/joeshaw/json-lossless

It builds on top of go-simplejson, keeping the parsed JSON state in a simplejson.Json and proxying state between it and the struct whenever the struct is marshaled or unmarshaled.

Example usage:

package main

import (
    "encoding/json"
    "fmt"
    "time"

    "github.com/joeshaw/json-lossless"
)

type Person struct {
    lossless.JSON       `json:"-"`

    Name string         `json:"name"`
    Age int             `json:"age"`
    Birthdate time.Time `json:"birthdate"`
}

func (p *Person) UnmarshalJSON(data []byte) error {
    return p.JSON.UnmarshalJSON(p, data)
}

func (p Person) MarshalJSON() []byte, error) {
    return p.JSON.MarshalJSON(p)
}

var jsondata = []byte(`
{"name": "David Von Wolf",
 "age": 33,
 "birthdate": "1980-09-16T10:44:40.295451647-04:00",
 "phone": "614-555-1212"}
`)

func main() {
    var p Person
    err := json.Unmarshal(jsondata, &p)
    if err != nil {
        panic(err)
    }

    // Set values on the struct
    p.Age++

    // Set arbitrary keys not in the struct
    p.Set("title", "Chief Wolf")

    fmt.Printf("%#v\n", p)

    data, err := json.Marshal(p)
    if err != nil {
        panic(err)
    }

    fmt.Println(string(data))
}

Prints (formatted for readability by me):

main.Person{JSON:lossless.JSON{json:(*simplejson.Json)(0x21020a190)}, 
            Name:"David Von Wolf", 
            Age:34, 
            Birthdate:time.Time{sec:62473560280, 
                                nsec:295451647, 
                                loc:(*time.Location)(0x16de60)}}
{"age":34,
 "birthdate":"1980-09-16T10:44:40.295451647-04:00",
 "name":"David Von Wolf",
 "phone":"614-555-1212",
 "title": "Chief Wolf"}

Eudiometer

Answer 3

Package go-simplejson comes handy for this kind of jobs.

Carmelo

Indeed. The solution I seem to be settling on is using go-simplejson to deserialize initially, and then writing my own reflect-based code to marshal between a simplejson.Json object and the desired struct. – Eudiometer