How to update multiple documents in Solr with JSON?

Question

How to update multiple documents in Solr 4.5.1 with JSON? I tried this but it does not work:

POST /solr/mycore/update/json:

{
  "commit": {},
  "add": {
    "overwrite": true,
    "doc": [{
        "thumbnail": "/images/404.png",
        "url": "/404.html?1",
        "id": "demo:/404.html?1",
        "channel": "demo",
        "display_name": "One entry",
        "description": "One entry is not enough."
      }, {
        "thumbnail": "/images/404.png",
        "url": "/404.html?2",
        "id": "demo:/404.html?2",
        "channel": "demo",
        "display_name": "Another entry",
        "description": "Another entry is required."
      }
    ]
  }
}

Inherent

Answer 1

Solr expects one "add"-key in the JSON-structure for each document (which might seem weird, if you think about the original meaning of the key in the object), since it maps directly to the XML format when doing the indexing - and this way you can have metadata for each document by itself.

{
    "commit": {},
    "add": {
        "doc": {
            "id": "321321",
            "name": "barfoo"
        }
    },
    "add": {
        "doc": {
            "id": "123123",
            "name": "Foobar"        
        }
    }
}

.. works. I think allowing an array as the element referenced by "add" would make more sense, but I haven't dug further into the source or know the reasoning behind this.

Stereography

Thanks. This is an invalid JSON format, right? There is no chance to produce this output but doing it hand coded. – Inherent

It passes some online validators, but freeformatter.com/json-validator.html explains that "The JSON input is NOT valid according to RFC 4627 (JSON specification). Unexpected duplicate key:add at position 136." – Labyrinth

I get an error: "Error parsing JSON field value. Unexpected OBJECT_START at [50], field=add" – Bozuwa

@Bozuwa Please add a new question for any new issues - that way it'll be far more visible instead of just popping up in my comment feed. – Stereography

Answer 2

I understand that (at least) from versions 4.0 and older of solr, this has been fixed. Look at http://wiki.apache.org/solr/UpdateJSON.

In ./exampledocs/books.json there is an example of a json file with multiple documents.

[
{
"id" : "978-0641723445",
"cat" : ["book","hardcover"],
"name" : "The Lightning Thief",
"author" : "Rick Riordan",
"series_t" : "Percy Jackson and the Olympians",
"sequence_i" : 1,
"genre_s" : "fantasy",
"inStock" : true,
"price" : 12.50,
"pages_i" : 384
}
,
{
"id" : "978-1423103349",
"cat" : ["book","paperback"],
"name" : "The Sea of Monsters",
"author" : "Rick Riordan",
"series_t" : "Percy Jackson and the Olympians",
"sequence_i" : 2,
"genre_s" : "fantasy",
"inStock" : true,
"price" : 6.49,
"pages_i" : 304
}, 
...
]

While @fiskfisk answer is still a valid JSON, it is not easy to be serializable from a data structure. This one is.

Knockabout

Answer 3

elachell is correct that the array format will work if you are just adding documents with the default settings. Unfortunately, that won't work if, for instance, you need to add a custom boost to some of the documents or change the overwrite setting. You then have to use the full object structure with an "add" key for each of them, which as they pointed out, makes this frustratingly annoying to try to serialize from most languages which don't allow the same key more than once in an object:

{
"commit": {},
"add": {
    "doc": {
        "id": "321321",
        "name": "barfoo"
    },
    "boost": 2.0
},
"add": {
    "doc": {
        "id": "123123",
        "name": "Foobar"        
    },
    "boost": 1.5,
    "overwrite": false
  }

}

Babylonia

Answer 4

Update for SOLR 8.8 (and maybe lower).

The following JSON works for /update/json:

{
  'add': [
    {'id': '123', 'field1': 'foo'},
    {'id': '124', 'field1': 'foo'}
  ],
  'delete': ['111', '106']
}

Incase

Answer 5

Another option if you are on Solr 4.10 or later is to use a custom JSON structure and tell Solr how to index it (not sure how to add boosts with this method either, but it's a nice option if you already have a data struct in JSON and don't want to convert it over to Solr's format). Here's the Solr documentation on this option:

https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-TransformingandIndexingCustomJSON

Babylonia