Elasticsearch search query: why params._source.nested_field.size() is not working in script?

Question

There are lot of questions and answers about this but still didn't get a satisfied answers. Elasticsearch version: 6.5

Index mapping

"_doc": {
    "properties": {
      "ssid": {
        "type": "long"
      },
      "nested_field": {
        "type": "nested"
      }
    }
  }
}

Search query:

{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": "params._source.nested_field.size() > 1"
        }
      }
    }
  }
}

Also tried below query but no luck

{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "nested_field",
            "query": {
              "bool": {
                "filter": {
                  "script": {
                    "script": "params._source.nested_field.size() > 1"
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

Error

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "params._source.nested_field.size() > 1",
          "              ^---- HERE"
        ],
        "script": "params._source.nested_field.size() > 1",
        "lang": "painless"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "testing_index",
        "node": "XXXXXXXXXXXXXXXXXXXXXXX",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "params._source.nested_field.size() > 1",
            "              ^---- HERE"
          ],
          "script": "params._source.nested_field.size() > 1",
          "lang": "painless",
          "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
          }
        }
      }
    ]
  },
  "status": 500
}

params._source.nested_field return nested array while using in scripted_field but not working in validation query. Documentation are not complete about nested query with painless script.

Mouflon

In a filter context, Painless doesn't have access to the _source document: elastic.co/guide/en/elasticsearch/painless/6.5/…, mainly for performance reasons. – Fillister

What is the alternate way to validate the nested value? – Mouflon

Did you end up solving this? I have the same problem... – Moulton

@Rob: No, We had moved on latest version 7.2 recently and I didn't yet tried the same script yet. will let you know if this works in latest version. I really surprised that no one in world is interested in these kind of problems of elaticsearch. – Mouflon

@Mouflon now I have a similar problem - discuss.elastic.co/t/… Did you resolve yours one? – Analgesia

Avi & @RobL -- if you're still facing this problem check my answer below. – Trichomoniasis

Answer 1

Despite the fact that this would've been slow and against the guidance of not accessing _source in search queries, the query would have worked before v6.4.

After v6.4, thanks to an unintentional side effect of refactoring, accessing _source in the script query context is not possible.

With that being said, you can "hijack" a function_score query whose Painless context still has access to the _source:

{
  "query": {
    "function_score": {
      "query": {
        "match_all": {}
      },
      "functions": [
        {
          "script_score": {
            "script": {
              "source": "params._source.containsKey('nested_field') && params._source['nested_field'] != null && params._source.nested_field.size() > 1 ? 1 : 0"
            }
          }
        }
      ],
      "min_score": 1
    }
  }
}

---

You could use a pipeline to calculate the field size and then save it on the doc's top level.

Alternatively, you could use copy_to as outlined in my related answer.

Trichomoniasis