Lucene not null query?

Question

How can we construct a query to search for particular field to be not null?

field_name:* is not working. I tried field_name:[a* to z*] this works fine for English, but does not cover all languages.

Any alternative suggestions?

Tum

Answer 1

This is currently not supported by Lucene. See this for a discussion.

An alternative option may be to store some pre-defined string (like nullnullnullnull) as the field value if it is null. Then you can use a negative filter to remove these records. (I don't like this much, but can't think of a better option)

Rathskeller

this answer is outdated, since Lucene introduced FieldValueQuery. Please refer to my answer below – Karlenekarlens

Answer 2

I found this to work in some cases field:([0 TO 9] [a TO z])

Deadwood

In that case I would prefer: field:([0 TO 9] [a TO z] [A TO Z]) to get also starting capital letter – Ancell

Answer 3

For anyone else arriving late to the question, the documentation includes this little snippet:

• where the field title has any non-null value:
  _exists_:title

Troup

Answer 4

Try field:[* TO *] or field:["" TO *]. But it's probably better to use a filter for this though.

Cackle

The latter worked for me with lucene version 6.6.1, es version 5.6.8. – Sheryllshetland

Answer 5

I was having the same problem but there's a property you can set on the query parser which lets you have wildcard characters at the start of a search term.

queryParser.setAllowLeadingWildcard(true);

This solved the problem for me

Please see Wildcard at the Beginning of a searchterm -Lucene

Chloromycetin

Answer 6

In the current Lucene version FieldValueQuery is outdated. Currently you can use:

new DocValuesFieldExistsQuery(name)

This works only for SortedDocValuesFields, so you have to add them while creating the document.

doc.add(new SortedDocValuesField(name, new BytesRef(value));
doc.add(new StringField(name, value, Field.Store.Yes)); //optional

The second line enables you to retrieve the value (DocValuesFields' values cannot be read as Strings).
Note that lucene allows for multiple fields with the same name as in the aforementioned example.

Balk

Answer 7

Have a look at org.apache.lucene.search.FieldValueQuery:

A Query that matches documents that have a value for a given field as reported by LeafReader#getDocsWithField(String).

please note, that it only works with DocValues, so you would need to change a way you create your document:

document.add(new StringField("field-name", "field-value", Field.Store.YES));
document.add(new SortedDocValuesField("field-name", new BytesRef("field-value")));

here I added two fields - you still need regular StringField to get value. You may choose to use BinaryDocValues#get() for older versions of Lucene, but as I see, it is removed in v7. Not sure, what is a proper way to retrieve a value now - please check this

Karlenekarlens

Answer 8

I have just started to play around with lucene (via logstash elastic search) and find that this seems to work from the kibana UI. I am not sure yet if this is some intelligence in elastic search or kibana, i just know that elastic search borrows from the lucene syntax.

application:unit-test && !exception

will return all results from my unit tests which have not had an exception

application:unit-test && exception

returns those which have a non null exception indexed. so you might try just

field

or

!field

Thilde