What's wrong with this Solr range filter query?

Question

The following filter query returns zero results (using *:* as query):

-startDate:[* TO *] OR startDate:[* TO NOW/DAY+1DAY]

But if I filter only by:

-startDate:[* TO *]

I get 3 results.

If I filter only by:

startDate:[* TO NOW/DAY+1DAY]

I get 161 reults.

Why is the combined FQ returning zero results? What I want is the filter to return any doc whose start date is null or start date is before today.

EDIT:

I'm using Solr 4.2.1.2013.03.26.08.26.55

EDIT:

Well, strange it may sound a colleague suggested putting parenthesis on the two parts like this:

(-startDate:[* TO *]) OR (startDate:[* TO NOW/DAY+1DAY])

And somehow it worked. I'm still curious why that made a difference. Hope someone can shed some light.

Thanks!

Waylin

It may be related to #635265.... – Waylin

possible duplicate of Searching for date range or null/no field in Solr – Toleration

Answer 1

Solr supports pure negative queries. They do this, essentially, by expanding the pure negative to something like:

*:* -startDate:[* TO *]

However, what you combine it in a BooleanQuery, I don't believe it applies this sort of logic anymore. A negative query does not, in lucene, fetch anything, but rather filters out matches brought in by other, positive, query terms. This differs from SQL queries, which in a sense start with an implicit *:*, or a full table of results, and allow you to pare it down.

I believe your OR is effectively being ignored, since it doesn't, strictly speaking, make sense in context. Generally, OR is just syntactic sugar, I believe (field:this OR field:that is equivalent to field:this field:that).

So, in effect your query is: startDate:[* TO NOW/DAY+1DAY] -startDate:[* TO *], which makes the results you see more obvious. When you wrap it in parentheses, then each term query is treated separately, and you gain access to solr's support of lonely negative queries.

---

A much better idea is to store a default value, if you need to search for unset/null values. *:* and by extension pure negative queries like this have to scan the entire index, and so perform very poorly. Providing a default value will improve performance, and prevent this sort of confusing situation.

Terraterrace

I think this is what is happening. A did a "debug" query and it is translated to startDate:[* TO NOW/DAY+1DAY] -startDate:[* TO *], which indeed doesn't make sense. I do think a default value is ideal, but the support for null is only temporal while the index is populated again (with non-null values). – Waylin

Answer 2

I used femtoRgon's answer and was able to construct a query that included a range and blank values.

The following includes all docs with a StartDate on or after 1/1/2014 and all docs without a StartDate.

(StartDate:[2014-01-01T00:00:00Z TO *]) OR (-StartDate:([* TO *]) AND *:*)

The magic is (-StartDate:([* TO *]) AND *:*). This will select the docs without a StartDate.

Toleration

Answer 3

Pure negative queries don't work, because they are omitting results from nothing.

Try:

: AND -startDate:[* TO *]

Heavyladen

Please notice that I'm trying to figure out a filter query (fq param) so negative queries should work OK. – Waylin

Answer 4

When you query with -startDate:[* TO *] you get documents which do not have any data for the startDate field.

When you query for startDate:[* TO NOW/DAY+1DAY] you get documents which have a value less than or equal to NOW/DAY+1DAY in the startDate field.

You could try -startDate:* OR startDate:[* TO NOW/DAY+1DAY]. The first part says documents that do not have a value and the second part says document having value less than or equal to NOW/DAY+1DAY in the startDate field.

Pilar

Seems that -startDate:* behaves as -startDate:[* TO *] does. I mean, if I set fq as -startDate:* I get 3 results. If I set fq as -startDate:* OR startDate:[* TO NOW/DAY+1DAY] I get zero again. – Waylin