Searching for future or "null" dates

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Searching for future or "null" dates

Chris Maxwell
I'm having a lot of trouble getting this query syntax to work correctly. How can I search for a date, which is either in the future OR missing completely (meaning open ended)

I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work. Adding parentheses doesn't help either.

Any help would be appreciated.
Reply | Threaded
Open this post in threaded view
|

AW: Searching for future or "null" dates

Christian Kolodziej
Hi Chris,

>I'm having a lot of trouble getting this query syntax to work correctly.
>How
>can I search for a date, which is either in the future OR missing
>completely
>(meaning open ended)
>
>I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work.
>Adding parentheses doesn't help either.

Why do you use a range query for the endDate, that is not set? It's not tested yet but try -endDate:19700101T00:00:00Z (if it's a date field) or -date:0 (if you have saved timestamp). Another solution I would try is to use specific dates instead of NOW or * for the future.

Can you post the results of your trials, this might be interesting for other users, too.

Best regards,
Christian

Reply | Threaded
Open this post in threaded view
|

Re: AW: Searching for future or "null" dates

Chris Maxwell

Christian Kolodziej wrote
Hi Chris,

>I'm having a lot of trouble getting this query syntax to work correctly.
>How
>can I search for a date, which is either in the future OR missing
>completely
>(meaning open ended)
>
>I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work.
>Adding parentheses doesn't help either.

Why do you use a range query for the endDate, that is not set? It's not tested yet but try -endDate:19700101T00:00:00Z (if it's a date field) or -date:0 (if you have saved timestamp). Another solution I would try is to use specific dates instead of NOW or * for the future.

Can you post the results of your trials, this might be interesting for other users, too.

Best regards,
Christian

Sorry, I think I over simplified my example, let me clarify my reasoning. I'm searching for products that have a start date in the past up to the present and have a end date in the present/future or none at all. I had looked into searching for null dates and had come across the -date[* : *] syntax, which searches for the absence of the index value for the field specified.

Example: ( I thought this one would work)
product:foo AND startDate:[* TO NOW] AND (endDate:[NOW TO *] OR -endDate[* : *])

I've gotten the syntax below to work but it seems overkill and redundant

((product:foo AND startDate:[* TO NOW] AND endDate:[NOW TO *]) OR (product:foo AND startDate:[* TO NOW] AND (-endDate:[* TO *])))

Again I'm simplifying the actual query syntax for this example, but it seems having to restate the product and startDate again in the query is overkill. I thought their might be an easier way.

Thanks for your example. I'll try it out and post back.

Reply | Threaded
Open this post in threaded view
|

RE: Searching for future or "null" dates

Lance Norskog-2
In reply to this post by Christian Kolodziej
If the query stars with a negative clause Lucene returns nothing.

 endDate[NOW TO *] OR -endDate:[* TO *]
Might work


-----Original Message-----
From: Kolodziej Christian [mailto:[hidden email]]
Sent: Tuesday, September 16, 2008 12:01 AM
To: [hidden email]
Subject: AW: Searching for future or "null" dates

Hi Chris,

>I'm having a lot of trouble getting this query syntax to work correctly.
>How
>can I search for a date, which is either in the future OR missing
>completely (meaning open ended)
>
>I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work.
>Adding parentheses doesn't help either.

Why do you use a range query for the endDate, that is not set? It's not
tested yet but try -endDate:19700101T00:00:00Z (if it's a date field) or
-date:0 (if you have saved timestamp). Another solution I would try is to
use specific dates instead of NOW or * for the future.

Can you post the results of your trials, this might be interesting for other
users, too.

Best regards,
Christian


Reply | Threaded
Open this post in threaded view
|

RE: Searching for future or "null" dates

hossman

: If the query stars with a negative clause Lucene returns nothing.

that's not true.  If a "Query" in lucene is a BooleanQuery that only
contains negative clauses, then Lucene returns nothing (because nothing is
positively selected) ... but it if there is a mix of negative lcauses and
positive clauses doesn't matter what order the clauses are in.

in *solr* there is code that attempts to detect a query containing purely
negative clauses and it adds a "MatchAllDocs" query in that case -- but it
only works at the top level of a query.  nested queries like this...

        +fieldA:foo +(-fieldB:bar -fieldC:baz)

...won't work as you expect, because that nested query is only negative
clauses.  you can add your own MatchAllDocs query explicitly using the *:*
syntax....

        +fieldA:foo +(*:* -fieldB:bar -fieldC:baz)

:  endDate[NOW TO *] OR -endDate:[* TO *]

side note: you really, REALLY don't wnat to mix the "+/-" syntax with
"ANT/OR/NOT" .. it almost never works out the way you expect...

: >can I search for a date, which is either in the future OR missing
: >completely (meaning open ended)
: >
: >I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work.

unless you've set the default op to "AND" this should work...

        fq = endDate:[NOW TO *] (*:* -endDate:[* TO *])


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: AW: Searching for future or "null" dates

hossman
In reply to this post by Chris Maxwell

: present and have a end date in the present/future or none at all. I had
: looked into searching for null dates and had come across the -date[* : *]
: syntax, which searches for the absence of the index value for the field
: specified.

to clarify: that isn't valid syntax.  i think you are confusing two
things...

1) has a value in field "foo":     foo:[* TO *]
2) match all docs:                 *:*


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: AW: Searching for future or "null" dates

Chris Maxwell
Yes your right I mistyped when I said [* : *]. I did mean [* TO *]

hossman wrote
: present and have a end date in the present/future or none at all. I had
: looked into searching for null dates and had come across the -date[* : *]
: syntax, which searches for the absence of the index value for the field
: specified.

to clarify: that isn't valid syntax.  i think you are confusing two
things...

1) has a value in field "foo":     foo:[* TO *]
2) match all docs:                 *:*


-Hoss
Reply | Threaded
Open this post in threaded view
|

RE: Searching for future or "null" dates

Chris Maxwell
In reply to this post by hossman
Here is what I was able to get working with your help.

(productId:(102685804)) AND liveDate:[* TO NOW] AND ((endDate:[NOW TO *]) OR ((*:* -endDate:[* TO *])))

the *:* is what I was missing.

Thanks for your help.


hossman wrote
: If the query stars with a negative clause Lucene returns nothing.

that's not true.  If a "Query" in lucene is a BooleanQuery that only
contains negative clauses, then Lucene returns nothing (because nothing is
positively selected) ... but it if there is a mix of negative lcauses and
positive clauses doesn't matter what order the clauses are in.

in *solr* there is code that attempts to detect a query containing purely
negative clauses and it adds a "MatchAllDocs" query in that case -- but it
only works at the top level of a query.  nested queries like this...

        +fieldA:foo +(-fieldB:bar -fieldC:baz)

...won't work as you expect, because that nested query is only negative
clauses.  you can add your own MatchAllDocs query explicitly using the *:*
syntax....

        +fieldA:foo +(*:* -fieldB:bar -fieldC:baz)

:  endDate[NOW TO *] OR -endDate:[* TO *]

side note: you really, REALLY don't wnat to mix the "+/-" syntax with
"ANT/OR/NOT" .. it almost never works out the way you expect...

: >can I search for a date, which is either in the future OR missing
: >completely (meaning open ended)
: >
: >I've tried -endDate:[* TO *] OR endDate[NOW TO *] but that doesn't work.

unless you've set the default op to "AND" this should work...

        fq = endDate:[NOW TO *] (*:* -endDate:[* TO *])


-Hoss
Reply | Threaded
Open this post in threaded view
|

RE: Searching for future or "null" dates

hossman

: Here is what I was able to get working with your help.
:
: (productId:(102685804)) AND liveDate:[* TO NOW] AND ((endDate:[NOW TO *]) OR
: ((*:* -endDate:[* TO *])))
:
: the *:* is what I was missing.

Please, PLEASE ... do yourself a favor and stop using "AND" and "OR" ...  
food will taste better, flowers will smell fresher, and the world will be
a happy shinny place...

+productId:102685804 +liveDate:[* TO NOW] +(endDate:[NOW TO *] (*:* -endDate:[* TO *]))



-Hoss

Reply | Threaded
Open this post in threaded view
|

RE: Searching for future or "null" dates

Chris Maxwell
Thanks I'll try it out.

hossman wrote
: Here is what I was able to get working with your help.
:
: (productId:(102685804)) AND liveDate:[* TO NOW] AND ((endDate:[NOW TO *]) OR
: ((*:* -endDate:[* TO *])))
:
: the *:* is what I was missing.

Please, PLEASE ... do yourself a favor and stop using "AND" and "OR" ...  
food will taste better, flowers will smell fresher, and the world will be
a happy shinny place...

+productId:102685804 +liveDate:[* TO NOW] +(endDate:[NOW TO *] (*:* -endDate:[* TO *]))



-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: Searching for future or "null" dates

Michael Lackhoff-2
In reply to this post by hossman
On 23.09.2008 00:30 Chris Hostetter wrote:

> : Here is what I was able to get working with your help.
> :
> : (productId:(102685804)) AND liveDate:[* TO NOW] AND ((endDate:[NOW TO *]) OR
> : ((*:* -endDate:[* TO *])))
> :
> : the *:* is what I was missing.
>
> Please, PLEASE ... do yourself a favor and stop using "AND" and "OR" ...  
> food will taste better, flowers will smell fresher, and the world will be
> a happy shinny place...
>
> +productId:102685804 +liveDate:[* TO NOW] +(endDate:[NOW TO *] (*:* -endDate:[* TO *]))

I would also like to follow your advice but don't know how to do it with
defaultOperator="AND". What I am missing is the equivalent to OR:
AND: +
NOT: -
OR: ???
I didn't find anything on the Solr or Lucene query syntax pages. If
there is such an equivalent then I guess the query would become:
productId:102685804 liveDate:[* TO NOW] (endDate:[NOW TO *] <OR>(*:*
-endDate:[* TO *]))

I switched to the AND-default because that is the default in my web
frontend so I don't have to change logic. What should I do in this
situation? Go back to the OR-default?

It is not so much this example I am after but I have a syntax translater
in my application that must be able to handle similar expressions and I
want to keep it simple and still have tasty food ;-)

-Michael
Reply | Threaded
Open this post in threaded view
|

Re: Searching for future or "null" dates

hossman

: I would also like to follow your advice but don't know how to do it with
: defaultOperator="AND". What I am missing is the equivalent to OR:
: AND: +
: NOT: -
: OR: ???
: I didn't find anything on the Solr or Lucene query syntax pages. If

that's true, regretably there is no prefix operator to indicate a "SHOULD"
clause in the Lucene query langauge, so if you set the default op to "AND"
you can't then override it on individual clauses.

this is one of hte reasons i never make the default op AND.

If i'm dealing with structured queries generated progromaticly or by
"advanced" users (ie: people who know they are querying Solr) i leave the
default op alone an let them specify the full syntax with total control.  
if i'm dealing with "novice" users who just want to search for stuff i use
dismax with it's shinny sexy "mm" param (disclaimer: i wrote it) and the
default op doesn't matter (even if i want to make every term a user types
mandatory)

: I switched to the AND-default because that is the default in my web
: frontend so I don't have to change logic. What should I do in this
: situation? Go back to the OR-default?

it depends on what exactly your goals are ... you could always leave the
defualt OR in the schema but have your front end send q.op when needed --
or set q.op as a default in a handler only used by your front end while
other queries use handlers without it (and get the default behavior) ...

...or you could just ignore the ramblings of a crazy person
like me who thinks AND and OR are abominations in a non-boolean logic
system since they make sense for you and go about your day.

i'm sure your food will still taste pretty good :)




-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Searching for future or "null" dates

Michael Lackhoff-2
On 26.09.2008 06:17 Chris Hostetter wrote:

> that's true, regretably there is no prefix operator to indicate a "SHOULD"
> clause in the Lucene query langauge, so if you set the default op to "AND"
> you can't then override it on individual clauses.
>
> this is one of hte reasons i never make the default op AND.

Just for symmetry or to get rid of this restriction wouldn't it be a
good idea to add such a prefix operator?

> i'm sure your food will still taste pretty good :)

That's what my wife keeps telling me ;-)

Many thanks. I think I will leave it as is for the current application
but use OR-Default plus prefix operators for new projects.

-Michael