"dismax" parameter "bq" filters instead of boosting

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

"dismax" parameter "bq" filters instead of boosting

Nicolas Franck
I noticed a change in the behaviour of the regular "dismax" parser.
At least in version 7.4:

when you add "bq", it filters the results (like "fq" does), instead of boosting the matches.


e.g.

defType=dismax
bq=format:periodical^30

gives only records with format "periodical".
removing the parameter "bq" returns all records

It does work when defType is set to "edismax".

Any idea?
Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Nicolas Franck
any update on this?

> On 5 Mar 2019, at 09:06, Nicolas Franck <[hidden email]> wrote:
>
> I noticed a change in the behaviour of the regular "dismax" parser.
> At least in version 7.4:
>
> when you add "bq", it filters the results (like "fq" does), instead of boosting the matches.
>
>
> e.g.
>
> defType=dismax
> bq=format:periodical^30
>
> gives only records with format "periodical".
> removing the parameter "bq" returns all records
>
> It does work when defType is set to "edismax".
>
> Any idea?

Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Alexandre Rafalovitch
That's a bit "fast" to expect somebody to reproduce this from
information given. Or even in general to check the mailing list, given
that we are not paid support :-)

Could you please
1) Download the latest 8.0 distribution
2) Do one of the basic examples
3) Give the search query that shows before/after BQ
4) Bonus points: test the same thing in 7.4 again with the same example

This allows somebody to reproduce the situation exactly without
wondering if it was something already fixed after 7.4, or an issue in
your example or something else.

Regards,
   Alex.

On Tue, 16 Apr 2019 at 13:17, Nicolas Franck <[hidden email]> wrote:

>
> any update on this?
>
> > On 5 Mar 2019, at 09:06, Nicolas Franck <[hidden email]> wrote:
> >
> > I noticed a change in the behaviour of the regular "dismax" parser.
> > At least in version 7.4:
> >
> > when you add "bq", it filters the results (like "fq" does), instead of boosting the matches.
> >
> >
> > e.g.
> >
> > defType=dismax
> > bq=format:periodical^30
> >
> > gives only records with format "periodical".
> > removing the parameter "bq" returns all records
> >
> > It does work when defType is set to "edismax".
> >
> > Any idea?
>
Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Nicolas Franck
I agree, but I thought my thread was lost in the long list of issues.

I prepared a simple case for solr 8.0:

  basic_dismax_set/config:

     schema.xml and solrconfig.xml

  basic_dismax_set/data:

     records_pp.json

 Total 6 records:


 5 records match format:book


and 1 format:film


But when I try this (defType is dismax) ..:


the result list is filtered on format:book (total of 5 records)

This url gives the same result by the way:


while the character ^ isn't supposed to work in fq, right?

The same result in both Solr 7.4.0 and Solr 8.0

Thanks in advance


basic_dismax_set.zip (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Alexandre Rafalovitch
If you set q.op=OR (and not as 'AND' you defined in your config), you
will see the difference between your last two queries. The second last
one will show 6 items and the last one still 5.

As is, with your custom config, booster query is added as one more
clause in the search. q.op=ALL forces it to be a compulsory clause,
rather than an optional (boosting one).

FQ is always a forced compulsory clause. Maybe it accepts boosts, but
all scores are ignored anyway (it is just 0 for fail and anything else
for pass).

Adding 'debug=all' into the query parameters (or defaults) would help
you see that for yourself.

But it does seem (in 7.2.1 I have here) that edismax seems to wrap
both query parts in individual brackets. Maybe there was a bug that
was fixed in eDismax only. No ideas there, except that most of the
effort goes into eDismax these days rather than dismax.

Regards,
   Alex
P.s. My suggestion was actually to give the queries against STOCK
examples. That would have made all these parameters explicit and more
obvious. And perhaps would have allowed you to discover the minimum
parameter set causing the issue without all those other qf and pf in
the game.

On Tue, 16 Apr 2019 at 16:13, Nicolas Franck <[hidden email]> wrote:

>
> I agree, but I thought my thread was lost in the long list of issues.
>
> I prepared a simple case for solr 8.0:
>
>   basic_dismax_set/config:
>
>      schema.xml and solrconfig.xml
>
>   basic_dismax_set/data:
>
>      records_pp.json
>
>  Total 6 records:
>
> http://localhost:8983/solr/test/select?echoParams=all
>
>  5 records match format:book
>
> http://localhost:8983/solr/test/select?echoParams=all&q=format:book&defType=lucene
>
> and 1 format:film
>
> http://localhost:8983/solr/test/select?echoParams=all&q=format:film&defType=lucene
>
> But when I try this (defType is dismax) ..:
>
> http://localhost:8983/solr/test/select?echoParams=all&bq=format:book^2
>
> the result list is filtered on format:book (total of 5 records)
>
> This url gives the same result by the way:
>
> http://localhost:8983/solr/test/select?echoParams=all&fq=format:book^2
>
> while the character ^ isn't supposed to work in fq, right?
>
> The same result in both Solr 7.4.0 and Solr 8.0
>
> Thanks in advance
>
Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Nicolas Franck
Ok, thanks for your investigation ;-) That was quick.

So you consider this as a bug, as it was fixed for edismax parser?

I thought the parameter q.op only applied to the terms in de main
query (parameter "q"), making ..

  jakarta apache

to be interpreted as

  +jakarta +apache

when q.op = AND

The documentation of bq at least describes it as an "optional" query that only
influences the score, not the result list.


> On 16 Apr 2019, at 23:59, Alexandre Rafalovitch <[hidden email]> wrote:
>
> If you set q.op=OR (and not as 'AND' you defined in your config), you
> will see the difference between your last two queries. The second last
> one will show 6 items and the last one still 5.
>
> As is, with your custom config, booster query is added as one more
> clause in the search. q.op=ALL forces it to be a compulsory clause,
> rather than an optional (boosting one).
>
> FQ is always a forced compulsory clause. Maybe it accepts boosts, but
> all scores are ignored anyway (it is just 0 for fail and anything else
> for pass).
>
> Adding 'debug=all' into the query parameters (or defaults) would help
> you see that for yourself.
>
> But it does seem (in 7.2.1 I have here) that edismax seems to wrap
> both query parts in individual brackets. Maybe there was a bug that
> was fixed in eDismax only. No ideas there, except that most of the
> effort goes into eDismax these days rather than dismax.
>
> Regards,
>   Alex
> P.s. My suggestion was actually to give the queries against STOCK
> examples. That would have made all these parameters explicit and more
> obvious. And perhaps would have allowed you to discover the minimum
> parameter set causing the issue without all those other qf and pf in
> the game.
>
> On Tue, 16 Apr 2019 at 16:13, Nicolas Franck <[hidden email]> wrote:
>>
>> I agree, but I thought my thread was lost in the long list of issues.
>>
>> I prepared a simple case for solr 8.0:
>>
>>  basic_dismax_set/config:
>>
>>     schema.xml and solrconfig.xml
>>
>>  basic_dismax_set/data:
>>
>>     records_pp.json
>>
>> Total 6 records:
>>
>> http://localhost:8983/solr/test/select?echoParams=all
>>
>> 5 records match format:book
>>
>> http://localhost:8983/solr/test/select?echoParams=all&q=format:book&defType=lucene
>>
>> and 1 format:film
>>
>> http://localhost:8983/solr/test/select?echoParams=all&q=format:film&defType=lucene
>>
>> But when I try this (defType is dismax) ..:
>>
>> http://localhost:8983/solr/test/select?echoParams=all&bq=format:book^2
>>
>> the result list is filtered on format:book (total of 5 records)
>>
>> This url gives the same result by the way:
>>
>> http://localhost:8983/solr/test/select?echoParams=all&fq=format:book^2
>>
>> while the character ^ isn't supposed to work in fq, right?
>>
>> The same result in both Solr 7.4.0 and Solr 8.0
>>
>> Thanks in advance
>>

Reply | Threaded
Open this post in threaded view
|

Re: "dismax" parameter "bq" filters instead of boosting

Alexandre Rafalovitch
I am not sure whether it is a bug or not actually. I never really used
dismax. Perhaps others can comment on that.

Regards,
   Alex.

On Wed, 17 Apr 2019 at 09:59, Nicolas Franck <[hidden email]> wrote:

>
> Ok, thanks for your investigation ;-) That was quick.
>
> So you consider this as a bug, as it was fixed for edismax parser?
>
> I thought the parameter q.op only applied to the terms in de main
> query (parameter "q"), making ..
>
>   jakarta apache
>
> to be interpreted as
>
>   +jakarta +apache
>
> when q.op = AND
>
> The documentation of bq at least describes it as an "optional" query that only
> influences the score, not the result list.
>
>
> > On 16 Apr 2019, at 23:59, Alexandre Rafalovitch <[hidden email]> wrote:
> >
> > If you set q.op=OR (and not as 'AND' you defined in your config), you
> > will see the difference between your last two queries. The second last
> > one will show 6 items and the last one still 5.
> >
> > As is, with your custom config, booster query is added as one more
> > clause in the search. q.op=ALL forces it to be a compulsory clause,
> > rather than an optional (boosting one).
> >
> > FQ is always a forced compulsory clause. Maybe it accepts boosts, but
> > all scores are ignored anyway (it is just 0 for fail and anything else
> > for pass).
> >
> > Adding 'debug=all' into the query parameters (or defaults) would help
> > you see that for yourself.
> >
> > But it does seem (in 7.2.1 I have here) that edismax seems to wrap
> > both query parts in individual brackets. Maybe there was a bug that
> > was fixed in eDismax only. No ideas there, except that most of the
> > effort goes into eDismax these days rather than dismax.
> >
> > Regards,
> >   Alex
> > P.s. My suggestion was actually to give the queries against STOCK
> > examples. That would have made all these parameters explicit and more
> > obvious. And perhaps would have allowed you to discover the minimum
> > parameter set causing the issue without all those other qf and pf in
> > the game.
> >
> > On Tue, 16 Apr 2019 at 16:13, Nicolas Franck <[hidden email]> wrote:
> >>
> >> I agree, but I thought my thread was lost in the long list of issues.
> >>
> >> I prepared a simple case for solr 8.0:
> >>
> >>  basic_dismax_set/config:
> >>
> >>     schema.xml and solrconfig.xml
> >>
> >>  basic_dismax_set/data:
> >>
> >>     records_pp.json
> >>
> >> Total 6 records:
> >>
> >> http://localhost:8983/solr/test/select?echoParams=all
> >>
> >> 5 records match format:book
> >>
> >> http://localhost:8983/solr/test/select?echoParams=all&q=format:book&defType=lucene
> >>
> >> and 1 format:film
> >>
> >> http://localhost:8983/solr/test/select?echoParams=all&q=format:film&defType=lucene
> >>
> >> But when I try this (defType is dismax) ..:
> >>
> >> http://localhost:8983/solr/test/select?echoParams=all&bq=format:book^2
> >>
> >> the result list is filtered on format:book (total of 5 records)
> >>
> >> This url gives the same result by the way:
> >>
> >> http://localhost:8983/solr/test/select?echoParams=all&fq=format:book^2
> >>
> >> while the character ^ isn't supposed to work in fq, right?
> >>
> >> The same result in both Solr 7.4.0 and Solr 8.0
> >>
> >> Thanks in advance
> >>
>