LTR original score feature

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

LTR original score feature

Brian Yee-2
I wanted to get some opinions on using the original score feature. The original score produced by Solr is intuitively a very important feature. In my data set I'm seeing that the original score varies wildly between different queries. This makes sense since the score generated by Solr is not normalized across all queries. However, won't this mess with our training data? If this feature is 3269.4 for the top result for one query, and then 32.7 for the top result for another query, it does not mean that the first document was 10x more relevant to its query than the second document. I am using a normalize param within Ranklib, but that only normalizes features between each other, not within one feature, right? How are people handling this? Am I missing something?
Reply | Threaded
Open this post in threaded view
|

Re: LTR original score feature

Michael Alcorn
What you're suggesting is that there's a "nonlinear relationship
<http://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-difference-between-linear-and-nonlinear-equations-in-regression-analysis>"
between the original score (the input variable) and some measure of
"relevance" (the output variable). Nonlinear models like decision trees
(which include LambdaMART) and neural networks (which include RankNet) can
handle these types of situations, assuming there's enough data. The
nonlinear phenomena you brought up are also probably part of the
reason why pairwise
models tend to perform better than pointwise models
<https://www.quora.com/What-are-the-differences-between-pointwise-pairwise-and-listwise-approaches-to-Learning-to-Rank>
in
learning to rank tasks.

On Fri, Jan 12, 2018 at 1:52 PM, Brian Yee <[hidden email]> wrote:

> I wanted to get some opinions on using the original score feature. The
> original score produced by Solr is intuitively a very important feature. In
> my data set I'm seeing that the original score varies wildly between
> different queries. This makes sense since the score generated by Solr is
> not normalized across all queries. However, won't this mess with our
> training data? If this feature is 3269.4 for the top result for one query,
> and then 32.7 for the top result for another query, it does not mean that
> the first document was 10x more relevant to its query than the second
> document. I am using a normalize param within Ranklib, but that only
> normalizes features between each other, not within one feature, right? How
> are people handling this? Am I missing something?
>
Reply | Threaded
Open this post in threaded view
|

Re: LTR original score feature

Alessandro Benedetti
This is actually an interesting point.
The original Solr score alone will mean nothing, the ranking position of the
document would be a more relevant feature at that stage.

When you put the original score together with the rest of features, it may
be of potential usage ( number of query terms, tf for a specific field, idf
for another field ...).
Also because some training algorithms will group the training samples by
query.

personally I start to believe it would be better to decompose the original
score into finer grain features and then rely on LTR to weight them ( as the
original score is effectively already mixing up finer grain features
following a standard formula).





-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

RE: LTR original score feature

Brian Yee-2
Thanks for the reply Alessandro. I'm starting to agree with you but I wanted to see if others agree. It seems to me that the original score feature is not useful because it is not normalized across all queries and therefore cannot be used to compare relevance in different queries.

-----Original Message-----
From: alessandro.benedetti [mailto:[hidden email]]
Sent: Wednesday, January 24, 2018 10:22 AM
To: [hidden email]
Subject: Re: LTR original score feature

This is actually an interesting point.
The original Solr score alone will mean nothing, the ranking position of the document would be a more relevant feature at that stage.

When you put the original score together with the rest of features, it may be of potential usage ( number of query terms, tf for a specific field, idf for another field ...).
Also because some training algorithms will group the training samples by query.

personally I start to believe it would be better to decompose the original score into finer grain features and then rely on LTR to weight them ( as the original score is effectively already mixing up finer grain features following a standard formula).





-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: LTR original score feature

Michael Alcorn
>It seems to me that the original score feature is not useful because it is
not normalized across all queries and therefore cannot be used to compare
relevance in different queries.

I don't agree with this statement and it's not what Alessandro was
suggesting ("When you put the original score together with the rest of
features, it may
be of potential usage."). The magnitude of the score could very well
contain useful information in certain contexts. The simplest way to
determine whether or not the score is useful is to just train and test the
model with and without the feature included and see which one performs
better.

On Thu, Jan 25, 2018 at 3:41 PM, Brian Yee <[hidden email]> wrote:

> Thanks for the reply Alessandro. I'm starting to agree with you but I
> wanted to see if others agree. It seems to me that the original score
> feature is not useful because it is not normalized across all queries and
> therefore cannot be used to compare relevance in different queries.
>
> -----Original Message-----
> From: alessandro.benedetti [mailto:[hidden email]]
> Sent: Wednesday, January 24, 2018 10:22 AM
> To: [hidden email]
> Subject: Re: LTR original score feature
>
> This is actually an interesting point.
> The original Solr score alone will mean nothing, the ranking position of
> the document would be a more relevant feature at that stage.
>
> When you put the original score together with the rest of features, it may
> be of potential usage ( number of query terms, tf for a specific field, idf
> for another field ...).
> Also because some training algorithms will group the training samples by
> query.
>
> personally I start to believe it would be better to decompose the original
> score into finer grain features and then rely on LTR to weight them ( as
> the original score is effectively already mixing up finer grain features
> following a standard formula).
>
>
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director Sease Ltd. -
> www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: LTR original score feature

Diego Ceccarelli (BLOOMBERG/ LONDON)
In reply to this post by Brian Yee-2
I think it really depends on the particular use case. Sometime the absolute score is a good feature, sometimes no.  

If you are using the default bm25, I think that increasing the number of terms in the query will increase the average doc. score in the results. So maybe I would normalize the score at least considering the number of terms in the query.

Using the rank has been proposed in academia [1] and it seems to improve quality of the results.

[1] http://hpc.isti.cnr.it/~claudio/web/archives/20150416b/index.html 

From: [hidden email] At: 01/29/18 14:19:41To:  [hidden email]
Subject: Re: LTR original score feature

>It seems to me that the original score feature is not useful because it is
not normalized across all queries and therefore cannot be used to compare
relevance in different queries.

I don't agree with this statement and it's not what Alessandro was
suggesting ("When you put the original score together with the rest of
features, it may
be of potential usage."). The magnitude of the score could very well
contain useful information in certain contexts. The simplest way to
determine whether or not the score is useful is to just train and test the
model with and without the feature included and see which one performs
better.

On Thu, Jan 25, 2018 at 3:41 PM, Brian Yee <[hidden email]> wrote:

> Thanks for the reply Alessandro. I'm starting to agree with you but I
> wanted to see if others agree. It seems to me that the original score
> feature is not useful because it is not normalized across all queries and
> therefore cannot be used to compare relevance in different queries.
>
> -----Original Message-----
> From: alessandro.benedetti [mailto:[hidden email]]
> Sent: Wednesday, January 24, 2018 10:22 AM
> To: [hidden email]
> Subject: Re: LTR original score feature
>
> This is actually an interesting point.
> The original Solr score alone will mean nothing, the ranking position of
> the document would be a more relevant feature at that stage.
>
> When you put the original score together with the rest of features, it may
> be of potential usage ( number of query terms, tf for a specific field, idf
> for another field ...).
> Also because some training algorithms will group the training samples by
> query.
>
> personally I start to believe it would be better to decompose the original
> score into finer grain features and then rely on LTR to weight them ( as
> the original score is effectively already mixing up finer grain features
> following a standard formula).
>
>
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director Sease Ltd. -
> www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>