[lucy-user] C library - Phrase Searches

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-user] C library - Phrase Searches

serkanmulayim@gmail.com
Hi guys again :)

I have a question regarding the phrase searches and their scoring. As I see when we search for a phrase in quotation marks, e.g. "the united states", only messages that contain "the united states" are being returned. (to be more exact messages containing "the unite state" would have returned as well).

My question is how is such queries being handled in the library. Is it by looking at the consecutive term positions in documents? What is the performance impact for such queries?

Secondly how are they being scored? Is it still tf/idf? If so what is the definition of tf and of idf, for these queries?

Thanks as always,
Serkan
Reply | Threaded
Open this post in threaded view
|

RE: [lucy-user] C library - Phrase Searches

Zebrowski, Zak
You want to look to see how the documents are being analyzed... Look at the Lucy::Analysis set of perl modules (for hints as to how this is handled in c).  You can write your own analyzer to take into special cases for you.  $0.02

Zachary Zebrowski
Forensic Database Engineer / Division Mentoring Liaison
The MITRE Corporation
(W) 202-406-6346
(C) 571-232-5643
(AR) KM4ZZE

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Tuesday, November 28, 2017 12:56 PM
To: [hidden email]
Subject: [lucy-user] C library - Phrase Searches

Hi guys again :)

I have a question regarding the phrase searches and their scoring. As I see when we search for a phrase in quotation marks, e.g. "the united states", only messages that contain "the united states" are being returned. (to be more exact messages containing "the unite state" would have returned as well).

My question is how is such queries being handled in the library. Is it by looking at the consecutive term positions in documents? What is the performance impact for such queries?

Secondly how are they being scored? Is it still tf/idf? If so what is the definition of tf and of idf, for these queries?

Thanks as always,
Serkan

Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - Phrase Searches

Nick Wellnhofer
In reply to this post by serkanmulayim@gmail.com
On 28/11/2017 18:55, [hidden email] wrote:
> My question is how is such queries being handled in the library. Is it by looking at the consecutive term positions in documents?

Yes.

> What is the performance impact for such queries?

This depends on how you quantify "performance impact", but in general,
performance should be similar to an ANDQuery of all terms in the phrase.

> Secondly how are they being scored? Is it still tf/idf? If so what is the definition of tf and of idf, for these queries?

It's still tf/idf. For idf, the sum of each term's idf is used. For tf, it's
the number of phrases in a document.

For more details, see PhraseQuery.c and PhraseMatcher.c in core/Lucy/Search.

Nick
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - Phrase Searches

serkanmulayim@gmail.com
Hi Nick,

Thank you very much for your response. I think you are right in saying that it should be very similar to an ANDQuery. I looked at the PhraseMatcher and saw that there is an additional for loop (over the query terms) in order to check the positions of the terms are consecutive (to ensure that it is a phrase). I was concerned about the implication of this for loop, but thinking one more time it multiplies the complexity of an ANDQuery with a small value (which is the number of terms in the phrase query).

Thanks again,
Serkan



On 2017-12-04 07:20, Nick Wellnhofer <[hidden email]> wrote:

> On 28/11/2017 18:55, [hidden email] wrote:
> > My question is how is such queries being handled in the library. Is it by looking at the consecutive term positions in documents?
>
> Yes.
>
> > What is the performance impact for such queries?
>
> This depends on how you quantify "performance impact", but in general,
> performance should be similar to an ANDQuery of all terms in the phrase.
>
> > Secondly how are they being scored? Is it still tf/idf? If so what is the definition of tf and of idf, for these queries?
>
> It's still tf/idf. For idf, the sum of each term's idf is used. For tf, it's
> the number of phrases in a document.
>
> For more details, see PhraseQuery.c and PhraseMatcher.c in core/Lucy/Search.
>
> Nick
>