Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Otis Gospodnetic-2
Doron - this looks super useful!
Can you give an example for the lexical affinities you mention here? ("Juru creates posting lists for lexical affinities")
Also:

"Normalized term-frequency, as in Juru.
Here, tf(freq) is normalized by the average term frequency of the document."

I've never seen this mentioned anywhere except here and once here on the ML (was it you who mentioned this?), but this sounds intuitive.  What do others think?
Otis


--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
From: Apache Wiki <[hidden email]>
To: [hidden email]
Sent: Wednesday, January 30, 2008 5:15:02 PM
Subject: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Dear
Wiki
user,

You
have
subscribed
to
a
wiki
page
or
wiki
category
on
"Lucene-java
Wiki"
for
change
notification.

The
following
page
has
been
changed
by
DoronCohen:
http://wiki.apache.org/lucene-java/TREC_2007_Million_Queries_Track_-_IBM_Haifa_Team

The
comment
on
the
change
is:
Initial
version,
some
data
still
missing...

New
page:
=
TREC
2007
Million
Queries
Track
-
IBM
Haifa
Team
=

The
[http://ciir.cs.umass.edu/research/million/ 
Million
Queries
Track]
ran
for
the
first
time
in
2007.

Quoting
from
the
track
home
page:
 
*
"The
goal
of
this
track
is
to
run
a
retrieval
task
similar
to
standard
ad-hoc
retrieval,
 
 
but
to
evaluate
large
numbers
of
queries
incompletely,
rather
than
a
small
number
more
completely.  
 
 
 
Participants
will
run
10,000
queries
and
a
random
1,000
or
so
will
be
evaluated.
The
corpus
is
 
 
the
terabyte
track's
GOV2
corpus
of
roughly
25,000,000
.gov
web
pages,
amounting
to
just
 
 
under
half
a
terabyte
of
data."

We
participated
in
this
track
with
two
search
engines
-
our
home
brewed
search
engine
[http://trec.nist.gov/pubs/trec10/papers/JuruAtTrec.pdf 
Juru].

The
official
reports
and
papers
of
the
track
should
be
available
sometimes
in
February
2008,
but
here
is
a
summary
of
the
results
and
our
experience
with
our
first
ever
Lucene
submission
to
TREC.

In
summary,
the
out-of-the-box
search
quality
was
not
so
great,
but
by
altering
how
we
use
Lucene
(that
is,
our
application)
and
with
some
modifications
to
Lucene,
we
were
able
to
improve
the
search
quality
results
and
to
score
good
in
this
competition.

The
lessons
we
learned
can
be
of
interest
to
applications
using
Lucene,
to
Lucene
itself,
and
to
researchers
submitting
to
other
TREC
tracks
(or
elsewhere).

=
Training
=
As
preparation
for
the
track
runs
we
"trained"
Lucene
on
queries
from
previous
years
tracks
-
more
exactly
on
the
150
short
TREC
queries
for
which
there
are
existing
judgments
from
previous
years,
for
the
same
GOV2
data.

We
build
an
index
-
actually
27
indexes
-
for
this
data.
For
indexing
we
used
the
Trec-Doc-Maker
that
is
now
in
Lucene's
contrib
benchmark
(or
a
slight
modification
of
it).

We
found
that
best
results
are
obtained
when
all
data
is
in
a
single
field,
and
so
we
did,
keeping
only
stems
(English,
Porter,
from
Lucene
contrib).
We
used
the
Standard-Analyzer,
with
a
modified
stoplist
that
took
into
account
that
domain
specific
stopwords.

Running
with
both
Juru
and
Lucene,
and
having
obtained
good
results
with
Juru
in
previous
years,
we
had
something
to
compare
to.
For
this,
we
made
sure
to
HTML
parse
the
documents
in
the
same
way
in
both
systems
(we
used
Juru's
HTML
parser
for
this)
and
use
the
same
stoplist
etc.

In
addition,
anchor
text
was
collect
in
a
pre-indexing
global
analysis
pass,
and
so
anchors
of
(pointing
to)
pages
where
indexed
with
the
page
they
point
to,
up
to
a
limited
size.
The
number
of
in-links
to
each
page
was
saved
in
a
stored
field
and
we
used
it
as
a
static
score
element
(boosting
documents
that
had
more
in-links).
The
way
that
anchors
text
was
extracted
and
prepared
for
indexing
will
be
described
in
the
full
report.

=
Results
=

The
initial
results
were:

 
||<rowbgcolor="#80FF80">'''Run'''||'''MAP'''||'''P@5'''||'''P@10'''||'''P@20'''||
 
||
1.
Juru  
 
 
 
 
 
 
 
 
 
 
||
0.313
||
0.592
||
0.560  
||
0.529  
||
 
||
2.
Lucene
out-of-the-box  
 
 
||
0.154
||
0.313
||
0.303  
||
0.289  
||

We
made
the
following
changes:
 
1.
Add
a
proximity
scoring
element,
basing
on
our
experience
with
"Lexical
affinities"
in
Juru.
 
 
Juru
creates
posting
lists
for
lexical
affinities.
 
 
In
Lucene
we
used
augmented
the
query
with
Span-Near-Queries.
 
1.
Phrase
expansion
-
the
query
text
was
added
to
the
query
as
a
phrase.
 
1.
Replace
the
default
similarity
by
Sweet-Spot-Similarity
for
a
better
 
 
choice
of
document
length
normalization.
Juru
is
using
 
 
[http://citeseer.ist.psu.edu/singhal96pivoted.html 
pivoted
length
normalization]
 
 
and
we
experimented
with
it,
but
found
out
that
the
simpler
and
faster
sweet-spot-simiarity
 
 
performs
better.
 
1.
Normalized
term-frequency,
as
in
Juru.
 
 
Here,
tf(freq)
is
normalized
by
the
average
term
frequency
of
the
document.

So
these
are
the
updated
results:

 
||<rowbgcolor="#80FF80">'''Run'''  
 
 
 
 
 
 
 
 
||'''MAP'''||'''P@5'''||'''P@10'''||'''P@20'''||
 
||
1.
Juru  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
||
0.313  
 
||
0.592  
 
||
0.560  
 
||
0.529  
 
||
 
||
2.
Lucene
out-of-the-box  
 
 
 
 
 
 
 
 
 
 
 
||
0.154  
 
||
0.313  
 
||
0.303  
 
||
0.289  
 
||
 
||
3.
Lucene
+
LA
+
Phrase
+
Sweet
Spot
+
tf-norm
||
0.306  
 
||
0.627  
 
||
0.589  
 
||
0.543  
 
||
 
 
The
improvement
is
dramatic.

Perhaps
even
more
important,
once
the
track
results
were
published,
we
found
out
that
these
improvement
are
consistent
and
steady,
and
so
Lucene
with
these
changes
was
ranked
high
also
by
the
two
new
measures
introduced
in
this
track
-
NEU-Map
and
E-Map
(Epsilon-Map).

With
these
new
measures
more
queries
are
evaluated
but
less
documents
are
judged
for
each
query.
The
algorithms
for
documents
selection
for
judging
(during
the
evaluation
stage
of
the
track)
were
not
our
focus
in
this
work
-
as
there
were
actually
two
goals
to
this
TREC:

 
*
the
systems
evaluation
(our
main
goal)
and
 
*
the
evaluation
itself.

The
fact
that
modified
Lucene
scored
well
in
both
the
traditional
150
queries
and
the
new
1700
evaluated
queries
with
the
new
measures
was
reassuring
for
the
"usefulness"
or
perhaps
"validity"
of
these
modifications
to
Lucene.

For
certain
these
changes
are
not
a
100%
fit
for
every
application
and
every
data,
but
these
results
are
strong,
and
so
I
believe
can
be
be
valuable
for
many
applications,
and
certainly
for
research
aspects.

=
Search
time
penalty
=

These
improvements
did
not
come
for
free.
Adding
a
phrase
to
the
query
and
adding
Span-Near-Queries
for
every
pair
of
query
words
costs
query
time.

The
search
time
of
stock
Lucene
in
our
setup
was
1.4
seconds/query.
The
modified
search
time
took
8.0
seconds/query.
This
is
a
large
slowdown!

But
it
should
be
noticed
that
in
this
work
we
did
not
focus
in
search
time,
only
in
quality.
Now
is
the
time
to
see
how
the
search
time
penalty
can
be
reduced
while
keeping
most
of
the
search
time
improvements.

=
Implementation
Details
=

 
*
Contrib
benchmark
quality
package
was
used
for
the
search
quality
measures
and
submissions.

/!\
To
be
completed...

=
More
Detailed
Results
=

/!\
To
be
added...

=
Possible
Changes
in
Lucene
=

 
*
Move
Sweet-Spot-Similarity
to
core
 
*
Make
Sweer-Spot-Similarity
the
default
similarity?
 
*
Easier
and
more
efficient
ways
to
add
proximity
scoring?
 
*
Allow
easier
implementation/extension
of
tf-normalization

/!\
To
be
completed
&
refined...




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Doron Cohen-2
Hi Otis,

On Thu, Jan 31, 2008 at 7:21 AM, Otis Gospodnetic <
[hidden email]> wrote:

> Doron - this looks super useful!
> Can you give an example for the lexical affinities you mention here?
> ("Juru creates posting lists for lexical affinities")


Sure, - simply put, denote {X} as the posting list of term X, then for a
query - A B C D - in addition to the four posting lists {A}, {B}, {C}, {D}
which are processed ignoring position info (i.e. Lucene's termDocs()) Juru
also computes combined posting lists {A,B}, {A,C}, {A,D}, {B,C}, {B,D} and
{C,D} in which a (virtual) term {X,Y} is said to exist in a document D if
the two words X and Y are found in that document within a sliding window of
size L (say 5).

(You can also require LA's in order which is useful in some scenarios.)

Juru's tokenization detects sentences and so the two words must be in the
same sentence. The term-freq of that LA-term in the doc is as usual the
number of matches in that doc satisfying this sliding window rule.

The IDF of this term is not known in advance, and so it is first estimated
based on the DF of X and Y, and this estimate is later tuned as more
documents are processed and more statistics are available.

You can see the resemblance to SpanNear queries. Note that the IDF of this
virtual term is going to be high and as such it is "focusing" the query
search on the more relevant documents.

In my Lucene implementation for this I used a window size of 7, and note
that (1) there was no sentence boundaries knowledge in my Lucene
implementation and (2) the IDF was fixed all along, estimated by the
involved terms IDF, as computed once in SpanNear query. The default
computation is their sum. This is in most cases too low an IDF, I think.
Phrase query btw behaves the same.

So in both cases (Phrase, Span) I think it would be interesting to
experiment with adaptive IDF computation that updates the IDF as more
documents are processed. When the query is made of only a single span or
only a single phrase element this is a waste of time. But when the query is
more complex (as the query we built) and you have in the query both
multi-term parts and single-term parts, or several multi-term parts, then a
more accurate IDF can improve the quality I would think. Implementation wise
the "Weight.value" would need to be updated and might raise questions about
the normalizing of other query parts, but I am not sure about this now.

Well I hope this makes sense - I will update the Wiki page with similar
info...

Also:
>
> "Normalized term-frequency, as in Juru.
> Here, tf(freq) is normalized by the average term frequency of the
> document."
>
> I've never seen this mentioned anywhere except here and once here on the
> ML (was it you who mentioned this?), but this sounds intuitive.


Yes I think I mentioned this - I think it is not our idea - Juru uses it but
it was used before in the SMART system - see "Length Normalization in
Degraded Text Collections (1995)" - http://citeseer.ist.psu.edu/100699.html,
and "New Retrieval Approaches Using SMART : TREC 4" -
http://citeseer.ist.psu.edu/144841.html.


> What do others think?
> Otis
>
Reply | Threaded
Open this post in threaded view
|

Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Grant Ingersoll-2
Hey Doron,

I see you recommend that we think about making SweetSpot the default  
similarity.  Do you have numbers showing for running that alone?  Or  
for that matter, any of the other combinations of #3 individually?

Thanks,
Grant

On Jan 31, 2008, at 4:09 AM, Doron Cohen wrote:

> Hi Otis,
>
> On Thu, Jan 31, 2008 at 7:21 AM, Otis Gospodnetic <
> [hidden email]> wrote:
>
>> Doron - this looks super useful!
>> Can you give an example for the lexical affinities you mention here?
>> ("Juru creates posting lists for lexical affinities")
>
>
> Sure, - simply put, denote {X} as the posting list of term X, then  
> for a
> query - A B C D - in addition to the four posting lists {A}, {B},  
> {C}, {D}
> which are processed ignoring position info (i.e. Lucene's  
> termDocs()) Juru
> also computes combined posting lists {A,B}, {A,C}, {A,D}, {B,C},  
> {B,D} and
> {C,D} in which a (virtual) term {X,Y} is said to exist in a document  
> D if
> the two words X and Y are found in that document within a sliding  
> window of
> size L (say 5).
>
> (You can also require LA's in order which is useful in some  
> scenarios.)
>
> Juru's tokenization detects sentences and so the two words must be  
> in the
> same sentence. The term-freq of that LA-term in the doc is as usual  
> the
> number of matches in that doc satisfying this sliding window rule.
>
> The IDF of this term is not known in advance, and so it is first  
> estimated
> based on the DF of X and Y, and this estimate is later tuned as more
> documents are processed and more statistics are available.
>
> You can see the resemblance to SpanNear queries. Note that the IDF  
> of this
> virtual term is going to be high and as such it is "focusing" the  
> query
> search on the more relevant documents.
>
> In my Lucene implementation for this I used a window size of 7, and  
> note
> that (1) there was no sentence boundaries knowledge in my Lucene
> implementation and (2) the IDF was fixed all along, estimated by the
> involved terms IDF, as computed once in SpanNear query. The default
> computation is their sum. This is in most cases too low an IDF, I  
> think.
> Phrase query btw behaves the same.
>
> So in both cases (Phrase, Span) I think it would be interesting to
> experiment with adaptive IDF computation that updates the IDF as more
> documents are processed. When the query is made of only a single  
> span or
> only a single phrase element this is a waste of time. But when the  
> query is
> more complex (as the query we built) and you have in the query both
> multi-term parts and single-term parts, or several multi-term parts,  
> then a
> more accurate IDF can improve the quality I would think.  
> Implementation wise
> the "Weight.value" would need to be updated and might raise  
> questions about
> the normalizing of other query parts, but I am not sure about this  
> now.
>
> Well I hope this makes sense - I will update the Wiki page with  
> similar
> info...
>
> Also:
>>
>> "Normalized term-frequency, as in Juru.
>> Here, tf(freq) is normalized by the average term frequency of the
>> document."
>>
>> I've never seen this mentioned anywhere except here and once here  
>> on the
>> ML (was it you who mentioned this?), but this sounds intuitive.
>
>
> Yes I think I mentioned this - I think it is not our idea - Juru  
> uses it but
> it was used before in the SMART system - see "Length Normalization in
> Degraded Text Collections (1995)" - http://citeseer.ist.psu.edu/100699.html 
> ,
> and "New Retrieval Approaches Using SMART : TREC 4" -
> http://citeseer.ist.psu.edu/144841.html.
>
>
>> What do others think?
>> Otis
>>

--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Doron Cohen-2
Hi Grant, yes I have these combinations - I just updated the wiki page with
these numbers.

I still have the index as described,allowing to try other ideas that may
come up, or if we need more tests (on GOV2 data) to take better decisions
...

Cheers, Doron

On Wed, Feb 6, 2008 at 2:15 PM, Grant Ingersoll <[hidden email]> wrote:

> Hey Doron,
>
> I see you recommend that we think about making SweetSpot the default
> similarity.  Do you have numbers showing for running that alone?  Or
> for that matter, any of the other combinations of #3 individually?
>
> Thanks,
> Grant
>
> On Jan 31, 2008, at 4:09 AM, Doron Cohen wrote:
>
> > Hi Otis,
> >
> > On Thu, Jan 31, 2008 at 7:21 AM, Otis Gospodnetic <
> > [hidden email]> wrote:
> >
> >> Doron - this looks super useful!
> >> Can you give an example for the lexical affinities you mention here?
> >> ("Juru creates posting lists for lexical affinities")
> >
> >
> > Sure, - simply put, denote {X} as the posting list of term X, then
> > for a
> > query - A B C D - in addition to the four posting lists {A}, {B},
> > {C}, {D}
> > which are processed ignoring position info (i.e. Lucene's
> > termDocs()) Juru
> > also computes combined posting lists {A,B}, {A,C}, {A,D}, {B,C},
> > {B,D} and
> > {C,D} in which a (virtual) term {X,Y} is said to exist in a document
> > D if
> > the two words X and Y are found in that document within a sliding
> > window of
> > size L (say 5).
> >
> > (You can also require LA's in order which is useful in some
> > scenarios.)
> >
> > Juru's tokenization detects sentences and so the two words must be
> > in the
> > same sentence. The term-freq of that LA-term in the doc is as usual
> > the
> > number of matches in that doc satisfying this sliding window rule.
> >
> > The IDF of this term is not known in advance, and so it is first
> > estimated
> > based on the DF of X and Y, and this estimate is later tuned as more
> > documents are processed and more statistics are available.
> >
> > You can see the resemblance to SpanNear queries. Note that the IDF
> > of this
> > virtual term is going to be high and as such it is "focusing" the
> > query
> > search on the more relevant documents.
> >
> > In my Lucene implementation for this I used a window size of 7, and
> > note
> > that (1) there was no sentence boundaries knowledge in my Lucene
> > implementation and (2) the IDF was fixed all along, estimated by the
> > involved terms IDF, as computed once in SpanNear query. The default
> > computation is their sum. This is in most cases too low an IDF, I
> > think.
> > Phrase query btw behaves the same.
> >
> > So in both cases (Phrase, Span) I think it would be interesting to
> > experiment with adaptive IDF computation that updates the IDF as more
> > documents are processed. When the query is made of only a single
> > span or
> > only a single phrase element this is a waste of time. But when the
> > query is
> > more complex (as the query we built) and you have in the query both
> > multi-term parts and single-term parts, or several multi-term parts,
> > then a
> > more accurate IDF can improve the quality I would think.
> > Implementation wise
> > the "Weight.value" would need to be updated and might raise
> > questions about
> > the normalizing of other query parts, but I am not sure about this
> > now.
> >
> > Well I hope this makes sense - I will update the Wiki page with
> > similar
> > info...
> >
> > Also:
> >>
> >> "Normalized term-frequency, as in Juru.
> >> Here, tf(freq) is normalized by the average term frequency of the
> >> document."
> >>
> >> I've never seen this mentioned anywhere except here and once here
> >> on the
> >> ML (was it you who mentioned this?), but this sounds intuitive.
> >
> >
> > Yes I think I mentioned this - I think it is not our idea - Juru
> > uses it but
> > it was used before in the SMART system - see "Length Normalization in
> > Degraded Text Collections (1995)" -
> http://citeseer.ist.psu.edu/100699.html
> > ,
> > and "New Retrieval Approaches Using SMART : TREC 4" -
> > http://citeseer.ist.psu.edu/144841.html.
> >
> >
> >> What do others think?
> >> Otis
> >>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
> http://www.lucenebootcamp.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Lucene-java Wiki] Update of "TREC 2007 Million Queries Track - IBM Haifa Team" by DoronCohen

Doron Cohen-2
In reply to this post by Doron Cohen-2
On Thu, Jan 31, 2008 at 11:09 AM, Doron Cohen <[hidden email]> wrote:

> Hi Otis,
>
> On Thu, Jan 31, 2008 at 7:21 AM, Otis Gospodnetic <
> [hidden email]> wrote:
>
> > Doron - this looks super useful!
> > Can you give an example for the lexical affinities you mention here?
> > ("Juru creates posting lists for lexical affinities")
>
>
> Sure, - simply put, denote {X} as the posting list of term X, then for a
> query - A B C D - in addition to the four posting lists {A}, {B}, {C}, {D}
> which are processed ignoring position info (i.e. Lucene's termDocs()) Juru
> also computes combined posting lists {A,B}, {A,C}, {A,D}, {B,C}, {B,D} and
> {C,D} in which a (virtual) term {X,Y} is said to exist in a document D if
> the two words X and Y are found in that document within a sliding window of
> size L (say 5).
>

The wiki page now has a more complete example.

(You can also require LA's in order which is useful in some scenarios.)
>
> Juru's tokenization detects sentences and so the two words must be in the
> same sentence. The term-freq of that LA-term in the doc is as usual the
> number of matches in that doc satisfying this sliding window rule.
>
> The IDF of this term is not known in advance, and so it is first estimated
> based on the DF of X and Y, and this estimate is later tuned as more
> documents are processed and more statistics are available.
>

This was not so accurate a description. What Juru really does is compute in
advance the first e.g. 1MB of the LA posting and use its computed IDF for
the entire posting. Experiments with more accurate adaptive computation (for
longer LA postings) showed no advantae over this simpler approach.


> You can see the resemblance to SpanNear queries. Note that the IDF of this
> virtual term is going to be high and as such it is "focusing" the query
> search on the more relevant documents.
>
> In my Lucene implementation for this I used a window size of 7, and note
> that (1) there was no sentence boundaries knowledge in my Lucene
> implementation and (2) the IDF was fixed all along, estimated by the
> involved terms IDF, as computed once in SpanNear query. The default
> computation is their sum. This is in most cases too low an IDF, I think.
> Phrase query btw behaves the same.
>
> So in both cases (Phrase, Span) I think it would be interesting to
> experiment with adaptive IDF computation that updates the IDF as more
> documents are processed. When the query is made of only a single span or
> only a single phrase element this is a waste of time. But when the query is
> more complex (as the query we built) and you have in the query both
> multi-term parts and single-term parts, or several multi-term parts, then a
> more accurate IDF can improve the quality I would think. Implementation wise
> the "Weight.value" would need to be updated and might raise questions
> about the normalizing of other query parts, but I am not sure about this
> now.
>

Well after discussing this with my colleague David Carmel who pointed out
that summing the IDFs actually makes sense because each IDF is *nearly* a
log of the nDocs/DF and so summing the nearly logs is (nearly) the log of
the multiplication (of (1+nDocs/DF)). So I don't anymore see here a problem
to fix or an immediate oportunity to explore...


>
>
> Well I hope this makes sense - I will update the Wiki page with similar
> info...
>
> Also:
> >
> > "Normalized term-frequency, as in Juru.
> > Here, tf(freq) is normalized by the average term frequency of the
> > document."
> >
> > I've never seen this mentioned anywhere except here and once here on the
> > ML (was it you who mentioned this?), but this sounds intuitive.
>
>
> Yes I think I mentioned this - I think it is not our idea - Juru uses it
> but it was used before in the SMART system - see "Length Normalization in
> Degraded Text Collections (1995)" -
> http://citeseer.ist.psu.edu/100699.html, and "New Retrieval Approaches
> Using SMART : TREC 4" - http://citeseer.ist.psu.edu/144841.html.
>
>
> > What do others think?
> > Otis
> >
>
>