how to get approximate total matching

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

how to get approximate total matching

senthil kumaran-3
Hi.
    I have more index directories (>6) all in GB,and searching my query with
single IndexSearcher  to all indexes one after another.i.e. I create one
IndexSearcher for index1 and search over that.Finally I close that and
create new IndexSearcher for index2 and so on. If i get 200 total results
then i don't go to search other index directories and i print 200 results
and exit from search.
    I need to get approximate total matching documents all over the indexes
without going to search in other indexes.
    Please suggest me a easiest way to achieve this.

P.S: To avoid more memory usage and to reduce search timeI don't want to
search my query through all indexes if i got 200 results. MultiSearcher
create OOM error,  so that I'm using single IndexSearcher.


Thanks in Advance
Senthil
Reply | Threaded
Open this post in threaded view
|

Re: how to get approximate total matching

Erick Erickson
How much memory are you allocating for your JVM? Because you're
paying a huge search time penalty by opening and closing your
searcher sequentially, it would be a good thing to not do this.
But, as you say, if you're getting OOM errors, that's a problem.

What is the total size of all your indexes? That would help folks
give you better responses and perhaps suggest other ways of
solving your problem.....

Erick

On 3/14/07, senthil kumaran <[hidden email]> wrote:

>
> Hi.
>     I have more index directories (>6) all in GB,and searching my query
> with
> single IndexSearcher  to all indexes one after another.i.e. I create one
> IndexSearcher for index1 and search over that.Finally I close that and
> create new IndexSearcher for index2 and so on. If i get 200 total results
> then i don't go to search other index directories and i print 200 results
> and exit from search.
>     I need to get approximate total matching documents all over the
> indexes
> without going to search in other indexes.
>     Please suggest me a easiest way to achieve this.
>
> P.S: To avoid more memory usage and to reduce search timeI don't want to
> search my query through all indexes if i got 200 results. MultiSearcher
> create OOM error,  so that I'm using single IndexSearcher.
>
>
> Thanks in Advance
> Senthil
>
Reply | Threaded
Open this post in threaded view
|

Re: how to get approximate total matching

Xiaocheng Luan
In reply to this post by senthil kumaran-3
If I remember correctly, I once searched over 40G of indexes using multi-searcher with 512M max heap size, how much memory did you give the JVM?
Thanks,
Xiaocheng

senthil kumaran <[hidden email]> wrote: Hi.
    I have more index directories (>6) all in GB,and searching my query with
single IndexSearcher  to all indexes one after another.i.e. I create one
IndexSearcher for index1 and search over that.Finally I close that and
create new IndexSearcher for index2 and so on. If i get 200 total results
then i don't go to search other index directories and i print 200 results
and exit from search.
    I need to get approximate total matching documents all over the indexes
without going to search in other indexes.
    Please suggest me a easiest way to achieve this.

P.S: To avoid more memory usage and to reduce search timeI don't want to
search my query through all indexes if i got 200 results. MultiSearcher
create OOM error,  so that I'm using single IndexSearcher.


Thanks in Advance
Senthil


 
---------------------------------
Don't get soaked.  Take a quick peek at the forecast
 with theYahoo! Search weather shortcut.
Reply | Threaded
Open this post in threaded view
|

Re: how to get approximate total matching

senthil kumaran-3
In reply to this post by Erick Erickson
Hi,
    Sorry for my delayed reply. I can't reproduce that OOM error with
MultiSearcher now.I'll try to reproduce that error and send you the details
that you asked.
     I've tested to search my index with  Multisearcher & separate
IndexSearcher.The search time by using Multisearcher is nearly equal  the
time by separate IndexSearcher.But memory used is more for MultiSearcher.

    My Index size is  *7.4GB *having 17M documents. Total Search Time for
MultiSearcher = 1.1Secs and Max.Memory used = 50937920. And total search
time for Separate indexSearcher = 1.4Secs and Max.Memory used = 10039408.
Wild card query is used for search.

    I need to control memory usage. Is there any possible ways to use
multisearcher with less memory usage?
    (FYI- I don't use any sort.)

Thanks & Regards
Senthil




On 3/14/07, Erick Erickson <[hidden email]> wrote:

>
> How much memory are you allocating for your JVM? Because you're
> paying a huge search time penalty by opening and closing your
> searcher sequentially, it would be a good thing to not do this.
> But, as you say, if you're getting OOM errors, that's a problem.
>
> What is the total size of all your indexes? That would help folks
> give you better responses and perhaps suggest other ways of
> solving your problem.....
>
> Erick
>
> On 3/14/07, senthil kumaran <[hidden email]> wrote:
> >
> > Hi.
> >     I have more index directories (>6) all in GB,and searching my query
> > with
> > single IndexSearcher  to all indexes one after another.i.e. I create one
> > IndexSearcher for index1 and search over that.Finally I close that and
> > create new IndexSearcher for index2 and so on. If i get 200 total
> results
> > then i don't go to search other index directories and i print 200
> results
> > and exit from search.
> >     I need to get approximate total matching documents all over the
> > indexes
> > without going to search in other indexes.
> >     Please suggest me a easiest way to achieve this.
> >
> > P.S: To avoid more memory usage and to reduce search timeI don't want to
> > search my query through all indexes if i got 200 results. MultiSearcher
> > create OOM error,  so that I'm using single IndexSearcher.
> >
> >
> > Thanks in Advance
> > Senthil
> >
>