Lucene Search Performance

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene Search Performance

Jamie Band
Hi

I am looking for a way to improve the search performance of my
application. I've followed every suggestion in the Lucene Wiki but the
search is still too slow with large indexes. I was wondering whether
there was a way to restrict a search to a specific time period and in
doing so sacrifice the quality of search results? Any other suggestions
on how to improve search performance?

Much appreciate

Jamie


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Michael Stoppelman
On Tue, Feb 26, 2008 at 10:18 AM, Jamie <[hidden email]> wrote:

> Hi
>
> I am looking for a way to improve the search performance of my
> application. I've followed every suggestion in the Lucene Wiki but the
> search is still too slow with large indexes. I was wondering whether


Did you optimize your index yet? That gave me a 2x bump.

Have you put timers around parts of your code? Maybe it's something
unrelated to lucene.
You should probably give more details on your setup if you want more helpful
advice.


>
> there was a way to restrict a search to a specific time period and in
> doing so sacrifice the quality of search results? Any other suggestions
> on how to improve search performance?
>
> Much appreciate
>
> Jamie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Jamie Band
Hi Michael

Perhaps this will help. We are using Lucene to index emails and provide
a search interface to search through those emails. Many of our customers
have 3-5 TB's or more of email data. The index size tends to be around 5
GB per million messages. On a 3 GHZ intel core duo with standard 7200 mb
drive, it takes approx. 10 seconds to search across a million emails. We
need sub second search times, especially since, as time progresses, some
of our archives are expected to reach 10-20 TB of data. In future, we
will be recommending the use of SSD drives, but I'd like to know if they
are any other strategies can pursued. One such strategy is to
automatically create a new index after the index gets to a certain size.
Then, when a search is conducted, based on date, search only those
indexes that fall between specified dates. I've run my code through the
YourKit profiler. The time appears to be consumed by Lucene itself and
not by my code.

Any other ideas?


Michael Stoppelman wrote:

> On Tue, Feb 26, 2008 at 10:18 AM, Jamie <[hidden email]> wrote:
>
>  
>> Hi
>>
>> I am looking for a way to improve the search performance of my
>> application. I've followed every suggestion in the Lucene Wiki but the
>> search is still too slow with large indexes. I was wondering whether
>>    
>
>
> Did you optimize your index yet? That gave me a 2x bump.
>
> Have you put timers around parts of your code? Maybe it's something
> unrelated to lucene.
> You should probably give more details on your setup if you want more helpful
> advice.
>
>
>  
>> there was a way to restrict a search to a specific time period and in
>> doing so sacrifice the quality of search results? Any other suggestions
>> on how to improve search performance?
>>
>> Much appreciate
>>
>> Jamie
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>    
>
>  


--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-366-8072 ext 3
UK Tel: +44-20-80991035 ext 3
Email: [hidden email]
Web: http://www.mailarchiva.com

To receive MailArchiva Enterprise Edition product announcements, send a message to: <[hidden email]>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Michael Stoppelman
So you're saying searches are taking 10 seconds on a 5G index? If so that
seems ungodly slow.
If you're on *nix, have you watched your iostat statistics? Maybe something
is hammering your hds.
Something seems amiss.

What lucene methods were pointed to as hotspots by YourKit?

-M

On Tue, Feb 26, 2008 at 2:13 PM, Jamie <[hidden email]> wrote:

> Hi Michael
>
> Perhaps this will help. We are using Lucene to index emails and provide
> a search interface to search through those emails. Many of our customers
> have 3-5 TB's or more of email data. The index size tends to be around 5
> GB per million messages. On a 3 GHZ intel core duo with standard 7200 mb
> drive, it takes approx. 10 seconds to search across a million emails. We
> need sub second search times, especially since, as time progresses, some
> of our archives are expected to reach 10-20 TB of data. In future, we
> will be recommending the use of SSD drives, but I'd like to know if they
> are any other strategies can pursued. One such strategy is to
> automatically create a new index after the index gets to a certain size.
> Then, when a search is conducted, based on date, search only those
> indexes that fall between specified dates. I've run my code through the
> YourKit profiler. The time appears to be consumed by Lucene itself and
> not by my code.
>
> Any other ideas?
>
>
> Michael Stoppelman wrote:
> > On Tue, Feb 26, 2008 at 10:18 AM, Jamie <[hidden email]> wrote:
> >
> >
> >> Hi
> >>
> >> I am looking for a way to improve the search performance of my
> >> application. I've followed every suggestion in the Lucene Wiki but the
> >> search is still too slow with large indexes. I was wondering whether
> >>
> >
> >
> > Did you optimize your index yet? That gave me a 2x bump.
> >
> > Have you put timers around parts of your code? Maybe it's something
> > unrelated to lucene.
> > You should probably give more details on your setup if you want more
> helpful
> > advice.
> >
> >
> >
> >> there was a way to restrict a search to a specific time period and in
> >> doing so sacrifice the quality of search results? Any other suggestions
> >> on how to improve search performance?
> >>
> >> Much appreciate
> >>
> >> Jamie
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >>
> >
> >
>
>
> --
> Stimulus Software - MailArchiva
> Email Archiving And Compliance
> USA Tel: +1-713-366-8072 ext 3
> UK Tel: +44-20-80991035 ext 3
> Email: [hidden email]
> Web: http://www.mailarchiva.com
>
> To receive MailArchiva Enterprise Edition product announcements, send a
> message to: <[hidden email]>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Eric Th
Hi Michael,
I guess the hotspot of lucene is
org.apache.lucene.search.IndexSearcher.search()

Hi Jamie,
What's the original text size of a million emails?
I estimate the size of an email is around 100k, is this true?
When you doing search, what kind keywords did you input, words or short
sentence?
How many results return?
Did you use filter to shrink the results size?

2008/2/27, Michael Stoppelman <[hidden email]>:

>
> So you're saying searches are taking 10 seconds on a 5G index? If so that
> seems ungodly slow.
> If you're on *nix, have you watched your iostat statistics? Maybe
> something
> is hammering your hds.
> Something seems amiss.
>
> What lucene methods were pointed to as hotspots by YourKit?
>
>
> -M
>
>
> On Tue, Feb 26, 2008 at 2:13 PM, Jamie <[hidden email]> wrote:
>
> > Hi Michael
> >
> > Perhaps this will help. We are using Lucene to index emails and provide
> > a search interface to search through those emails. Many of our customers
> > have 3-5 TB's or more of email data. The index size tends to be around 5
> > GB per million messages. On a 3 GHZ intel core duo with standard 7200 mb
> > drive, it takes approx. 10 seconds to search across a million emails. We
> > need sub second search times, especially since, as time progresses, some
> > of our archives are expected to reach 10-20 TB of data. In future, we
> > will be recommending the use of SSD drives, but I'd like to know if they
> > are any other strategies can pursued. One such strategy is to
> > automatically create a new index after the index gets to a certain size.
> > Then, when a search is conducted, based on date, search only those
> > indexes that fall between specified dates. I've run my code through the
> > YourKit profiler. The time appears to be consumed by Lucene itself and
> > not by my code.
> >
> > Any other ideas?
> >
> >
> > Michael Stoppelman wrote:
> > > On Tue, Feb 26, 2008 at 10:18 AM, Jamie <[hidden email]>
> wrote:
> > >
> > >
> > >> Hi
> > >>
> > >> I am looking for a way to improve the search performance of my
> > >> application. I've followed every suggestion in the Lucene Wiki but
> the
> > >> search is still too slow with large indexes. I was wondering whether
> > >>
> > >
> > >
> > > Did you optimize your index yet? That gave me a 2x bump.
> > >
> > > Have you put timers around parts of your code? Maybe it's something
> > > unrelated to lucene.
> > > You should probably give more details on your setup if you want more
> > helpful
> > > advice.
> > >
> > >
> > >
> > >> there was a way to restrict a search to a specific time period and in
> > >> doing so sacrifice the quality of search results? Any other
> suggestions
> > >> on how to improve search performance?
> > >>
> > >> Much appreciate
> > >>
> > >> Jamie
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: [hidden email]
> > >> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> > >>
> > >
> > >
> >
> >
> > --
> > Stimulus Software - MailArchiva
> > Email Archiving And Compliance
> > USA Tel: +1-713-366-8072 ext 3
> > UK Tel: +44-20-80991035 ext 3
> > Email: [hidden email]
> > Web: http://www.mailarchiva.com
> >
> > To receive MailArchiva Enterprise Edition product announcements, send a
> > message to: <[hidden email]>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Anshum
Hi Jamie,

Are you running concurrent searches on the index i.e. spawning multiple
threads and not handling them?
I have been having similar issues and I am planning to try out a
workaround for it using Java's Interface Executor.
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Executor.html
This might help as it would control the resource utilization and of what
I know it as, it would pool resources and threads to handle concurrency,
thereby decreasing time.
Let me know in case you come across something better.

--
Anshum



On Wed, 2008-02-27 at 07:37 +0530, h t wrote:

> Hi Michael,
> I guess the hotspot of lucene is
> org.apache.lucene.search.IndexSearcher.search()
>
> Hi Jamie,
> What's the original text size of a million emails?
> I estimate the size of an email is around 100k, is this true?
> When you doing search, what kind keywords did you input, words or short
> sentence?
> How many results return?
> Did you use filter to shrink the results size?
>
> 2008/2/27, Michael Stoppelman <[hidden email]>:
> >
> > So you're saying searches are taking 10 seconds on a 5G index? If so that
> > seems ungodly slow.
> > If you're on *nix, have you watched your iostat statistics? Maybe
> > something
> > is hammering your hds.
> > Something seems amiss.
> >
> > What lucene methods were pointed to as hotspots by YourKit?
> >
> >
> > -M
> >
> >
> > On Tue, Feb 26, 2008 at 2:13 PM, Jamie <[hidden email]> wrote:
> >
> > > Hi Michael
> > >
> > > Perhaps this will help. We are using Lucene to index emails and provide
> > > a search interface to search through those emails. Many of our customers
> > > have 3-5 TB's or more of email data. The index size tends to be around 5
> > > GB per million messages. On a 3 GHZ intel core duo with standard 7200 mb
> > > drive, it takes approx. 10 seconds to search across a million emails. We
> > > need sub second search times, especially since, as time progresses, some
> > > of our archives are expected to reach 10-20 TB of data. In future, we
> > > will be recommending the use of SSD drives, but I'd like to know if they
> > > are any other strategies can pursued. One such strategy is to
> > > automatically create a new index after the index gets to a certain size.
> > > Then, when a search is conducted, based on date, search only those
> > > indexes that fall between specified dates. I've run my code through the
> > > YourKit profiler. The time appears to be consumed by Lucene itself and
> > > not by my code.
> > >
> > > Any other ideas?
> > >
> > >
> > > Michael Stoppelman wrote:
> > > > On Tue, Feb 26, 2008 at 10:18 AM, Jamie <[hidden email]>
> > wrote:
> > > >
> > > >
> > > >> Hi
> > > >>
> > > >> I am looking for a way to improve the search performance of my
> > > >> application. I've followed every suggestion in the Lucene Wiki but
> > the
> > > >> search is still too slow with large indexes. I was wondering whether
> > > >>
> > > >
> > > >
> > > > Did you optimize your index yet? That gave me a 2x bump.
> > > >
> > > > Have you put timers around parts of your code? Maybe it's something
> > > > unrelated to lucene.
> > > > You should probably give more details on your setup if you want more
> > > helpful
> > > > advice.
> > > >
> > > >
> > > >
> > > >> there was a way to restrict a search to a specific time period and in
> > > >> doing so sacrifice the quality of search results? Any other
> > suggestions
> > > >> on how to improve search performance?
> > > >>
> > > >> Much appreciate
> > > >>
> > > >> Jamie
> > > >>
> > > >>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: [hidden email]
> > > >> For additional commands, e-mail: [hidden email]
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Stimulus Software - MailArchiva
> > > Email Archiving And Compliance
> > > USA Tel: +1-713-366-8072 ext 3
> > > UK Tel: +44-20-80991035 ext 3
> > > Email: [hidden email]
> > > Web: http://www.mailarchiva.com
> > >
> > > To receive MailArchiva Enterprise Edition product announcements, send a
> > > message to: <[hidden email]>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Jamie Band
In reply to this post by Eric Th
Hi Michael & Others

Ok. I've gathered some more statistics from a different machine for your
analysis.
(I had to switch machines because the original one was in production and
my tests were interfering).

Here are the statistics from the new machine:

Total Documents: 1.2 million
Results Returned:  900k
Store Size 238G (size of original documents)
Index Size 1.2G (lucene index size)
Index / Store Ratio 0.5%

The search query is as follows:

archivedate:[d20071229010000 TO d20080228235900]

As you can see, I am using a range query to search between specific dates.
Question: should this query be moved to a filter rather? I did not do
this as I needed to have the option to sort on date.

There are no other specific filters applied and in this example sorting
is turned off.

On this particular machine the search time varies between 2.64 seconds
and about 5 seconds.

The limitations of this machine are that it does uses a normal IDE drive
to house the index, not a SATA drive

IOStat Statistics

Linux 2.6.20-15-server 27/02/2008

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         20.25    0.00    3.23    0.34    0.00   76.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               7.12        50.67       186.41   38936841  143240688

See attached for hardware info and the CPU call tree (taken from YourKit).

I would appreciate your recommendations.

Jamie

h t wrote:
Hi Michael,
I guess the hotspot of lucene is
org.apache.lucene.search.IndexSearcher.search()

Hi Jamie,
What's the original text size of a million emails?
I estimate the size of an email is around 100k, is this true?
When you doing search, what kind keywords did you input, words or short
sentence?
How many results return?
Did you use filter to shrink the results size?

2008/2/27, Michael Stoppelman <[hidden email]>:
  So you're saying searches are taking 10 seconds on a 5G index? If so that
seems ungodly slow.
If you're on *nix, have you watched your iostat statistics? Maybe
something
is hammering your hds.
Something seems amiss.

What lucene methods were pointed to as hotspots by YourKit?




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

stats.zip (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

heybluez
I'm wondering if your date field's precision may be a little too  
much?  What I mean is that you are going all the way down to  
seconds.  Whenever you do a range query you are essentially spawning  
a BooleanQuery with a representation of that range.  Do you really  
need to be that precise?  I usually stick with YYYYMMDD for search  
date fields and it works pretty well.  So you know, I have a 13 GB  
index with 3 million records and my search time is very low.  
Definitely under 1 second.

Just a thought.....

On Feb 27, 2008, at 6:14 AM, Jamie wrote:

> Hi Michael & Others
>
> Ok. I've gathered some more statistics from a different machine for  
> your analysis.
> (I had to switch machines because the original one was in  
> production and my tests were interfering).
>
> Here are the statistics from the new machine:
>
> Total Documents: 1.2 million
> Results Returned:  900k
> Store Size 238G (size of original documents)
> Index Size 1.2G (lucene index size)
> Index / Store Ratio 0.5%
>
> The search query is as follows:
>
> archivedate:[d20071229010000 TO d20080228235900]
>
> As you can see, I am using a range query to search between specific  
> dates.
> Question: should this query be moved to a filter rather? I did not  
> do this as I needed to have the option to sort on date.
>
> There are no other specific filters applied and in this example  
> sorting is turned off.
>
> On this particular machine the search time varies between 2.64  
> seconds and about 5 seconds.
>
> The limitations of this machine are that it does uses a normal IDE  
> drive to house the index, not a SATA drive
>
> IOStat Statistics
>
> Linux 2.6.20-15-server 27/02/2008
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>         20.25    0.00    3.23    0.34    0.00   76.19
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               7.12        50.67       186.41   38936841  143240688
>
> See attached for hardware info and the CPU call tree (taken from  
> YourKit).
>
> I would appreciate your recommendations.
>
> Jamie
>
> h t wrote:
> Hi Michael,
> I guess the hotspot of lucene is
> org.apache.lucene.search.IndexSearcher.search()
>
> Hi Jamie,
> What's the original text size of a million emails?
> I estimate the size of an email is around 100k, is this true?
> When you doing search, what kind keywords did you input, words or  
> short
> sentence?
> How many results return?
> Did you use filter to shrink the results size?
>
> 2008/2/27, Michael Stoppelman <[hidden email]>:
>  So you're saying searches are taking 10 seconds on a 5G index? If  
> so that
> seems ungodly slow.
> If you're on *nix, have you watched your iostat statistics? Maybe
> something
> is hammering your hds.
> Something seems amiss.
>
> What lucene methods were pointed to as hotspots by YourKit?
>
>
>
> <stats.zip>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Eric Th
In reply to this post by Jamie Band
1. redefine the archivedate field as YYmmDD format,
2. add another field using timestamp for sort use.
3. use RangeFilter to get result and then sort by timestamp.

2008/2/27, Jamie <[hidden email]>:

>
> Hi Michael & Others
>
> Ok. I've gathered some more statistics from a different machine for your
> analysis.
> (I had to switch machines because the original one was in production and
> my tests were interfering).
>
> Here are the statistics from the new machine:
>
> Total Documents: 1.2 million
> Results Returned:  900k
> Store Size 238G (size of original documents)
> Index Size 1.2G (lucene index size)
> Index / Store Ratio 0.5%
>
> The search query is as follows:
>
> archivedate:[d20071229010000 TO d20080228235900]
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~why there is an
> extra 'd' ?~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> As you can see, I am using a range query to search between specific dates.
> Question: should this query be moved to a filter rather? I did not do
> this as I needed to have the option to sort on date.
>
> There are no other specific filters applied and in this example sorting
> is turned off.
>
> On this particular machine the search time varies between 2.64 seconds
> and about 5 seconds.
>
> The limitations of this machine are that it does uses a normal IDE drive
> to house the index, not a SATA drive
>
> IOStat Statistics
>
> Linux 2.6.20-15-server 27/02/2008
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>          20.25    0.00    3.23    0.34    0.00   76.19
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               7.12        50.67       186.41   38936841  143240688
>
> See attached for hardware info and the CPU call tree (taken from YourKit).
>
> I would appreciate your recommendations.
>
>
> Jamie
>
>
> h t wrote:
> Hi Michael,
> I guess the hotspot of lucene is
> org.apache.lucene.search.IndexSearcher.search()
>
> Hi Jamie,
> What's the original text size of a million emails?
> I estimate the size of an email is around 100k, is this true?
> When you doing search, what kind keywords did you input, words or short
> sentence?
> How many results return?
> Did you use filter to shrink the results size?
>
> 2008/2/27, Michael Stoppelman <[hidden email]>:
>   So you're saying searches are taking 10 seconds on a 5G index? If so
> that
> seems ungodly slow.
> If you're on *nix, have you watched your iostat statistics? Maybe
> something
> is hammering your hds.
> Something seems amiss.
>
> What lucene methods were pointed to as hotspots by YourKit?
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Performance

Jamie Band
Hi

Thanks for the suggestions. This would require us to change the index
and right now we literally have millions of documents stored in current
index format. I'll bear it in mind, but I am not entirely sure how I
would go about implementing the change at this point.

Much appreciate

Jamie


h t wrote:

> 1. redefine the archivedate field as YYmmDD format,
> 2. add another field using timestamp for sort use.
> 3. use RangeFilter to get result and then sort by timestamp.
>
> 2008/2/27, Jamie <[hidden email]>:
>  
>> Hi Michael & Others
>>
>> Ok. I've gathered some more statistics from a different machine for your
>> analysis.
>> (I had to switch machines because the original one was in production and
>> my tests were interfering).
>>
>> Here are the statistics from the new machine:
>>
>> Total Documents: 1.2 million
>> Results Returned:  900k
>> Store Size 238G (size of original documents)
>> Index Size 1.2G (lucene index size)
>> Index / Store Ratio 0.5%
>>
>> The search query is as follows:
>>
>> archivedate:[d20071229010000 TO d20080228235900]
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~why there is an
>> extra 'd' ?~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> As you can see, I am using a range query to search between specific dates.
>> Question: should this query be moved to a filter rather? I did not do
>> this as I needed to have the option to sort on date.
>>
>> There are no other specific filters applied and in this example sorting
>> is turned off.
>>
>> On this particular machine the search time varies between 2.64 seconds
>> and about 5 seconds.
>>
>> The limitations of this machine are that it does uses a normal IDE drive
>> to house the index, not a SATA drive
>>
>> IOStat Statistics
>>
>> Linux 2.6.20-15-server 27/02/2008
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>          20.25    0.00    3.23    0.34    0.00   76.19
>>
>> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
>> sda               7.12        50.67       186.41   38936841  143240688
>>
>> See attached for hardware info and the CPU call tree (taken from YourKit).
>>
>> I would appreciate your recommendations.
>>
>>
>> Jamie
>>
>>
>> h t wrote:
>> Hi Michael,
>> I guess the hotspot of lucene is
>> org.apache.lucene.search.IndexSearcher.search()
>>
>> Hi Jamie,
>> What's the original text size of a million emails?
>> I estimate the size of an email is around 100k, is this true?
>> When you doing search, what kind keywords did you input, words or short
>> sentence?
>> How many results return?
>> Did you use filter to shrink the results size?
>>
>> 2008/2/27, Michael Stoppelman <[hidden email]>:
>>   So you're saying searches are taking 10 seconds on a 5G index? If so
>> that
>> seems ungodly slow.
>> If you're on *nix, have you watched your iostat statistics? Maybe
>> something
>> is hammering your hds.
>> Something seems amiss.
>>
>> What lucene methods were pointed to as hotspots by YourKit?
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>    
>
>  


--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-366-8072 ext 3
UK Tel: +44-20-80991035 ext 3
Email: [hidden email]
Web: http://www.mailarchiva.com

To receive MailArchiva Enterprise Edition product announcements, send a message to: <[hidden email]>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene Search Performance

Andreas Guther
Just some comment and I understand that you cannot change your index:

What we did is to organize our index based on creation date of entries.
We limit our search to a given number of years starting from the current
year.  Organizing the index in that way allows us to take off outdated
information.  We can provide the user an option to search across older
indexes as well, if wanted.

Andreas

-----Original Message-----
From: Jamie [mailto:[hidden email]]
Sent: Wednesday, February 27, 2008 10:17 PM
To: [hidden email]
Subject: Re: Lucene Search Performance

Hi

Thanks for the suggestions. This would require us to change the index
and right now we literally have millions of documents stored in current
index format. I'll bear it in mind, but I am not entirely sure how I
would go about implementing the change at this point.

Much appreciate

Jamie


h t wrote:
> 1. redefine the archivedate field as YYmmDD format,
> 2. add another field using timestamp for sort use.
> 3. use RangeFilter to get result and then sort by timestamp.
>
> 2008/2/27, Jamie <[hidden email]>:
>  
>> Hi Michael & Others
>>
>> Ok. I've gathered some more statistics from a different machine for
your
>> analysis.
>> (I had to switch machines because the original one was in production
and

>> my tests were interfering).
>>
>> Here are the statistics from the new machine:
>>
>> Total Documents: 1.2 million
>> Results Returned:  900k
>> Store Size 238G (size of original documents)
>> Index Size 1.2G (lucene index size)
>> Index / Store Ratio 0.5%
>>
>> The search query is as follows:
>>
>> archivedate:[d20071229010000 TO d20080228235900]
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~why there
is an
>> extra 'd' ?~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> As you can see, I am using a range query to search between specific
dates.
>> Question: should this query be moved to a filter rather? I did not do
>> this as I needed to have the option to sort on date.
>>
>> There are no other specific filters applied and in this example
sorting
>> is turned off.
>>
>> On this particular machine the search time varies between 2.64
seconds
>> and about 5 seconds.
>>
>> The limitations of this machine are that it does uses a normal IDE
drive

>> to house the index, not a SATA drive
>>
>> IOStat Statistics
>>
>> Linux 2.6.20-15-server 27/02/2008
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>          20.25    0.00    3.23    0.34    0.00   76.19
>>
>> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read
Blk_wrtn
>> sda               7.12        50.67       186.41   38936841
143240688
>>
>> See attached for hardware info and the CPU call tree (taken from
YourKit).

>>
>> I would appreciate your recommendations.
>>
>>
>> Jamie
>>
>>
>> h t wrote:
>> Hi Michael,
>> I guess the hotspot of lucene is
>> org.apache.lucene.search.IndexSearcher.search()
>>
>> Hi Jamie,
>> What's the original text size of a million emails?
>> I estimate the size of an email is around 100k, is this true?
>> When you doing search, what kind keywords did you input, words or
short
>> sentence?
>> How many results return?
>> Did you use filter to shrink the results size?
>>
>> 2008/2/27, Michael Stoppelman <[hidden email]>:
>>   So you're saying searches are taking 10 seconds on a 5G index? If
so

>> that
>> seems ungodly slow.
>> If you're on *nix, have you watched your iostat statistics? Maybe
>> something
>> is hammering your hds.
>> Something seems amiss.
>>
>> What lucene methods were pointed to as hotspots by YourKit?
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>    
>
>  


--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-366-8072 ext 3
UK Tel: +44-20-80991035 ext 3
Email: [hidden email]
Web: http://www.mailarchiva.com

To receive MailArchiva Enterprise Edition product announcements, send a
message to: <[hidden email]>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]