Lucene Search Very Slow

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene Search Very Slow

Sebastin
Hi All,

I need to fetch approximately 225 GB of Index Store records in a web page .the total time to fetch the record and display to the user takes 10 minutes.is it possible to reduce the time to milliseconds


sample code snippet:


 
        IndexReader[] readArray =
        { indexIR1, indexIR2, indexIR3, indexIR4, indexIR5, indexIR6, indexIR7,
          indexIR8, indexIR9, indexIR10, indexIR11,indexIR12, indexIR13, indexIR14,indexIR15, indexIR16, indexIR17, indexIR18, indexIR19, indexIR20, indexIR21,
          indexIR22, indexIR23, indexIR24, indexIR25,indexIR26, indexIR27, indexIR28,indexIR29, indexIR29, indexIR30, indexIR31, indexIR32, indexIR33, indexIR34,
          indexIR35, indexIR36, indexIR37, indexIR38,indexIR39, indexIR40, indexIR41,indexIR42, indexIR43, indexIR44, indexIR45, indexIR46, indexIR47, indexIR48,
          indexIR49, indexIR50, indexIR51, indexIR52,indexIR53, indexIR54, indexIR55,indexIR56, indexIR57, indexIR58, indexIR59, indexIR60, indexIR61, indexIR62,
          indexIR63, indexIR64, indexIR65, indexIR66,indexIR67, indexIR68, indexIR69,indexIR69, indexIR70, indexIR71, indexIR72, indexIR73, indexIR74, indexIR75,
          indexIR76, indexIR77, indexIR78, indexIR79,indexIR80, indexIR81, indexIR82,indexIR83, indexIR84, indexIR85, indexIR86, indexIR87, indexIR88, indexIR89,
          indexIR90, indexIR91, indexIR92, indexIR93,indexIR94, indexIR95, indexIR96,indexIR97, indexIR98, indexIR99, indexIR100, indexIR101, indexIR102, indexIR103,
          indexIR104, indexIR105, indexIR106, indexIR107,indexIR108, indexIR109, indexIR110,indexIR111, indexIR112, indexIR113, indexIR114, indexIR115, indexIR116, indexIR117,
          indexIR118, indexIR119, indexIR120, indexIR121,indexIR122, indexIR123, indexIR124,indexIR125, indexIR126, indexIR127, indexIR128, indexIR129, indexIR130, indexIR131,
          indexIR132, indexIR133, indexIR134, indexIR135,indexIR136, indexIR137, indexIR138,indexIR139, indexIR140, indexIR141, indexIR142, indexIR143, indexIR144, indexIR145,
          indexIR146, indexIR147, indexIR148, indexIR149,indexIR150, indexIR151, indexIR152, indexIR153, indexIR154};
         
   
*/
        //merged reader
      //  IndexReader mergedReader = new MultiReader(readArray);
       
      //  IndexSearcher is = new IndexSearcher(mergedReader);
      //  q = search.integerToCompressedString(q);
        BooleanQuery.setMaxClauseCount(100000);
        QueryParser parser =
            new QueryParser("indexRecords", new StandardAnalyzer());
           
           
        Query query = parser.parse(searchQuery);

Hits hits = is.search(query);

);




for every search i open 154 folders of size 2 GB .






Reply | Threaded
Open this post in threaded view
|

Re: Lucene Search Very Slow

Toke Eskildsen
On Wed, 2008-06-18 at 07:17 -0700, Sebastin wrote:
> I need to fetch approximately 225 GB of Index Store records in a web page
> .the total time to fetch the record and display to the user takes 10
> minutes.is it possible to reduce the time to milliseconds

Depends on your indexes and your queries, but it shouldn't be
impossible: Keep the searcher open between searches instead of starting
from scratch every time and do warm up searches before the real ones.

How many hits does a normal search return and what do you do with them?

You might also consider the viability of that many indexes, as there is
an upper limit to the number of available file handles. I guess it's
system-specific, but on my machine it's around 1000 and since an active
Lucene index uses more than one file handle, you might well hit the
ceiling not far from your current number of searchers.

All else failing, buy 4 * 64GB quality solid state drives, put them in
RAID 0 and be happy. Lucene still needs to warm up on those, but nowhere
near as much as conventional harddrives, and they are not as reliant on
free RAM for disk-cache.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]