Warming Indexes

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Warming Indexes

Siraj Haider
Hi,

We are in the process of moving from lucene 2.9 to 6.3. We are going to use SearcherManager and SearcherFactory classes for the first time and are wondering on how to warm our indexes the first time. Below are the snippets of code, where we create our IndexWriter and SearcherManager. And then we use searcherManager.acquire() to get IndexSearcher. Can somebody please guide us on how to implement search warming? Would we need to warm each IndexSearcher that we get from .acquire()?

   IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new File(_indexLocation).toPath()), writer_config);
    SearcherManager searcherManager = new SearcherManager(indexWriter, new SearcherFactory());


------
Regards
-Siraj Haider
(212) 306-0154


________________________________

This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Uwe Schindler
Hi,

How about subclassing SearcherFactory and include warming there?
https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/SearcherFactory.html

Refer especially to the last bullet point. As described there to do warming in the background without blocking in the near realtime case, use warmers on IndexWriter.setMergedSegmentWarmer() instead. This has the effect that only new index segments are warmed as needed instead of the whole index on every reopen.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Wednesday, December 14, 2016 9:15 PM
> To: [hidden email]
> Subject: Warming Indexes
>
> Hi,
>
> We are in the process of moving from lucene 2.9 to 6.3. We are going to use
> SearcherManager and SearcherFactory classes for the first time and are
> wondering on how to warm our indexes the first time. Below are the
> snippets of code, where we create our IndexWriter and SearcherManager.
> And then we use searcherManager.acquire() to get IndexSearcher. Can
> somebody please guide us on how to implement search warming? Would we
> need to warm each IndexSearcher that we get from .acquire()?
>
>    IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> File(_indexLocation).toPath()), writer_config);
>     SearcherManager searcherManager = new SearcherManager(indexWriter,
> new SearcherFactory());
>
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Siraj Haider
Thanks for the information Uwe, it was very helpful. Do you have any example code implementing IndexWriter.IndexReaderWarmer class? I am having difficulty finding any examples on internet.

------
Regards
-Siraj Haider
(212) 306-0154

-----Original Message-----
From: Uwe Schindler [mailto:[hidden email]]
Sent: Wednesday, December 14, 2016 3:34 PM
To: [hidden email]
Subject: RE: Warming Indexes

Hi,

How about subclassing SearcherFactory and include warming there?
https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/SearcherFactory.html

Refer especially to the last bullet point. As described there to do warming in the background without blocking in the near realtime case, use warmers on IndexWriter.setMergedSegmentWarmer() instead. This has the effect that only new index segments are warmed as needed instead of the whole index on every reopen.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Wednesday, December 14, 2016 9:15 PM
> To: [hidden email]
> Subject: Warming Indexes
>
> Hi,
>
> We are in the process of moving from lucene 2.9 to 6.3. We are going
> to use SearcherManager and SearcherFactory classes for the first time
> and are wondering on how to warm our indexes the first time. Below are
> the snippets of code, where we create our IndexWriter and SearcherManager.
> And then we use searcherManager.acquire() to get IndexSearcher. Can
> somebody please guide us on how to implement search warming? Would we
> need to warm each IndexSearcher that we get from .acquire()?
>
>    IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> File(_indexLocation).toPath()), writer_config);
>     SearcherManager searcherManager = new SearcherManager(indexWriter,
> new SearcherFactory());
>
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


________________________________

This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Uwe Schindler
Hi,

Just implement this interface and override the sole abstract method. The argument is a LeafReader that you can quickly wrap with an IndexSearcher (LeafReader is a subclass of IndexReader, so it's standard pattern) and execute one or multiple queries on it. This depends on your use case. Do whatever you want with the to-be warmed LeafReader.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 4:10 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Thanks for the information Uwe, it was very helpful. Do you have any
> example code implementing IndexWriter.IndexReaderWarmer class? I am
> having difficulty finding any examples on internet.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:[hidden email]]
> Sent: Wednesday, December 14, 2016 3:34 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Hi,
>
> How about subclassing SearcherFactory and include warming there?
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sear
> cherFactory.html
>
> Refer especially to the last bullet point. As described there to do warming in
> the background without blocking in the near realtime case, use warmers on
> IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> only new index segments are warmed as needed instead of the whole index
> on every reopen.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:[hidden email]]
> > Sent: Wednesday, December 14, 2016 9:15 PM
> > To: [hidden email]
> > Subject: Warming Indexes
> >
> > Hi,
> >
> > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > to use SearcherManager and SearcherFactory classes for the first time
> > and are wondering on how to warm our indexes the first time. Below are
> > the snippets of code, where we create our IndexWriter and
> SearcherManager.
> > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > somebody please guide us on how to implement search warming? Would
> we
> > need to warm each IndexSearcher that we get from .acquire()?
> >
> >    IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > File(_indexLocation).toPath()), writer_config);
> >     SearcherManager searcherManager = new
> SearcherManager(indexWriter,
> > new SearcherFactory());
> >
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended only
> > for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Siraj Haider
Thanks Uwe, I implemented the interface and am printing some lines to see if the warmup is happening, but I never see those prints in the log after some documents are indexed. Another question is how to warmup the index when I open the index first time, i.e. when the application starts. Should I simply run warmup queries on the acquired IndexSearcher from SearcherManager or is there a better way to accomplish that.

------
Regards
-Siraj Haider
(212) 306-0154

-----Original Message-----
From: Uwe Schindler [mailto:[hidden email]]
Sent: Monday, December 19, 2016 1:29 PM
To: [hidden email]
Subject: RE: Warming Indexes

Hi,

Just implement this interface and override the sole abstract method. The argument is a LeafReader that you can quickly wrap with an IndexSearcher (LeafReader is a subclass of IndexReader, so it's standard pattern) and execute one or multiple queries on it. This depends on your use case. Do whatever you want with the to-be warmed LeafReader.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 4:10 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Thanks for the information Uwe, it was very helpful. Do you have any
> example code implementing IndexWriter.IndexReaderWarmer class? I am
> having difficulty finding any examples on internet.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:[hidden email]]
> Sent: Wednesday, December 14, 2016 3:34 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Hi,
>
> How about subclassing SearcherFactory and include warming there?
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> r
> cherFactory.html
>
> Refer especially to the last bullet point. As described there to do
> warming in the background without blocking in the near realtime case,
> use warmers on
> IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> only new index segments are warmed as needed instead of the whole
> index on every reopen.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:[hidden email]]
> > Sent: Wednesday, December 14, 2016 9:15 PM
> > To: [hidden email]
> > Subject: Warming Indexes
> >
> > Hi,
> >
> > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > to use SearcherManager and SearcherFactory classes for the first
> > time and are wondering on how to warm our indexes the first time.
> > Below are the snippets of code, where we create our IndexWriter and
> SearcherManager.
> > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > somebody please guide us on how to implement search warming? Would
> we
> > need to warm each IndexSearcher that we get from .acquire()?
> >
> >    IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > File(_indexLocation).toPath()), writer_config);
> >     SearcherManager searcherManager = new
> SearcherManager(indexWriter,
> > new SearcherFactory());
> >
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended
> > only for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the
> > return of
> any electronic mail or attachments. Thank You.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


________________________________

This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Uwe Schindler
Hi,

The warmup only happens if you reopen the searcher using the NRT APIs or searcher manager. If you just index with a single IndexWriter that has no open NRT readers, nothing will happen.

To warmup when the application starts, I'd suggest to use the first IndexSearcher. The warmer in IndexWriter is just there to run queries on new index segments that get live through SearcherManager or other NRT APIs.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Wednesday, December 21, 2016 3:58 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Thanks Uwe, I implemented the interface and am printing some lines to see if
> the warmup is happening, but I never see those prints in the log after some
> documents are indexed. Another question is how to warmup the index when
> I open the index first time, i.e. when the application starts. Should I simply
> run warmup queries on the acquired IndexSearcher from SearcherManager
> or is there a better way to accomplish that.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 1:29 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Hi,
>
> Just implement this interface and override the sole abstract method. The
> argument is a LeafReader that you can quickly wrap with an IndexSearcher
> (LeafReader is a subclass of IndexReader, so it's standard pattern) and
> execute one or multiple queries on it. This depends on your use case. Do
> whatever you want with the to-be warmed LeafReader.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:[hidden email]]
> > Sent: Monday, December 19, 2016 4:10 PM
> > To: [hidden email]
> > Subject: RE: Warming Indexes
> >
> > Thanks for the information Uwe, it was very helpful. Do you have any
> > example code implementing IndexWriter.IndexReaderWarmer class? I am
> > having difficulty finding any examples on internet.
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:[hidden email]]
> > Sent: Wednesday, December 14, 2016 3:34 PM
> > To: [hidden email]
> > Subject: RE: Warming Indexes
> >
> > Hi,
> >
> > How about subclassing SearcherFactory and include warming there?
> >
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> > r
> > cherFactory.html
> >
> > Refer especially to the last bullet point. As described there to do
> > warming in the background without blocking in the near realtime case,
> > use warmers on
> > IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> > only new index segments are warmed as needed instead of the whole
> > index on every reopen.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > http://www.thetaphi.de
> > eMail: [hidden email]
> >
> > > -----Original Message-----
> > > From: Siraj Haider [mailto:[hidden email]]
> > > Sent: Wednesday, December 14, 2016 9:15 PM
> > > To: [hidden email]
> > > Subject: Warming Indexes
> > >
> > > Hi,
> > >
> > > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > > to use SearcherManager and SearcherFactory classes for the first
> > > time and are wondering on how to warm our indexes the first time.
> > > Below are the snippets of code, where we create our IndexWriter and
> > SearcherManager.
> > > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > > somebody please guide us on how to implement search warming? Would
> > we
> > > need to warm each IndexSearcher that we get from .acquire()?
> > >
> > >    IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > > File(_indexLocation).toPath()), writer_config);
> > >     SearcherManager searcherManager = new
> > SearcherManager(indexWriter,
> > > new SearcherFactory());
> > >
> > >
> > > ------
> > > Regards
> > > -Siraj Haider
> > > (212) 306-0154
> > >
> > >
> > > ________________________________
> > >
> > > This electronic mail message and any attachments may contain
> > > information which is privileged, sensitive and/or otherwise exempt
> > > from disclosure under applicable law. The information is intended
> > > only for the use of the individual or entity named as the addressee above.
> > > If you are not the intended recipient, you are hereby notified that
> > > any disclosure, copying, distribution (electronic or otherwise) or
> > > forwarding of, or the taking of any action in reliance on, the
> > > contents of this transmission is strictly prohibited. If you have
> > > received this electronic transmission in error, please notify us by
> > > telephone, facsimile, or e-mail as noted above to arrange for the
> > > return of
> > any electronic mail or attachments. Thank You.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended only
> > for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Warming Indexes

Siraj Haider
Hi Uwe,
Below is the code that shows how we are opening writer and searchermanager:

    LimitTokenCountAnalyzer limit_analyzer = new LimitTokenCountAnalyzer(analyzer, 100000, true);
    IndexWriterConfig writer_config = new IndexWriterConfig(limit_analyzer);
    writer_config.setRAMBufferSizeMB(320);
    writer_config.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
    JobDivaIndexWarmer indexWarmer = new JobDivaIndexWarmer();
    writer_config.setMergedSegmentWarmer(indexWarmer);
    if (_useCompoundIndexFormat==1) writer_config.setUseCompoundFile(true);
    else writer_config.setUseCompoundFile(false);
    indexWriter = new IndexWriter(fsDir, writer_config);
    searcherManager = new SearcherManager(indexWriter, new SearcherFactory());

And then we keep that indexwriter alive and after each batch of document we do the following:

        indexWriter.commit();
        searcherManager.maybeRefresh();

Should these sequence of events cause the MergedSegmentWarmer to get called?
------
Regards
-Siraj Haider
(212) 306-0154

-----Original Message-----
From: Uwe Schindler [mailto:[hidden email]]
Sent: Wednesday, December 21, 2016 11:37 AM
To: [hidden email]
Subject: RE: Warming Indexes

Hi,

The warmup only happens if you reopen the searcher using the NRT APIs or searcher manager. If you just index with a single IndexWriter that has no open NRT readers, nothing will happen.

To warmup when the application starts, I'd suggest to use the first IndexSearcher. The warmer in IndexWriter is just there to run queries on new index segments that get live through SearcherManager or other NRT APIs.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Siraj Haider [mailto:[hidden email]]
> Sent: Wednesday, December 21, 2016 3:58 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Thanks Uwe, I implemented the interface and am printing some lines to
> see if the warmup is happening, but I never see those prints in the
> log after some documents are indexed. Another question is how to
> warmup the index when I open the index first time, i.e. when the
> application starts. Should I simply run warmup queries on the acquired
> IndexSearcher from SearcherManager or is there a better way to accomplish that.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 1:29 PM
> To: [hidden email]
> Subject: RE: Warming Indexes
>
> Hi,
>
> Just implement this interface and override the sole abstract method.
> The argument is a LeafReader that you can quickly wrap with an
> IndexSearcher (LeafReader is a subclass of IndexReader, so it's
> standard pattern) and execute one or multiple queries on it. This
> depends on your use case. Do whatever you want with the to-be warmed LeafReader.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:[hidden email]]
> > Sent: Monday, December 19, 2016 4:10 PM
> > To: [hidden email]
> > Subject: RE: Warming Indexes
> >
> > Thanks for the information Uwe, it was very helpful. Do you have any
> > example code implementing IndexWriter.IndexReaderWarmer class? I am
> > having difficulty finding any examples on internet.
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:[hidden email]]
> > Sent: Wednesday, December 14, 2016 3:34 PM
> > To: [hidden email]
> > Subject: RE: Warming Indexes
> >
> > Hi,
> >
> > How about subclassing SearcherFactory and include warming there?
> >
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> > r
> > cherFactory.html
> >
> > Refer especially to the last bullet point. As described there to do
> > warming in the background without blocking in the near realtime
> > case, use warmers on
> > IndexWriter.setMergedSegmentWarmer() instead. This has the effect
> > that only new index segments are warmed as needed instead of the
> > whole index on every reopen.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > http://www.thetaphi.de
> > eMail: [hidden email]
> >
> > > -----Original Message-----
> > > From: Siraj Haider [mailto:[hidden email]]
> > > Sent: Wednesday, December 14, 2016 9:15 PM
> > > To: [hidden email]
> > > Subject: Warming Indexes
> > >
> > > Hi,
> > >
> > > We are in the process of moving from lucene 2.9 to 6.3. We are
> > > going to use SearcherManager and SearcherFactory classes for the
> > > first time and are wondering on how to warm our indexes the first time.
> > > Below are the snippets of code, where we create our IndexWriter
> > > and
> > SearcherManager.
> > > And then we use searcherManager.acquire() to get IndexSearcher.
> > > Can somebody please guide us on how to implement search warming?
> > > Would
> > we
> > > need to warm each IndexSearcher that we get from .acquire()?
> > >
> > >    IndexWriter indexWriter = new IndexWriter(new
> > > NIOFSDirectory(new File(_indexLocation).toPath()), writer_config);
> > >     SearcherManager searcherManager = new
> > SearcherManager(indexWriter,
> > > new SearcherFactory());
> > >
> > >
> > > ------
> > > Regards
> > > -Siraj Haider
> > > (212) 306-0154
> > >
> > >
> > > ________________________________
> > >
> > > This electronic mail message and any attachments may contain
> > > information which is privileged, sensitive and/or otherwise exempt
> > > from disclosure under applicable law. The information is intended
> > > only for the use of the individual or entity named as the addressee above.
> > > If you are not the intended recipient, you are hereby notified
> > > that any disclosure, copying, distribution (electronic or
> > > otherwise) or forwarding of, or the taking of any action in
> > > reliance on, the contents of this transmission is strictly
> > > prohibited. If you have received this electronic transmission in
> > > error, please notify us by telephone, facsimile, or e-mail as
> > > noted above to arrange for the return of
> > any electronic mail or attachments. Thank You.
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended
> > only for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the
> > return of
> any electronic mail or attachments. Thank You.
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


________________________________

This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]