Index availability during merge

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Index availability during merge

David Pratt
Hi. Merging indexes requires that the indexes be closed for the
operation occur. I am interested in setting up a cron to merge indexes
that are in use to generate a fresh consolidated index at specific time
intervals. I don't want the smaller indexes to be taken out of service
while this occurs. Can someone suggest a strategy that would not result
in the loss of availability during merges. Does snapshooter fit into
this scenario, can a safe copy be made while index is running, etc? Many
thanks.

Regards,
David
Reply | Threaded
Open this post in threaded view
|

Re: Index availability during merge

Otis Gospodnetic-2
David,

Well, presumably the merging would be done on the master, while the indices on your search slaves would still happily be serving queries.  Thus, you really just need to coordinate your index merging app and the app that sends documents to your Solr master for indexing.  Since no new documents will be added and there will be no updates whle your merger app is running (and no commits and optimize calls), there will be no new snapshooter calls.  Communication between the apps could be as simple as FS-based file (e.g. /foo/bar/i.am.merging.now-dont.touch.the.index.lock)

Otis

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: David Pratt <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, February 27, 2008 9:16:08 AM
> Subject: Index availability during merge
>
> Hi. Merging indexes requires that the indexes be closed for the
> operation occur. I am interested in setting up a cron to merge indexes
> that are in use to generate a fresh consolidated index at specific time
> intervals. I don't want the smaller indexes to be taken out of service
> while this occurs. Can someone suggest a strategy that would not result
> in the loss of availability during merges. Does snapshooter fit into
> this scenario, can a safe copy be made while index is running, etc? Many
> thanks.
>
> Regards,
> David
>


Reply | Threaded
Open this post in threaded view
|

Re: Index availability during merge

David Pratt
Hi Otis. Many thanks for your reply. My inclination is to create
separate slave and to merge from it. I was thinking to give it a
different cron cycle (than any other slave) for applying snapshots. So
I'd try and determine a reasonable amount of time for a merge and then
resuming snapinstaller. Can anyone see a problem with this scenario?
Many thanks.

Regards,
David

Otis Gospodnetic wrote:

> David,
>
> Well, presumably the merging would be done on the master, while the indices on your search slaves would still happily be serving queries.  Thus, you really just need to coordinate your index merging app and the app that sends documents to your Solr master for indexing.  Since no new documents will be added and there will be no updates whle your merger app is running (and no commits and optimize calls), there will be no new snapshooter calls.  Communication between the apps could be as simple as FS-based file (e.g. /foo/bar/i.am.merging.now-dont.touch.the.index.lock)
>
> Otis
>
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
> ----- Original Message ----
>> From: David Pratt <[hidden email]>
>> To: [hidden email]
>> Sent: Wednesday, February 27, 2008 9:16:08 AM
>> Subject: Index availability during merge
>>
>> Hi. Merging indexes requires that the indexes be closed for the
>> operation occur. I am interested in setting up a cron to merge indexes
>> that are in use to generate a fresh consolidated index at specific time
>> intervals. I don't want the smaller indexes to be taken out of service
>> while this occurs. Can someone suggest a strategy that would not result
>> in the loss of availability during merges. Does snapshooter fit into
>> this scenario, can a safe copy be made while index is running, etc? Many
>> thanks.
>>
>> Regards,
>> David
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Index availability during merge

Otis Gospodnetic-2
In reply to this post by David Pratt
To be honest, I don't follow, David.  "Merging" and "Slave" in the same sentence sounds suspicious.  Master is where you want to do index merging, as I described in my original reply.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: David Pratt <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, February 27, 2008 5:44:33 PM
> Subject: Re: Index availability during merge
>
> Hi Otis. Many thanks for your reply. My inclination is to create
> separate slave and to merge from it. I was thinking to give it a
> different cron cycle (than any other slave) for applying snapshots. So
> I'd try and determine a reasonable amount of time for a merge and then
> resuming snapinstaller. Can anyone see a problem with this scenario?
> Many thanks.
>
> Regards,
> David
>
> Otis Gospodnetic wrote:
> > David,
> >
> > Well, presumably the merging would be done on the master, while the indices on
> your search slaves would still happily be serving queries.  Thus, you really
> just need to coordinate your index merging app and the app that sends documents
> to your Solr master for indexing.  Since no new documents will be added and
> there will be no updates whle your merger app is running (and no commits and
> optimize calls), there will be no new snapshooter calls.  Communication between
> the apps could be as simple as FS-based file (e.g.
> /foo/bar/i.am.merging.now-dont.touch.the.index.lock)
> >
> > Otis
> >
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> > ----- Original Message ----
> >> From: David Pratt
> >> To: [hidden email]
> >> Sent: Wednesday, February 27, 2008 9:16:08 AM
> >> Subject: Index availability during merge
> >>
> >> Hi. Merging indexes requires that the indexes be closed for the
> >> operation occur. I am interested in setting up a cron to merge indexes
> >> that are in use to generate a fresh consolidated index at specific time
> >> intervals. I don't want the smaller indexes to be taken out of service
> >> while this occurs. Can someone suggest a strategy that would not result
> >> in the loss of availability during merges. Does snapshooter fit into
> >> this scenario, can a safe copy be made while index is running, etc? Many
> >> thanks.
> >>
> >> Regards,
> >> David
> >>
> >
> >
>