Question re snapinstaller

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Question re snapinstaller

kkrugler
Hi all,

In looking at the snapinstaller script, it seems to do the following:

1. Copy a new index directory from the master to the slave's Solr
data directory, giving it a name "index.tmp<number>".

2. Delete the current index directory ("index").

3. Rename the temp index directory to be "index".

Then the commit script will send a <commit/> POST to the
.../solr/update service, and the new index gets swapped into use.

I feel like I must be missing something, because it seems like any
request that's in the middle of being processed between step #2 and
the end of a successful swap could fail due to the index changing
underneath. Any insights here?

Thanks,

-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Reply | Threaded
Open this post in threaded view
|

Re: Question re snapinstaller

Yonik Seeley-2
On 2/13/07, Ken Krugler <[hidden email]> wrote:

> Hi all,
>
> In looking at the snapinstaller script, it seems to do the following:
>
> 1. Copy a new index directory from the master to the slave's Solr
> data directory, giving it a name "index.tmp<number>".
>
> 2. Delete the current index directory ("index").
>
> 3. Rename the temp index directory to be "index".
>
> Then the commit script will send a <commit/> POST to the
> .../solr/update service, and the new index gets swapped into use.
>
> I feel like I must be missing something, because it seems like any
> request that's in the middle of being processed between step #2 and
> the end of a successful swap could fail due to the index changing
> underneath. Any insights here?

A Lucene's IndexReader opens all index files it needs when it is instantiated.
Changes to a Lucene index via IndexWriter never change an existing
file... new files are always created.
Put the two together and it allows an IndexWriter (or anything else,
like snapinstaller) to change the index in the background without
impact to open readers.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Question re snapinstaller

kkrugler
>On 2/13/07, Ken Krugler <[hidden email]> wrote:
>>Hi all,
>>
>>In looking at the snapinstaller script, it seems to do the following:
>>
>>1. Copy a new index directory from the master to the slave's Solr
>>data directory, giving it a name "index.tmp<number>".
>>
>>2. Delete the current index directory ("index").
>>
>>3. Rename the temp index directory to be "index".
>>
>>Then the commit script will send a <commit/> POST to the
>>.../solr/update service, and the new index gets swapped into use.
>>
>>I feel like I must be missing something, because it seems like any
>>request that's in the middle of being processed between step #2 and
>>the end of a successful swap could fail due to the index changing
>>underneath. Any insights here?
>
>A Lucene's IndexReader opens all index files it needs when it is instantiated.
>Changes to a Lucene index via IndexWriter never change an existing
>file... new files are always created.
>Put the two together and it allows an IndexWriter (or anything else,
>like snapinstaller) to change the index in the background without
>impact to open readers.

Right, but from looking at the snapinstaller script it seemed as
though the deletion of the "index" directory would be deleting files
out from under the IndexReader.

Thanks,

-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Reply | Threaded
Open this post in threaded view
|

Re: Question re snapinstaller

Mike Klaas
In reply to this post by Yonik Seeley-2
On 2/13/07, Ken Krugler <[hidden email]> wrote:

> >A Lucene's IndexReader opens all index files it needs when it is instantiated.
> >Changes to a Lucene index via IndexWriter never change an existing
> >file... new files are always created.
> >Put the two together and it allows an IndexWriter (or anything else,
> >like snapinstaller) to change the index in the background without
> >impact to open readers.
>
> Right, but from looking at the snapinstaller script it seemed as
> though the deletion of the "index" directory would be deleting files
> out from under the IndexReader.

Files aren't truly deleted until no process has an open file descriptor:

http://users.actcom.co.il/~choo/lupg/tutorials/handling-files/handling-files.html#sys_file_unlink

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: Question re snapinstaller

Bill Au
Solr snapshots are created using hard links.  The file is not deleted as
long as there is 1 or more
link to it.

Bill

On 2/13/07, Mike Klaas <[hidden email]> wrote:

>
> On 2/13/07, Ken Krugler <[hidden email]> wrote:
>
> > >A Lucene's IndexReader opens all index files it needs when it is
> instantiated.
> > >Changes to a Lucene index via IndexWriter never change an existing
> > >file... new files are always created.
> > >Put the two together and it allows an IndexWriter (or anything else,
> > >like snapinstaller) to change the index in the background without
> > >impact to open readers.
> >
> > Right, but from looking at the snapinstaller script it seemed as
> > though the deletion of the "index" directory would be deleting files
> > out from under the IndexReader.
>
> Files aren't truly deleted until no process has an open file descriptor:
>
>
> http://users.actcom.co.il/~choo/lupg/tutorials/handling-files/handling-files.html#sys_file_unlink
>
> -Mike
>
Reply | Threaded
Open this post in threaded view
|

Re: Question re snapinstaller

Yonik Seeley-2
On 2/13/07, Bill Au <[hidden email]> wrote:
> Solr snapshots are created using hard links.  The file is not deleted as
> long as there is 1 or more
> link to it.

Or a process that holds it open.  It would work even if there were no
links in the filesystem because the IndexReader would still be holding
them open.

-Yonik