detected corrupted index / performance improvement

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

detected corrupted index / performance improvement

Robert Engels
I had a recent sidebar with another user, and it got me to thinking.

Do we have any way of determining if a segment is definitely OK/VALID ?

If so, a much more efficient transactional system could be developed.

Serialize the updates to a log file. Sync the log. Update the lucene  
index WITHOUT any sync.  Log file writing/sync is VERY efficient  
since it is sequential, and a single file.

Upon open of the index, detect if index was not shutdown cleanly. If  
so, determine the last valid segment, delete the bad segments, and  
then perform the updates (from the log file) since the last valid  
segment was written.

The detection could be a VERY slow operation, but this is ok, since  
it should be rare, and then you will only pay this price on the rare  
occasion, not on every update.




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2

robert engels wrote:

> Do we have any way of determining if a segment is definitely OK/
> VALID ?

The only way I know is the CheckIndex tool, and it's rather slow (and
it's not clear that it always catches all corruption).

> If so, a much more efficient transactional system could be developed.
>
> Serialize the updates to a log file. Sync the log. Update the  
> lucene index WITHOUT any sync.  Log file writing/sync is VERY  
> efficient since it is sequential, and a single file.
>
> Upon open of the index, detect if index was not shutdown cleanly.  
> If so, determine the last valid segment, delete the bad segments,  
> and then perform the updates (from the log file) since the last  
> valid segment was written.
>
> The detection could be a VERY slow operation, but this is ok, since  
> it should be rare, and then you will only pay this price on the  
> rare occasion, not on every update.

Wouldn't you still need to sync periodically, so you can prune the
transaction log?  Else your transaction log is growing as fast as the
index?  (You've doubled disk usage).

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

DM Smith

On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:

>
> robert engels wrote:
>
>> Do we have any way of determining if a segment is definitely OK/
>> VALID ?
>
> The only way I know is the CheckIndex tool, and it's rather slow (and
> it's not clear that it always catches all corruption).

Just a thought. It seems that the discussion has revolved around  
whether a crash or similar event has left the file in an inconsistent  
state. Without looking into how it is actually done, I'm going to  
guess that the writing is done from the start of the file to its end.  
That is, no "out of order" writing.

If this is the case, how about adding a marker to the end of the file  
of a known size and pattern. If it is present then it is presumed that  
there were no errors in getting to that point.

Even with out of order writing, one could write an 'INVALID' marker at  
the beginning of the operation and then upon reaching the end of the  
writing, replace it with the valid marker.

If neither marker is found then the index is one from before the  
capability was added and nothing can be said about the validity.

-- DM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
In reply to this post by Michael McCandless-2
Yes, but this pruning could be more efficient. On a background  
thread, get current segment from segments file, call the system wide  
sync ( e.g. System.exec("fsync"), then you can purge the transaction  
logs for all segments up to that one. Since it is a background  
operation, you are not blocking the writing of new segments and tx logs.

On Feb 6, 2008, at 4:42 PM, Michael McCandless wrote:

>
> robert engels wrote:
>
>> Do we have any way of determining if a segment is definitely OK/
>> VALID ?
>
> The only way I know is the CheckIndex tool, and it's rather slow (and
> it's not clear that it always catches all corruption).
>
>> If so, a much more efficient transactional system could be developed.
>>
>> Serialize the updates to a log file. Sync the log. Update the  
>> lucene index WITHOUT any sync.  Log file writing/sync is VERY  
>> efficient since it is sequential, and a single file.
>>
>> Upon open of the index, detect if index was not shutdown cleanly.  
>> If so, determine the last valid segment, delete the bad segments,  
>> and then perform the updates (from the log file) since the last  
>> valid segment was written.
>>
>> The detection could be a VERY slow operation, but this is ok,  
>> since it should be rare, and then you will only pay this price on  
>> the rare occasion, not on every update.
>
> Wouldn't you still need to sync periodically, so you can prune the
> transaction log?  Else your transaction log is growing as fast as the
> index?  (You've doubled disk usage).
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
In reply to this post by DM Smith
That doesn't help, with lazy writing/buffering by the OS, there is no  
guarantee that if the last written block is ok, that earlier blocks  
in the file are....

The OS/drive is going to physically write them in the most efficient  
manner. Only after a sync would this hold true (which is what we are  
trying to avoid).

On Feb 6, 2008, at 5:15 PM, DM Smith wrote:

>
> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>
>>
>> robert engels wrote:
>>
>>> Do we have any way of determining if a segment is definitely OK/
>>> VALID ?
>>
>> The only way I know is the CheckIndex tool, and it's rather slow (and
>> it's not clear that it always catches all corruption).
>
> Just a thought. It seems that the discussion has revolved around  
> whether a crash or similar event has left the file in an  
> inconsistent state. Without looking into how it is actually done,  
> I'm going to guess that the writing is done from the start of the  
> file to its end. That is, no "out of order" writing.
>
> If this is the case, how about adding a marker to the end of the  
> file of a known size and pattern. If it is present then it is  
> presumed that there were no errors in getting to that point.
>
> Even with out of order writing, one could write an 'INVALID' marker  
> at the beginning of the operation and then upon reaching the end of  
> the writing, replace it with the valid marker.
>
> If neither marker is found then the index is one from before the  
> capability was added and nothing can be said about the validity.
>
> -- DM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Mark Miller-3
Hey DM,

Just to recap an earlier thread, you need the sync and you need hardware
that doesn't lie to you about the result of the sync.

Here is an excerpt about Digg running into that issue:

"They had problems with their storage system telling them writes were on
disk when they really weren't. Controllers do this to improve the
appearance of their performance. But what it does is leave a giant data
integrity whole in failure scenarios. This is really a pretty common
problem and can be hard to fix, depending on your hardware setup."

There is a lot of good stuff relating to this in the discussion
surrounding the JIRA issue.

robert engels wrote:

> That doesn't help, with lazy writing/buffering by the OS, there is no
> guarantee that if the last written block is ok, that earlier blocks in
> the file are....
>
> The OS/drive is going to physically write them in the most efficient
> manner. Only after a sync would this hold true (which is what we are
> trying to avoid).
>
> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>
>>
>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>
>>>
>>> robert engels wrote:
>>>
>>>> Do we have any way of determining if a segment is definitely
>>>> OK/VALID ?
>>>
>>> The only way I know is the CheckIndex tool, and it's rather slow (and
>>> it's not clear that it always catches all corruption).
>>
>> Just a thought. It seems that the discussion has revolved around
>> whether a crash or similar event has left the file in an inconsistent
>> state. Without looking into how it is actually done, I'm going to
>> guess that the writing is done from the start of the file to its end.
>> That is, no "out of order" writing.
>>
>> If this is the case, how about adding a marker to the end of the file
>> of a known size and pattern. If it is present then it is presumed
>> that there were no errors in getting to that point.
>>
>> Even with out of order writing, one could write an 'INVALID' marker
>> at the beginning of the operation and then upon reaching the end of
>> the writing, replace it with the valid marker.
>>
>> If neither marker is found then the index is one from before the
>> capability was added and nothing can be said about the validity.
>>
>> -- DM
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

DM Smith

On Feb 6, 2008, at 6:42 PM, Mark Miller wrote:

> Hey DM,
>
> Just to recap an earlier thread, you need the sync and you need  
> hardware that doesn't lie to you about the result of the sync.
>
> Here is an excerpt about Digg running into that issue:
>
> "They had problems with their storage system telling them writes  
> were on disk when they really weren't. Controllers do this to  
> improve the appearance of their performance. But what it does is  
> leave a giant data integrity whole in failure scenarios. This is  
> really a pretty common problem and can be hard to fix, depending on  
> your hardware setup."
>
> There is a lot of good stuff relating to this in the discussion  
> surrounding the JIRA issue.

I guess I can take that dull tool out of my tool box. :(

BTW, I followed the thread and the Jira discussion, but I missed that.

>
>
> robert engels wrote:
>> That doesn't help, with lazy writing/buffering by the OS, there is  
>> no guarantee that if the last written block is ok, that earlier  
>> blocks in the file are....
>>
>> The OS/drive is going to physically write them in the most  
>> efficient manner. Only after a sync would this hold true (which is  
>> what we are trying to avoid).
>>
>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>
>>>
>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>
>>>>
>>>> robert engels wrote:
>>>>
>>>>> Do we have any way of determining if a segment is definitely OK/
>>>>> VALID ?
>>>>
>>>> The only way I know is the CheckIndex tool, and it's rather slow  
>>>> (and
>>>> it's not clear that it always catches all corruption).
>>>
>>> Just a thought. It seems that the discussion has revolved around  
>>> whether a crash or similar event has left the file in an  
>>> inconsistent state. Without looking into how it is actually done,  
>>> I'm going to guess that the writing is done from the start of the  
>>> file to its end. That is, no "out of order" writing.
>>>
>>> If this is the case, how about adding a marker to the end of the  
>>> file of a known size and pattern. If it is present then it is  
>>> presumed that there were no errors in getting to that point.
>>>
>>> Even with out of order writing, one could write an 'INVALID'  
>>> marker at the beginning of the operation and then upon reaching  
>>> the end of the writing, replace it with the valid marker.
>>>
>>> If neither marker is found then the index is one from before the  
>>> capability was added and nothing can be said about the validity.
>>>
>>> -- DM
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Andrew Zhang-2
In reply to this post by Robert Engels
On Feb 7, 2008 7:22 AM, robert engels <[hidden email]> wrote:

> That doesn't help, with lazy writing/buffering by the OS, there is no
> guarantee that if the last written block is ok, that earlier blocks
> in the file are....
>
> The OS/drive is going to physically write them in the most efficient
> manner. Only after a sync would this hold true (which is what we are
> trying to avoid).


Hi, how about asynchronous commit? i.e. use a thread to sync the data.

We only need to make sure that all data are written to the storage before
the next operation?

>
>
> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>
> >
> > On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
> >
> >>
> >> robert engels wrote:
> >>
> >>> Do we have any way of determining if a segment is definitely OK/
> >>> VALID ?
> >>
> >> The only way I know is the CheckIndex tool, and it's rather slow (and
> >> it's not clear that it always catches all corruption).
> >
> > Just a thought. It seems that the discussion has revolved around
> > whether a crash or similar event has left the file in an
> > inconsistent state. Without looking into how it is actually done,
> > I'm going to guess that the writing is done from the start of the
> > file to its end. That is, no "out of order" writing.
> >
> > If this is the case, how about adding a marker to the end of the
> > file of a known size and pattern. If it is present then it is
> > presumed that there were no errors in getting to that point.
> >
> > Even with out of order writing, one could write an 'INVALID' marker
> > at the beginning of the operation and then upon reaching the end of
> > the writing, replace it with the valid marker.
> >
> > If neither marker is found then the index is one from before the
> > capability was added and nothing can be said about the validity.
> >
> > -- DM
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Best regards,
Andrew Zhang

db4o - database for Android: www.db4o.com
http://zhanghuangzhu.blogspot.com/
Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
That is the problem, waiting for the full sync (of all of the segment  
files) takes quite a while... syncing a single log file is much more  
efficient.

On Feb 6, 2008, at 9:41 PM, Andrew Zhang wrote:

> On Feb 7, 2008 7:22 AM, robert engels <[hidden email]> wrote:
>
>> That doesn't help, with lazy writing/buffering by the OS, there is no
>> guarantee that if the last written block is ok, that earlier blocks
>> in the file are....
>>
>> The OS/drive is going to physically write them in the most efficient
>> manner. Only after a sync would this hold true (which is what we are
>> trying to avoid).
>
>
> Hi, how about asynchronous commit? i.e. use a thread to sync the data.
>
> We only need to make sure that all data are written to the storage  
> before
> the next operation?
>
>>
>>
>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>
>>>
>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>
>>>>
>>>> robert engels wrote:
>>>>
>>>>> Do we have any way of determining if a segment is definitely OK/
>>>>> VALID ?
>>>>
>>>> The only way I know is the CheckIndex tool, and it's rather slow  
>>>> (and
>>>> it's not clear that it always catches all corruption).
>>>
>>> Just a thought. It seems that the discussion has revolved around
>>> whether a crash or similar event has left the file in an
>>> inconsistent state. Without looking into how it is actually done,
>>> I'm going to guess that the writing is done from the start of the
>>> file to its end. That is, no "out of order" writing.
>>>
>>> If this is the case, how about adding a marker to the end of the
>>> file of a known size and pattern. If it is present then it is
>>> presumed that there were no errors in getting to that point.
>>>
>>> Even with out of order writing, one could write an 'INVALID' marker
>>> at the beginning of the operation and then upon reaching the end of
>>> the writing, replace it with the valid marker.
>>>
>>> If neither marker is found then the index is one from before the
>>> capability was added and nothing can be said about the validity.
>>>
>>> -- DM
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> --
> Best regards,
> Andrew Zhang
>
> db4o - database for Android: www.db4o.com
> http://zhanghuangzhu.blogspot.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2
In reply to this post by Robert Engels

But then you're back to syncing in a BG thread, right?  We've come
full circle.

Asynchronously syncing give the best performance we've seen so far,
and so that's the current patch on LUCENE-1044 (using CMS's threads).
Using a transaction log would also require async. syncing, but then
would also add 2X IO cost of flushing and 2X disk usage between
commits.

I don't see how that could be faster.  I expect it to perform quite a
bit worse.

Also, I tested system wide sync in LUCENE-1044 and found it no better
than syncing individual files synchronously (which was our worst
performance number). And I don't think Lucene should be doing a system
wide sync.  There may be other processes doing IO whose buffers we
shouldn't, and don't need to, sync.

Mike

robert engels wrote:

> Yes, but this pruning could be more efficient. On a background  
> thread, get current segment from segments file, call the system  
> wide sync ( e.g. System.exec("fsync"), then you can purge the  
> transaction logs for all segments up to that one. Since it is a  
> background operation, you are not blocking the writing of new  
> segments and tx logs.
>
> On Feb 6, 2008, at 4:42 PM, Michael McCandless wrote:
>
>>
>> robert engels wrote:
>>
>>> Do we have any way of determining if a segment is definitely OK/
>>> VALID ?
>>
>> The only way I know is the CheckIndex tool, and it's rather slow (and
>> it's not clear that it always catches all corruption).
>>
>>> If so, a much more efficient transactional system could be  
>>> developed.
>>>
>>> Serialize the updates to a log file. Sync the log. Update the  
>>> lucene index WITHOUT any sync.  Log file writing/sync is VERY  
>>> efficient since it is sequential, and a single file.
>>>
>>> Upon open of the index, detect if index was not shutdown cleanly.  
>>> If so, determine the last valid segment, delete the bad segments,  
>>> and then perform the updates (from the log file) since the last  
>>> valid segment was written.
>>>
>>> The detection could be a VERY slow operation, but this is ok,  
>>> since it should be rare, and then you will only pay this price on  
>>> the rare occasion, not on every update.
>>
>> Wouldn't you still need to sync periodically, so you can prune the
>> transaction log?  Else your transaction log is growing as fast as the
>> index?  (You've doubled disk usage).
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2
In reply to this post by DM Smith

DM Smith wrote:

>
> On Feb 6, 2008, at 6:42 PM, Mark Miller wrote:
>
>> Hey DM,
>>
>> Just to recap an earlier thread, you need the sync and you need  
>> hardware that doesn't lie to you about the result of the sync.
>>
>> Here is an excerpt about Digg running into that issue:
>>
>> "They had problems with their storage system telling them writes  
>> were on disk when they really weren't. Controllers do this to  
>> improve the appearance of their performance. But what it does is  
>> leave a giant data integrity whole in failure scenarios. This is  
>> really a pretty common problem and can be hard to fix, depending  
>> on your hardware setup."
>>
>> There is a lot of good stuff relating to this in the discussion  
>> surrounding the JIRA issue.
>
> I guess I can take that dull tool out of my tool box. :(
>
> BTW, I followed the thread and the Jira discussion, but I missed that.

I too followed the thread & Jira discussion and missed this!

>>
>>
>> robert engels wrote:
>>> That doesn't help, with lazy writing/buffering by the OS, there  
>>> is no guarantee that if the last written block is ok, that  
>>> earlier blocks in the file are....
>>>
>>> The OS/drive is going to physically write them in the most  
>>> efficient manner. Only after a sync would this hold true (which  
>>> is what we are trying to avoid).
>>>
>>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>>
>>>>
>>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>>
>>>>>
>>>>> robert engels wrote:
>>>>>
>>>>>> Do we have any way of determining if a segment is definitely  
>>>>>> OK/VALID ?
>>>>>
>>>>> The only way I know is the CheckIndex tool, and it's rather  
>>>>> slow (and
>>>>> it's not clear that it always catches all corruption).
>>>>
>>>> Just a thought. It seems that the discussion has revolved around  
>>>> whether a crash or similar event has left the file in an  
>>>> inconsistent state. Without looking into how it is actually  
>>>> done, I'm going to guess that the writing is done from the start  
>>>> of the file to its end. That is, no "out of order" writing.
>>>>
>>>> If this is the case, how about adding a marker to the end of the  
>>>> file of a known size and pattern. If it is present then it is  
>>>> presumed that there were no errors in getting to that point.
>>>>
>>>> Even with out of order writing, one could write an 'INVALID'  
>>>> marker at the beginning of the operation and then upon reaching  
>>>> the end of the writing, replace it with the valid marker.
>>>>
>>>> If neither marker is found then the index is one from before the  
>>>> capability was added and nothing can be said about the validity.
>>>>
>>>> -- DM
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2
In reply to this post by Andrew Zhang-2

In fact this is exactly the approach in the final patch on  
LUCENE-1044 and it gives far better performance than the simply  
synchronous (original) approach of syncing every segment file on close.

Using a transaction log would also require periodic syncing.

LUCENE-1044 syncs files after every merge, in the background thread  
of ConcurrentMergeScheduler, which is nice because it does not block  
further add/update/deleteDocument calls on the writer.

Mike

Andrew Zhang wrote:

> On Feb 7, 2008 7:22 AM, robert engels <[hidden email]> wrote:
>
>> That doesn't help, with lazy writing/buffering by the OS, there is no
>> guarantee that if the last written block is ok, that earlier blocks
>> in the file are....
>>
>> The OS/drive is going to physically write them in the most efficient
>> manner. Only after a sync would this hold true (which is what we are
>> trying to avoid).
>
>
> Hi, how about asynchronous commit? i.e. use a thread to sync the data.
>
> We only need to make sure that all data are written to the storage  
> before
> the next operation?
>
>>
>>
>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>
>>>
>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>
>>>>
>>>> robert engels wrote:
>>>>
>>>>> Do we have any way of determining if a segment is definitely OK/
>>>>> VALID ?
>>>>
>>>> The only way I know is the CheckIndex tool, and it's rather slow  
>>>> (and
>>>> it's not clear that it always catches all corruption).
>>>
>>> Just a thought. It seems that the discussion has revolved around
>>> whether a crash or similar event has left the file in an
>>> inconsistent state. Without looking into how it is actually done,
>>> I'm going to guess that the writing is done from the start of the
>>> file to its end. That is, no "out of order" writing.
>>>
>>> If this is the case, how about adding a marker to the end of the
>>> file of a known size and pattern. If it is present then it is
>>> presumed that there were no errors in getting to that point.
>>>
>>> Even with out of order writing, one could write an 'INVALID' marker
>>> at the beginning of the operation and then upon reaching the end of
>>> the writing, replace it with the valid marker.
>>>
>>> If neither marker is found then the index is one from before the
>>> capability was added and nothing can be said about the validity.
>>>
>>> -- DM
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> --
> Best regards,
> Andrew Zhang
>
> db4o - database for Android: www.db4o.com
> http://zhanghuangzhu.blogspot.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Mark Miller-3
In reply to this post by Michael McCandless-2
We should really probably mention it in the JavaDoc when the issue is
done. I think both yonik and robert pointed it out, and ever since then
I have seen issues regarding it everywhere.

http://hardware.slashdot.org/article.pl?sid=05/05/13/0529252

Apparently, your just not ACID unless you have hardware you know is
properly reporting the sync call.

Here is a good snippet from the h2database faq:
http://www.h2database.com/html/frame.html?advanced.html%23durability_problems&main


Michael McCandless wrote:

>
> DM Smith wrote:
>
>>
>> On Feb 6, 2008, at 6:42 PM, Mark Miller wrote:
>>
>>> Hey DM,
>>>
>>> Just to recap an earlier thread, you need the sync and you need
>>> hardware that doesn't lie to you about the result of the sync.
>>>
>>> Here is an excerpt about Digg running into that issue:
>>>
>>> "They had problems with their storage system telling them writes
>>> were on disk when they really weren't. Controllers do this to
>>> improve the appearance of their performance. But what it does is
>>> leave a giant data integrity whole in failure scenarios. This is
>>> really a pretty common problem and can be hard to fix, depending on
>>> your hardware setup."
>>>
>>> There is a lot of good stuff relating to this in the discussion
>>> surrounding the JIRA issue.
>>
>> I guess I can take that dull tool out of my tool box. :(
>>
>> BTW, I followed the thread and the Jira discussion, but I missed that.
>
> I too followed the thread & Jira discussion and missed this!
>
>>>
>>>
>>> robert engels wrote:
>>>> That doesn't help, with lazy writing/buffering by the OS, there is
>>>> no guarantee that if the last written block is ok, that earlier
>>>> blocks in the file are....
>>>>
>>>> The OS/drive is going to physically write them in the most
>>>> efficient manner. Only after a sync would this hold true (which is
>>>> what we are trying to avoid).
>>>>
>>>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>>>
>>>>>
>>>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>>>
>>>>>>
>>>>>> robert engels wrote:
>>>>>>
>>>>>>> Do we have any way of determining if a segment is definitely
>>>>>>> OK/VALID ?
>>>>>>
>>>>>> The only way I know is the CheckIndex tool, and it's rather slow
>>>>>> (and
>>>>>> it's not clear that it always catches all corruption).
>>>>>
>>>>> Just a thought. It seems that the discussion has revolved around
>>>>> whether a crash or similar event has left the file in an
>>>>> inconsistent state. Without looking into how it is actually done,
>>>>> I'm going to guess that the writing is done from the start of the
>>>>> file to its end. That is, no "out of order" writing.
>>>>>
>>>>> If this is the case, how about adding a marker to the end of the
>>>>> file of a known size and pattern. If it is present then it is
>>>>> presumed that there were no errors in getting to that point.
>>>>>
>>>>> Even with out of order writing, one could write an 'INVALID'
>>>>> marker at the beginning of the operation and then upon reaching
>>>>> the end of the writing, replace it with the valid marker.
>>>>>
>>>>> If neither marker is found then the index is one from before the
>>>>> capability was added and nothing can be said about the validity.
>>>>>
>>>>> -- DM
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2

Good idea; I'll call this ("if your hardware ignores the sync() call  
then you're in trouble") out in the javadocs with LUCENE-1044.

Mike

Mark Miller wrote:

> We should really probably mention it in the JavaDoc when the issue  
> is done. I think both yonik and robert pointed it out, and ever  
> since then I have seen issues regarding it everywhere.
>
> http://hardware.slashdot.org/article.pl?sid=05/05/13/0529252
>
> Apparently, your just not ACID unless you have hardware you know is  
> properly reporting the sync call.
>
> Here is a good snippet from the h2database faq: http://
> www.h2database.com/html/frame.html?advanced.html%
> 23durability_problems&main
>
>
> Michael McCandless wrote:
>>
>> DM Smith wrote:
>>
>>>
>>> On Feb 6, 2008, at 6:42 PM, Mark Miller wrote:
>>>
>>>> Hey DM,
>>>>
>>>> Just to recap an earlier thread, you need the sync and you need  
>>>> hardware that doesn't lie to you about the result of the sync.
>>>>
>>>> Here is an excerpt about Digg running into that issue:
>>>>
>>>> "They had problems with their storage system telling them writes  
>>>> were on disk when they really weren't. Controllers do this to  
>>>> improve the appearance of their performance. But what it does is  
>>>> leave a giant data integrity whole in failure scenarios. This is  
>>>> really a pretty common problem and can be hard to fix, depending  
>>>> on your hardware setup."
>>>>
>>>> There is a lot of good stuff relating to this in the discussion  
>>>> surrounding the JIRA issue.
>>>
>>> I guess I can take that dull tool out of my tool box. :(
>>>
>>> BTW, I followed the thread and the Jira discussion, but I missed  
>>> that.
>>
>> I too followed the thread & Jira discussion and missed this!
>>
>>>>
>>>>
>>>> robert engels wrote:
>>>>> That doesn't help, with lazy writing/buffering by the OS, there  
>>>>> is no guarantee that if the last written block is ok, that  
>>>>> earlier blocks in the file are....
>>>>>
>>>>> The OS/drive is going to physically write them in the most  
>>>>> efficient manner. Only after a sync would this hold true (which  
>>>>> is what we are trying to avoid).
>>>>>
>>>>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>>>>
>>>>>>
>>>>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>>>>
>>>>>>>
>>>>>>> robert engels wrote:
>>>>>>>
>>>>>>>> Do we have any way of determining if a segment is definitely  
>>>>>>>> OK/VALID ?
>>>>>>>
>>>>>>> The only way I know is the CheckIndex tool, and it's rather  
>>>>>>> slow (and
>>>>>>> it's not clear that it always catches all corruption).
>>>>>>
>>>>>> Just a thought. It seems that the discussion has revolved  
>>>>>> around whether a crash or similar event has left the file in  
>>>>>> an inconsistent state. Without looking into how it is actually  
>>>>>> done, I'm going to guess that the writing is done from the  
>>>>>> start of the file to its end. That is, no "out of order" writing.
>>>>>>
>>>>>> If this is the case, how about adding a marker to the end of  
>>>>>> the file of a known size and pattern. If it is present then it  
>>>>>> is presumed that there were no errors in getting to that point.
>>>>>>
>>>>>> Even with out of order writing, one could write an 'INVALID'  
>>>>>> marker at the beginning of the operation and then upon  
>>>>>> reaching the end of the writing, replace it with the valid  
>>>>>> marker.
>>>>>>
>>>>>> If neither marker is found then the index is one from before  
>>>>>> the capability was added and nothing can be said about the  
>>>>>> validity.
>>>>>>
>>>>>> -- DM
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> ----
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> ---
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
In reply to this post by Michael McCandless-2
This is simply not true. Two different issues are at play. You cannot  
have a true 'commit' unless it is synchronous!

Lucene-1044 might allow the index to be brought back to a consistent  
state, but not one that is consistent with a synchronization point.

For example, I write three documents to the index. I call commit. It  
returns. After this, those documents MUST be in the index under any  
conditions. Lucene 1044 does not ensure this.

By writing the operations (deletes and updates) to a log file first,  
and syncing the log file, then a failure during the index writing/
merging can be fixed by rolling forward the log.


On Feb 7, 2008, at 4:29 AM, Michael McCandless wrote:

>
> In fact this is exactly the approach in the final patch on  
> LUCENE-1044 and it gives far better performance than the simply  
> synchronous (original) approach of syncing every segment file on  
> close.
>
> Using a transaction log would also require periodic syncing.
>
> LUCENE-1044 syncs files after every merge, in the background thread  
> of ConcurrentMergeScheduler, which is nice because it does not  
> block further add/update/deleteDocument calls on the writer.
>
> Mike
>
> Andrew Zhang wrote:
>
>> On Feb 7, 2008 7:22 AM, robert engels <[hidden email]> wrote:
>>
>>> That doesn't help, with lazy writing/buffering by the OS, there  
>>> is no
>>> guarantee that if the last written block is ok, that earlier blocks
>>> in the file are....
>>>
>>> The OS/drive is going to physically write them in the most efficient
>>> manner. Only after a sync would this hold true (which is what we are
>>> trying to avoid).
>>
>>
>> Hi, how about asynchronous commit? i.e. use a thread to sync the  
>> data.
>>
>> We only need to make sure that all data are written to the storage  
>> before
>> the next operation?
>>
>>>
>>>
>>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>>
>>>>
>>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>>
>>>>>
>>>>> robert engels wrote:
>>>>>
>>>>>> Do we have any way of determining if a segment is definitely OK/
>>>>>> VALID ?
>>>>>
>>>>> The only way I know is the CheckIndex tool, and it's rather  
>>>>> slow (and
>>>>> it's not clear that it always catches all corruption).
>>>>
>>>> Just a thought. It seems that the discussion has revolved around
>>>> whether a crash or similar event has left the file in an
>>>> inconsistent state. Without looking into how it is actually done,
>>>> I'm going to guess that the writing is done from the start of the
>>>> file to its end. That is, no "out of order" writing.
>>>>
>>>> If this is the case, how about adding a marker to the end of the
>>>> file of a known size and pattern. If it is present then it is
>>>> presumed that there were no errors in getting to that point.
>>>>
>>>> Even with out of order writing, one could write an 'INVALID' marker
>>>> at the beginning of the operation and then upon reaching the end of
>>>> the writing, replace it with the valid marker.
>>>>
>>>> If neither marker is found then the index is one from before the
>>>> capability was added and nothing can be said about the validity.
>>>>
>>>> -- DM
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrew Zhang
>>
>> db4o - database for Android: www.db4o.com
>> http://zhanghuangzhu.blogspot.com/
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
I might be misunderstanding 1044.  There were several approaches, and  
I am not certain what was the final???

I reread the bug and am still a bit unclear.

If the segments are sync'd as part of the commit, then yes, that  
would suffice. The merges don't need to commit, you just can't delete  
the segments until the merge completes.

I  think that building the segments, and syncing each segment - since  
in most cases the caller is going to call commit as part of each  
update, is going to be slower than writing the documents/operations  
to a log file, but a lot depends on how Lucene is used (interactive  
vs. batch, lots of updates vs. a few).

I am not sure how deletions are impacted by all of this.


On Feb 7, 2008, at 9:21 AM, robert engels wrote:

> This is simply not true. Two different issues are at play. You  
> cannot have a true 'commit' unless it is synchronous!
>
> Lucene-1044 might allow the index to be brought back to a  
> consistent state, but not one that is consistent with a  
> synchronization point.
>
> For example, I write three documents to the index. I call commit.  
> It returns. After this, those documents MUST be in the index under  
> any conditions. Lucene 1044 does not ensure this.
>
> By writing the operations (deletes and updates) to a log file  
> first, and syncing the log file, then a failure during the index  
> writing/merging can be fixed by rolling forward the log.
>
>
> On Feb 7, 2008, at 4:29 AM, Michael McCandless wrote:
>
>>
>> In fact this is exactly the approach in the final patch on  
>> LUCENE-1044 and it gives far better performance than the simply  
>> synchronous (original) approach of syncing every segment file on  
>> close.
>>
>> Using a transaction log would also require periodic syncing.
>>
>> LUCENE-1044 syncs files after every merge, in the background  
>> thread of ConcurrentMergeScheduler, which is nice because it does  
>> not block further add/update/deleteDocument calls on the writer.
>>
>> Mike
>>
>> Andrew Zhang wrote:
>>
>>> On Feb 7, 2008 7:22 AM, robert engels <[hidden email]> wrote:
>>>
>>>> That doesn't help, with lazy writing/buffering by the OS, there  
>>>> is no
>>>> guarantee that if the last written block is ok, that earlier blocks
>>>> in the file are....
>>>>
>>>> The OS/drive is going to physically write them in the most  
>>>> efficient
>>>> manner. Only after a sync would this hold true (which is what we  
>>>> are
>>>> trying to avoid).
>>>
>>>
>>> Hi, how about asynchronous commit? i.e. use a thread to sync the  
>>> data.
>>>
>>> We only need to make sure that all data are written to the  
>>> storage before
>>> the next operation?
>>>
>>>>
>>>>
>>>> On Feb 6, 2008, at 5:15 PM, DM Smith wrote:
>>>>
>>>>>
>>>>> On Feb 6, 2008, at 5:42 PM, Michael McCandless wrote:
>>>>>
>>>>>>
>>>>>> robert engels wrote:
>>>>>>
>>>>>>> Do we have any way of determining if a segment is definitely OK/
>>>>>>> VALID ?
>>>>>>
>>>>>> The only way I know is the CheckIndex tool, and it's rather  
>>>>>> slow (and
>>>>>> it's not clear that it always catches all corruption).
>>>>>
>>>>> Just a thought. It seems that the discussion has revolved around
>>>>> whether a crash or similar event has left the file in an
>>>>> inconsistent state. Without looking into how it is actually done,
>>>>> I'm going to guess that the writing is done from the start of the
>>>>> file to its end. That is, no "out of order" writing.
>>>>>
>>>>> If this is the case, how about adding a marker to the end of the
>>>>> file of a known size and pattern. If it is present then it is
>>>>> presumed that there were no errors in getting to that point.
>>>>>
>>>>> Even with out of order writing, one could write an 'INVALID'  
>>>>> marker
>>>>> at the beginning of the operation and then upon reaching the  
>>>>> end of
>>>>> the writing, replace it with the valid marker.
>>>>>
>>>>> If neither marker is found then the index is one from before the
>>>>> capability was added and nothing can be said about the validity.
>>>>>
>>>>> -- DM
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> ---
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrew Zhang
>>>
>>> db4o - database for Android: www.db4o.com
>>> http://zhanghuangzhu.blogspot.com/
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Michael McCandless-2

robert engels wrote:

> I might be misunderstanding 1044.  There were several approaches,  
> and I am not certain what was the final???

The final approach (take 7) is to make the index consistent (sync the  
files) after finishing a merge.  Also, a new method ("commit") is  
added which will force a synchronous sync while you wait.  Close also  
does this.

> I reread the bug and am still a bit unclear.
>
> If the segments are sync'd as part of the commit, then yes, that  
> would suffice. The merges don't need to commit, you just can't  
> delete the segments until the merge completes.
>
> I  think that building the segments, and syncing each segment -  
> since in most cases the caller is going to call commit as part of  
> each update, is going to be slower than writing the documents/
> operations to a log file, but a lot depends on how Lucene is used  
> (interactive vs. batch, lots of updates vs. a few).

Well, and based on how frequently you prune the transaction log (sync  
the real files).  I think the 2X IO cost is going to make performance  
worse with the transaction log.

> I am not sure how deletions are impacted by all of this.

Should be fine?  The *.del files need to be sync'd just like the rest  
of the segments files.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
My point is that commit needs to be used in most applications, and  
the commit in Lucene is very slow.

You don't have 2x the IO cost, mainly because only the log file needs  
to be sync'd.  The index only has to be sync'd eventually, in order  
to prune the logfile - this can be done in the background, improving  
the performance of update and commit cycle.

Also, writing the log file is very efficiently because it is an  
append/sequential operation. Writing the segment files writes  
multiple files - essentially causing random access writes.

I guess I don't see the benefit of 1044 if you can't guarantee the  
index is at a certain point (you can by calling commit(), but it is  
VERY slow).

I was thinking a better design is to serialize the documents/
operations to disk, and maintain an in memory index of updates/
removes, and then merge those indexes to the main when needed - using  
a parallel reader on both in the mean-time.

On Feb 7, 2008, at 3:06 PM, Michael McCandless wrote:

>
> robert engels wrote:
>
>> I might be misunderstanding 1044.  There were several approaches,  
>> and I am not certain what was the final???
>
> The final approach (take 7) is to make the index consistent (sync  
> the files) after finishing a merge.  Also, a new method ("commit")  
> is added which will force a synchronous sync while you wait.  Close  
> also does this.
>
>> I reread the bug and am still a bit unclear.
>>
>> If the segments are sync'd as part of the commit, then yes, that  
>> would suffice. The merges don't need to commit, you just can't  
>> delete the segments until the merge completes.
>>
>> I  think that building the segments, and syncing each segment -  
>> since in most cases the caller is going to call commit as part of  
>> each update, is going to be slower than writing the documents/
>> operations to a log file, but a lot depends on how Lucene is used  
>> (interactive vs. batch, lots of updates vs. a few).
>
> Well, and based on how frequently you prune the transaction log  
> (sync the real files).  I think the 2X IO cost is going to make  
> performance worse with the transaction log.
>
>> I am not sure how deletions are impacted by all of this.
>
> Should be fine?  The *.del files need to be sync'd just like the  
> rest of the segments files.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Mike Klaas

On 7-Feb-08, at 2:00 PM, robert engels wrote:

> My point is that commit needs to be used in most applications, and  
> the commit in Lucene is very slow.
>
> You don't have 2x the IO cost, mainly because only the log file  
> needs to be sync'd.  The index only has to be sync'd eventually, in  
> order to prune the logfile - this can be done in the background,  
> improving the performance of update and commit cycle.
>
> Also, writing the log file is very efficiently because it is an  
> append/sequential operation. Writing the segment files writes  
> multiple files - essentially causing random access writes.

For large segments, multiple sequentially-written large files should  
perform similarly to one large sequentially-written file.  It is only  
close to random access on the smallest segments (which a sufficiently-
large flush-by-ram shouldn't produce).

-Mike


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: detected corrupted index / performance improvement

Robert Engels
I don't think that is true - but I'm probably wrong though :).

My understanding is that several files are written in parallel  
(during the merge), causing random access. After the files are  
written, then they are all reread and written as a CFS file  
(essential sequential - although the read and write is going to cause  
head movement).

The code:

private IndexOutput tvx, tvf, tvd;              // To write term vectors
private FieldsWriter fieldsWriter;

is my clue that several files are written at once.

On Feb 7, 2008, at 5:19 PM, Mike Klaas wrote:

>
> On 7-Feb-08, at 2:00 PM, robert engels wrote:
>
>> My point is that commit needs to be used in most applications, and  
>> the commit in Lucene is very slow.
>>
>> You don't have 2x the IO cost, mainly because only the log file  
>> needs to be sync'd.  The index only has to be sync'd eventually,  
>> in order to prune the logfile - this can be done in the  
>> background, improving the performance of update and commit cycle.
>>
>> Also, writing the log file is very efficiently because it is an  
>> append/sequential operation. Writing the segment files writes  
>> multiple files - essentially causing random access writes.
>
> For large segments, multiple sequentially-written large files  
> should perform similarly to one large sequentially-written file.  
> It is only close to random access on the smallest segments (which a  
> sufficiently-large flush-by-ram shouldn't produce).
>
> -Mike
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

12