Fwd: index format error because disk full - possible Lucene bug?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: index format error because disk full - possible Lucene bug?

Lance Norskog-2
Forwarded to lucene-dev as a possible bug.

On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[hidden email]> wrote:

> I use SegmentInfos to read the segment_N file and found the error is
> that it try to load deletedDocs but the .del file's size is 0(because
> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
> deleted Docs.
> But I think there is some bug. The logic of  write my be -- it first
> writes the .del file then write the segment_N file. But it only write
> to buffer and don't flush to disk immediately. So when disk full. it
> may happen that segment_N file is flushed but del file faild.
>
> 2010/7/8 Lance Norskog <[hidden email]>:
>> If autocommit does not to an automatic rollback, that is a serious bug.
>>
>> There should be a way to detect that an automatic rollback has
>> happened, but I don't know what it is. Maybe something in the Solr
>> MBeans?
>>
>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[hidden email]> wrote:
>>>
>>> I haven't used this myself, but Solr supports a
>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>> function. It is supposed to rollback to the state at the previous commit. So
>>> you may want to turn off auto-commit on the index you are updating if you
>>> want to control what that last commit level is.
>>>
>>> However, in your case if the index gets corrupted due to a disk full
>>> situation, I don't know what rollback would do, if anything, to help. You
>>> may need to play with the scenario to see what would happen.
>>>
>>> If you are using the DataImportHandler it may handle the rollback for
>>> you...again, however, it may not deal with disk full situations gracefully
>>> either.
>>> --
>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> [hidden email]
>>
>



--
Lance Norskog
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index format error because disk full - possible Lucene bug?

Lance Norskog-2
I think he saying that there is a race condition involving writing out
the buffered changes from ram, doing deletes, doing commits and
writing out new segment files when Lucene runs out of disk. This race
condition (or possibly a swallowed I/O Exception) may cause bogus
segment files, and the index becomes unuseable even though the
existing files make up a clean index.

On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <[hidden email]> wrote:

> Forwarded to lucene-dev as a possible bug.
>
> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[hidden email]> wrote:
>> I use SegmentInfos to read the segment_N file and found the error is
>> that it try to load deletedDocs but the .del file's size is 0(because
>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>> deleted Docs.
>> But I think there is some bug. The logic of  write my be -- it first
>> writes the .del file then write the segment_N file. But it only write
>> to buffer and don't flush to disk immediately. So when disk full. it
>> may happen that segment_N file is flushed but del file faild.
>>
>> 2010/7/8 Lance Norskog <[hidden email]>:
>>> If autocommit does not to an automatic rollback, that is a serious bug.
>>>
>>> There should be a way to detect that an automatic rollback has
>>> happened, but I don't know what it is. Maybe something in the Solr
>>> MBeans?
>>>
>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[hidden email]> wrote:
>>>>
>>>> I haven't used this myself, but Solr supports a
>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>>> function. It is supposed to rollback to the state at the previous commit. So
>>>> you may want to turn off auto-commit on the index you are updating if you
>>>> want to control what that last commit level is.
>>>>
>>>> However, in your case if the index gets corrupted due to a disk full
>>>> situation, I don't know what rollback would do, if anything, to help. You
>>>> may need to play with the scenario to see what would happen.
>>>>
>>>> If you are using the DataImportHandler it may handle the rollback for
>>>> you...again, however, it may not deal with disk full situations gracefully
>>>> either.
>>>> --
>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> [hidden email]
>>>
>>
>
>
>
> --
> Lance Norskog
> [hidden email]
>



--
Lance Norskog
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index format error because disk full - possible Lucene bug?

Michael McCandless-2
I responded on the original thread.

Disk full should never cause index corruption except on very old
versions of Lucene...

Mike

On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <[hidden email]> wrote:

> I think he saying that there is a race condition involving writing out
> the buffered changes from ram, doing deletes, doing commits and
> writing out new segment files when Lucene runs out of disk. This race
> condition (or possibly a swallowed I/O Exception) may cause bogus
> segment files, and the index becomes unuseable even though the
> existing files make up a clean index.
>
> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <[hidden email]> wrote:
>> Forwarded to lucene-dev as a possible bug.
>>
>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[hidden email]> wrote:
>>> I use SegmentInfos to read the segment_N file and found the error is
>>> that it try to load deletedDocs but the .del file's size is 0(because
>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>>> deleted Docs.
>>> But I think there is some bug. The logic of  write my be -- it first
>>> writes the .del file then write the segment_N file. But it only write
>>> to buffer and don't flush to disk immediately. So when disk full. it
>>> may happen that segment_N file is flushed but del file faild.
>>>
>>> 2010/7/8 Lance Norskog <[hidden email]>:
>>>> If autocommit does not to an automatic rollback, that is a serious bug.
>>>>
>>>> There should be a way to detect that an automatic rollback has
>>>> happened, but I don't know what it is. Maybe something in the Solr
>>>> MBeans?
>>>>
>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[hidden email]> wrote:
>>>>>
>>>>> I haven't used this myself, but Solr supports a
>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>>>> function. It is supposed to rollback to the state at the previous commit. So
>>>>> you may want to turn off auto-commit on the index you are updating if you
>>>>> want to control what that last commit level is.
>>>>>
>>>>> However, in your case if the index gets corrupted due to a disk full
>>>>> situation, I don't know what rollback would do, if anything, to help. You
>>>>> may need to play with the scenario to see what would happen.
>>>>>
>>>>> If you are using the DataImportHandler it may handle the rollback for
>>>>> you...again, however, it may not deal with disk full situations gracefully
>>>>> either.
>>>>> --
>>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lance Norskog
>>>> [hidden email]
>>>>
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> [hidden email]
>>
>
>
>
> --
> Lance Norskog
> [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index format error because disk full - possible Lucene bug?

Earwin Burrfoot
I believe I've seen a similar condition a few times.
A segments file referring zero-length segment files after a disk full event.

On Fri, Jul 9, 2010 at 13:37, Michael McCandless
<[hidden email]> wrote:

> I responded on the original thread.
>
> Disk full should never cause index corruption except on very old
> versions of Lucene...
>
> Mike
>
> On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <[hidden email]> wrote:
>> I think he saying that there is a race condition involving writing out
>> the buffered changes from ram, doing deletes, doing commits and
>> writing out new segment files when Lucene runs out of disk. This race
>> condition (or possibly a swallowed I/O Exception) may cause bogus
>> segment files, and the index becomes unuseable even though the
>> existing files make up a clean index.
>>
>> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <[hidden email]> wrote:
>>> Forwarded to lucene-dev as a possible bug.
>>>
>>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[hidden email]> wrote:
>>>> I use SegmentInfos to read the segment_N file and found the error is
>>>> that it try to load deletedDocs but the .del file's size is 0(because
>>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>>>> deleted Docs.
>>>> But I think there is some bug. The logic of  write my be -- it first
>>>> writes the .del file then write the segment_N file. But it only write
>>>> to buffer and don't flush to disk immediately. So when disk full. it
>>>> may happen that segment_N file is flushed but del file faild.
>>>>
>>>> 2010/7/8 Lance Norskog <[hidden email]>:
>>>>> If autocommit does not to an automatic rollback, that is a serious bug.
>>>>>
>>>>> There should be a way to detect that an automatic rollback has
>>>>> happened, but I don't know what it is. Maybe something in the Solr
>>>>> MBeans?
>>>>>
>>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[hidden email]> wrote:
>>>>>>
>>>>>> I haven't used this myself, but Solr supports a
>>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>>>>> function. It is supposed to rollback to the state at the previous commit. So
>>>>>> you may want to turn off auto-commit on the index you are updating if you
>>>>>> want to control what that last commit level is.
>>>>>>
>>>>>> However, in your case if the index gets corrupted due to a disk full
>>>>>> situation, I don't know what rollback would do, if anything, to help. You
>>>>>> may need to play with the scenario to see what would happen.
>>>>>>
>>>>>> If you are using the DataImportHandler it may handle the rollback for
>>>>>> you...again, however, it may not deal with disk full situations gracefully
>>>>>> either.
>>>>>> --
>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> [hidden email]
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> [hidden email]
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> [hidden email]
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>



--
Kirill Zakharenko/Кирилл Захаренко ([hidden email])
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index format error because disk full - possible Lucene bug?

Michael McCandless-2
Which version of Lucene, and which OS/IO system?

Is it possible fsync lies in your env?

Mike

On Fri, Jul 9, 2010 at 7:59 AM, Earwin Burrfoot <[hidden email]> wrote:

> I believe I've seen a similar condition a few times.
> A segments file referring zero-length segment files after a disk full event.
>
> On Fri, Jul 9, 2010 at 13:37, Michael McCandless
> <[hidden email]> wrote:
>> I responded on the original thread.
>>
>> Disk full should never cause index corruption except on very old
>> versions of Lucene...
>>
>> Mike
>>
>> On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <[hidden email]> wrote:
>>> I think he saying that there is a race condition involving writing out
>>> the buffered changes from ram, doing deletes, doing commits and
>>> writing out new segment files when Lucene runs out of disk. This race
>>> condition (or possibly a swallowed I/O Exception) may cause bogus
>>> segment files, and the index becomes unuseable even though the
>>> existing files make up a clean index.
>>>
>>> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <[hidden email]> wrote:
>>>> Forwarded to lucene-dev as a possible bug.
>>>>
>>>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[hidden email]> wrote:
>>>>> I use SegmentInfos to read the segment_N file and found the error is
>>>>> that it try to load deletedDocs but the .del file's size is 0(because
>>>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>>>>> deleted Docs.
>>>>> But I think there is some bug. The logic of  write my be -- it first
>>>>> writes the .del file then write the segment_N file. But it only write
>>>>> to buffer and don't flush to disk immediately. So when disk full. it
>>>>> may happen that segment_N file is flushed but del file faild.
>>>>>
>>>>> 2010/7/8 Lance Norskog <[hidden email]>:
>>>>>> If autocommit does not to an automatic rollback, that is a serious bug.
>>>>>>
>>>>>> There should be a way to detect that an automatic rollback has
>>>>>> happened, but I don't know what it is. Maybe something in the Solr
>>>>>> MBeans?
>>>>>>
>>>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[hidden email]> wrote:
>>>>>>>
>>>>>>> I haven't used this myself, but Solr supports a
>>>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>>>>>> function. It is supposed to rollback to the state at the previous commit. So
>>>>>>> you may want to turn off auto-commit on the index you are updating if you
>>>>>>> want to control what that last commit level is.
>>>>>>>
>>>>>>> However, in your case if the index gets corrupted due to a disk full
>>>>>>> situation, I don't know what rollback would do, if anything, to help. You
>>>>>>> may need to play with the scenario to see what would happen.
>>>>>>>
>>>>>>> If you are using the DataImportHandler it may handle the rollback for
>>>>>>> you...again, however, it may not deal with disk full situations gracefully
>>>>>>> either.
>>>>>>> --
>>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lance Norskog
>>>>>> [hidden email]
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lance Norskog
>>>> [hidden email]
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> [hidden email]
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
>
> --
> Kirill Zakharenko/Кирилл Захаренко ([hidden email])
> Phone: +7 (495) 683-567-4
> ICQ: 104465785
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]