[jira] Created: (LUCENE-1173) index corruption autoCommit=false

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
index corruption autoCommit=false
---------------------------------

                 Key: LUCENE-1173
                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
    Affects Versions: 2.3
            Reporter: Yonik Seeley
            Priority: Critical


In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated LUCENE-1173:
---------------------------------

    Attachment: indexstress.patch

Attaching a patch that can reproduce.
With autoCommit=true, the test passes. Change it to false and it fails.

The test basic uses multiple threads to update documents.  The last document for any id is kept, and then all these docs are indexed again serially.  The two indicies are them compared.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Priority: Critical
>         Attachments: indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567818#action_12567818 ]

Yonik Seeley commented on LUCENE-1173:
--------------------------------------

Note: if I reduce the test to indexing with a single thread, it still fails.
    Map docs = indexRandom(1, 50, 50, dir1);
The test still does the indexing in a different thread than the close(), so it's not quite a single threaded test.

Another thing to note: all of the terms are matching up (the test succeeds if I don't test the stored fields).


> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Priority: Critical
>         Attachments: indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless reassigned LUCENE-1173:
------------------------------------------

    Assignee: Michael McCandless

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567819#action_12567819 ]

Michael McCandless commented on LUCENE-1173:
--------------------------------------------

Uh oh ... I'll take this!

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Priority: Critical
>         Attachments: indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated LUCENE-1173:
---------------------------------

    Attachment: indexstress.patch

Thanks Mike!

Attaching new version of test that correctly deals with terms with no docs (because of deletions).
Other variations were failing before, now it's just those with autoCommit=false

Note that it's possible to trigger this bug by indexing only 3 documents:
mergeFactor=2; maxBufferedDocs=2; Map docs = indexRandom(1, 3, 2, dir1);

I love random testing:-)

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567849#action_12567849 ]

Michael McCandless commented on LUCENE-1173:
--------------------------------------------

Yes this is one awesome test case :)

Thanks.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-1173:
---------------------------------------

    Attachment: LUCENE-1173.patch

I just sent email to java-user to give a heads up on this.

Attach patch fixes the issue.  All tests pass.

I think we should spin 2.3.1 for this one?

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (LUCENE-1173) index corruption autoCommit=false

Michael Busch
Michael McCandless (JIRA) wrote:

>      [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Michael McCandless updated LUCENE-1173:
> ---------------------------------------
>
>     Attachment: LUCENE-1173.patch
>
> I just sent email to java-user to give a heads up on this.
>
> Attach patch fixes the issue.  All tests pass.
>
> I think we should spin 2.3.1 for this one?
>

+1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Updated: (LUCENE-1173) index corruption autoCommit=false

Michael McCandless-2

OK I'll backport this fix.

I'd also like to backport LUCENE-1168 (another corruption case when  
autoCommit=false) and LUCENE-1171 (deadlock on hitting OOM).

Mike

Michael Busch wrote:

> Michael McCandless (JIRA) wrote:
>>      [ https://issues.apache.org/jira/browse/LUCENE-1173?
>> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>>
>> Michael McCandless updated LUCENE-1173:
>> ---------------------------------------
>>
>>     Attachment: LUCENE-1173.patch
>>
>> I just sent email to java-user to give a heads up on this.
>>
>> Attach patch fixes the issue.  All tests pass.
>>
>> I think we should spin 2.3.1 for this one?
>>
>
> +1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567873#action_12567873 ]

Yonik Seeley commented on LUCENE-1173:
--------------------------------------

Patch looks good (heh... a one liner!)
At least it won't break previously working code since autoCommit=true is the default.  The only risk is people trying out the new setting and not realizing it can break things.
2.3.1 might be nice, but I'll leave to others (who have the actual time to do the work) to decide.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567878#action_12567878 ]

Yonik Seeley commented on LUCENE-1173:
--------------------------------------

Hold up a bit... my random testing may have hit another bug....
testMultiConfig hit an error at some point when I cranked up the iterations... I'm trying to reproduce.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1173) index corruption autoCommit=false

Michael Busch
Yonik Seeley (JIRA) wrote:
>     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567878#action_12567878 ]
>
> Yonik Seeley commented on LUCENE-1173:
> --------------------------------------
>
> Hold up a bit... my random testing may have hit another bug....
> testMultiConfig hit an error at some point when I cranked up the iterations... I'm trying to reproduce.
>

OK, I suggest that we should wait a couple of days before we cut 2.3.1
in case there are more problems. We should backport the patches and
commit them to the 2.3 branch. I'll then end of this week create a 2.3.1
tag, build release artifacts and call a vote. Sounds good?

-Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1173) index corruption autoCommit=false

Michael McCandless-2

Michael Busch wrote:

> OK, I suggest that we should wait a couple of days before we cut 2.3.1
> in case there are more problems. We should backport the patches and
> commit them to the 2.3 branch. I'll then end of this week create a  
> 2.3.1
> tag, build release artifacts and call a vote. Sounds good?

Sounds good.  Thanks Michael!

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568053#action_12568053 ]

Michael McCandless commented on LUCENE-1173:
--------------------------------------------


Backported to 2.3.

{quote}
Patch looks good (heh... a one liner!)
{quote}
Yeah the worst ones always seem to be one-liner fixes.  Sigh.

{quote}
Hold up a bit... my random testing may have hit another bug....
testMultiConfig hit an error at some point when I cranked up the iterations... I'm trying to reproduce.
{quote}
I'll go dig on that one next.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1173) index corruption autoCommit=false

Grant Ingersoll-2
In reply to this post by Michael Busch

On Feb 11, 2008, at 6:49 PM, Michael Busch wrote:

> Yonik Seeley (JIRA) wrote:
>>    [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567878 
>> #action_12567878 ]
>>
>> Yonik Seeley commented on LUCENE-1173:
>> --------------------------------------
>>
>> Hold up a bit... my random testing may have hit another bug....
>> testMultiConfig hit an error at some point when I cranked up the  
>> iterations... I'm trying to reproduce.
>>
>
> OK, I suggest that we should wait a couple of days before we cut 2.3.1
> in case there are more problems. We should backport the patches and
> commit them to the 2.3 branch. I'll then end of this week create a  
> 2.3.1
> tag, build release artifacts and call a vote. Sounds good?

I'd suggest at least a week, as it sounds like we need to put this  
through the wringer a bit more.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (LUCENE-1173) index corruption autoCommit=false

Michael Busch
Grant Ingersoll wrote:
>
>
> I'd suggest at least a week, as it sounds like we need to put this
> through the wringer a bit more.
>

I agree! Shall we add a news item to the website where we list these
known issues and announce that there will be a 2.3.1 release in aprox.
1-2 weeks?

-Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-1173.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.3
                   2.4

Committed to 2.3 & trunk.

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>             Fix For: 2.4, 2.3
>
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1173) index corruption autoCommit=false

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated LUCENE-1173:
-----------------------------

    Fix Version/s:     (was: 2.3)
                   2.3.1

> index corruption autoCommit=false
> ---------------------------------
>
>                 Key: LUCENE-1173
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1173
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.3
>            Reporter: Yonik Seeley
>            Assignee: Michael McCandless
>            Priority: Critical
>             Fix For: 2.3.1, 2.4
>
>         Attachments: indexstress.patch, indexstress.patch, LUCENE-1173.patch
>
>
> In both Lucene 2.3 and trunk, the index becomes corrupted when autoCommit=false

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]