[jira] Created: (LUCENE-2536) Rollback doesn't preserve integrity of original index

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
Rollback doesn't preserve integrity of original index
-----------------------------------------------------

                 Key: LUCENE-2536
                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
    Affects Versions: 3.0.2
         Environment: Windows XP pro
            Reporter: Mark Harwood
         Attachments: TestRollback.java

After several "updateDocuments" calls a rollback call does not return the index to the prior state.
This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.

Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2

JUnit to follow.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Harwood updated LUCENE-2536:
---------------------------------

    Attachment: TestRollback.java

Example JUnit

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>         Attachments: TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887757#action_12887757 ]

Mark Harwood commented on LUCENE-2536:
--------------------------------------

Looks like on the "close" call IndexWriter is flushing a superfluous ".del" file of buffered deletes.


> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>         Attachments: TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shai Erera updated LUCENE-2536:
-------------------------------

    Attachment: TestRollback.java

It did not fail for me on 3.1, however did fail on 3.0.1.

I'm attaching a version of this test, which creates far less docs (5, instead of 10,000) and uses a RAMDirectory). The test still fails, which is expected. I assume if we want to make that an official unit test, the test case which passes should go away? (or not)

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>         Attachments: TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless reassigned LUCENE-2536:
------------------------------------------

    Assignee: Michael McCandless

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887775#action_12887775 ]

Michael McCandless commented on LUCENE-2536:
--------------------------------------------

Duh -- in DocumentsWriter.abort, right after deletesInRAM.clear() we must add deletesFlush.clear().  So, indeed, any deletions pending (either due to .updateDoc or .deleteDocs) when segments were flushed, fail to be cleared and are then applied on close.

How awful!

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2536:
---------------------------------------

    Attachment: LUCENE-2536.patch

Attached patch -- started from Shai's patch, tweaked it a bit, added the one line fix & CHANGES entry.

I see the test in fact failing on 3.x.

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: LUCENE-2536.patch, TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887793#action_12887793 ]

Shai Erera commented on LUCENE-2536:
------------------------------------

Looks good to me. I think now that the test class contains just this method, we can consolidate all the helper ones into 1? It was required before 'cause there were two test cases. And we can remove the first assert in the test, since we're not checking here that the index contains the number of added docs after it was committed - there are plenty of other tests that do that.

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: LUCENE-2536.patch, TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2536:
---------------------------------------

    Attachment: LUCENE-2536.patch

Thanks Shai -- new patch.

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: LUCENE-2536.patch, LUCENE-2536.patch, TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887813#action_12887813 ]

Mark Harwood commented on LUCENE-2536:
--------------------------------------

Thanks for the rapid response Mike/Shai.

I'll need to patch a couple of production systems using 2.4.0 and 2.9.1 with this.
For the record -  with this same change applied, the test passes OK on these older versions too.

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>         Attachments: LUCENE-2536.patch, LUCENE-2536.patch, TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-2536) Rollback doesn't preserve integrity of original index

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2536.
----------------------------------------

    Fix Version/s: 2.4.2
                   2.9.4
                   3.0.3
                   3.1
                   4.0
       Resolution: Fixed

Thanks Mark -- what a doozie!

> Rollback doesn't preserve integrity of original index
> -----------------------------------------------------
>
>                 Key: LUCENE-2536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2536
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.2
>         Environment: Windows XP pro
>            Reporter: Mark Harwood
>            Assignee: Michael McCandless
>             Fix For: 2.4.2, 2.9.4, 3.0.3, 3.1, 4.0
>
>         Attachments: LUCENE-2536.patch, LUCENE-2536.patch, TestRollback.java, TestRollback.java
>
>
> After several "updateDocuments" calls a rollback call does not return the index to the prior state.
> This seems to occur if the number of updates exceeds the RAM buffer size i.e. when some flushing of updates occurs.
> Test fails in Lucene 2.4, 2.9, 3.0.1 and 3.0.2
> JUnit to follow.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]