[jira] Created: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
Intermittent failure in 3.x's backwards TestThreadedOptimize
------------------------------------------------------------

                 Key: LUCENE-2618
                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
            Reporter: Michael McCandless
             Fix For: 3.1, 4.0


Failure looks like this:

{noformat}
    [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
    [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
    [junit] null
    [junit] junit.framework.AssertionFailedError: null
    [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
    [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
    [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
{noformat}

I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902018#action_12902018 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

I'm guessing that this line fails (which is 126 in my most recent checkout):

{code}
      assertTrue(reader.isOptimized());
{code}

Is this the one that's pointed by your code Mike? If so, I suggest we include a message to the assertion, something like "index should be optimized". It's annoying that JUnit does not print "should be true but was false", or something like that, and instead prints 'null', which is more intimidating :).

Perhaps we should also add some more info to the print, like the number of segments in and index and whether there are deletions, so we'd have a better clue why the test failed?

I've tried to run the test a couple of times, but it passed ...

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902023#action_12902023 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

Sorry, I've missed the part about this happening in backwards tests. The line numbers match for me, and I see the assertion messages. I do think though that additional info such as the number of segments and deleted docs would be useful, since reader.isOptimize() will return false if either of these two is wrong.

And we can add the same message to the regular test as well ...

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902550#action_12902550 ]

Mark Miller commented on LUCENE-2618:
-------------------------------------

I'm catching something similar on current tests I think:
{code}
   [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
    [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
    [junit] expected:<248> but was:<256>
    [junit] junit.framework.AssertionFailedError: expected:<248> but was:<256>
    [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:119)
    [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:142)
    [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:380)
    [junit] at org.apache.lucene.util.LuceneTestCase.run(LuceneTestCase.java:372)
    [junit]
    [junit]
    [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.733 sec
{code}

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2618:
---------------------------------------

    Attachment: LUCENE-2618.patch

I think I found this!

After a merge completes, IW then checks w/ the merge policy to see if followon merges are now necessary.

But this check is skipped if IW.close is pending (ie has been called and is waiting for merges to complete).

However, if that merge is an optimize, then we should in fact consult the merge policy even when a close is pending, which we are not doing today.

Tiny patch (attached) should fix it.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923395#action_12923395 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

For education purposes - why should we consult the MP if it's an optimize, even while closing? If close(false) is called after optimize() was called, it means the app would like to abort merges ASAP. If so, why would we consult the MP if we're instructed to abort?

Are you talking about a different use case?

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923426#action_12923426 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

{quote}
If close(false) is called after optimize() was called, it means the app would like to abort merges ASAP. If so, why would we consult the MP if we're instructed to abort?

Are you talking about a different use case?
{quote}

Sorry, different use case.

This use case is you call .optimize(doWait=false) then you call a normal .close() (ie, wait for merges).  In this case we wait for all running merges to finish, but don't start any new ones.  My patch would still allow new ones to start if the merges are due to a running optimize.

Your use case, where .close(false) is called, will in fact abort all running merges and close quickly.  Ie we will not start new merges, even for optimize, if you pass false to close, with this pattch.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923428#action_12923428 ]

Robert Muir commented on LUCENE-2618:
-------------------------------------

thanks for tracking this down...!

I think if we fix this one, then we are really into the long tail of random test fails (at least for now)


> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923430#action_12923430 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

Ok Mike, that makes sense. You want to allow optimize() to finish all possible merges. Why then not let regular merges finish all the way through, even if we're closing? I mean, the application wants to wait for all running merges, so why is optimize() different than maybeMerge()?

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923564#action_12923564 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

We do allow all running merges to run to completion.

But, we don't allow new merges to start, unless it's part of an ongoing optimize (as of this patch).

I think this distinction makes sense?  Since optimize was an explicit call, it should run until completion.  But merging can simply pick up the next time the index is opened?

If an app really wants to allow all merges to run before closing (even new ones starting) it can call waitForMerges and then close.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923579#action_12923579 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

I don't personally mind either way. Just want to point out that calling maybeMerge is as explicit as calling optimize. You can argue for both that if an app wants to wait for merges it can call waitForMerges. In fact, an app calling close() already stated it wants to wait for merges - it's as if it called waitForMerges followed by close.

I think you're trying to distinguish merges that started because the MP decided they should run following a certain commit to those triggered by explicit call to optimize. So IMO maybeMerge and optimize are the same as both were explicitly initiated by the application.

This test fails because it assumes optimize will run to completion. What if the test assumed maybeMerge runs to completion? Isn't that a valid expectation from an application calling close()? We're also distinguishing the first round of merges from subsequent rounds, only when maybeMerge is called, but not optimize...

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923665#action_12923665 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

bq. Just want to point out that calling maybeMerge is as explicit as calling optimize.

But: apps don't normally call maybeMerge?  This is typically called within IW, eg on segment flush.

I mean, it is public so apps can call it, but I expect very few do (vs optimize which apps use alot).  It's the exception not the rule...

I guess I feel that close should try to close quickly -- an app would not expect close to randomly take a long time (it's already bad enough since a large merge could be in process...).   So, allowing other merges to start up, which could easily be large merges since they are follow-on ones, would make that worse.

Alternatively, we could define the semantics of close as being allowed to prevent a running optimize from actually completing?  Then we'd have to change this test, eg to call .waitForMerges before close.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923761#action_12923761 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

Ok - I agree maybeMerge is probably less frequently called than optimize. And perhaps we can look at it that way: when you call optimize, you know exactly what to expect. You control the # of final segments. When you call maybeMerge lucene does not guarantee the final result. Unless you know exactly the state of all the segments in the index (which except than from unit tests I think it's very unlikely) and exactly what your MP is doing, you cannot expect any guaranteed outcome from calling maybeMerge, except for it "doing the best effort".

What bothered me is that even if maybeMerge and optimize may go through several levels of merging following one call to them, one is guaranteed to complete and the other isn't. But since optimize is more common in apps than the other, I'm willing to make that exception. Perhaps then add to maybeMerge docs that if you want to guarantee merges finish when close is called, you should wait for merges? Or should we add it to close?

I'm fine now with this fix. +1 to commit.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923866#action_12923866 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

OK thanks Shai... I'll commit shortly.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2618.
----------------------------------------

    Resolution: Fixed

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Reopened: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler reopened LUCENE-2618:
-----------------------------------


This commit sometimes fails TestBackwards because IndexWriter.close() now also throws IndexFormatTooOldException if the previous call to optimize() have thrown it already.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923946#action_12923946 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

Hmm.... indeed you can repro with:

{noformat}ant test-core -Dtestcase=TestBackwardsCompatibility -Dtestmethod=testUnsupportedOldIndexes -Dtests.seed=-7202471693621265890:9015568443891620555{noformat}

I'll revert until I can figure this out... sorry!

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923987#action_12923987 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

OK so I think we should fix this test to also accept an IndexTooOldExc during close.

The .optimize() call for only the 29.nocfs case (for some reason) enrolls 2 pending merges to IW.

The 1st merge hits an exception, throwing up through the .optimize() to the test.  But the 2nd merge remains queued, and in IW.close() we give MS a chance to run any merges it needs to, and that 2nd merge then also hits an exc.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2618:
---------------------------------------

    Attachment: LUCENE-2618.patch

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923989#action_12923989 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

Does this mean I'll need to catch that exception every time I close an IW, or at least prepare to catch it? If so, shouldn't we document it? Is it only relevant to the test?

Somehow this change / fix starts to get complicated. Can IW swallow those exceptions internally, and relieve the application from all this? When I close(false), I should be prepared to hit MergeAbortedException, it's kinda part of the API contract. But when I close(true), why do I need to be prepared to handle any exception, except for real IO ones?

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12