[jira] Created: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923995#action_12923995 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

bq. Does this mean I'll need to catch that exception every time I close an IW, or at least prepare to catch it?

Well, IndexFormatTooOldExc subclasses IOE... but, yes, if there's a risk you'll open a too-old index, you should try to handle this.

IW.close does alot... flush the last segment, let MS run any pending merges, do commit, delete now-not-need files, etc.  So there's plenty of chances for interested excs.

bq. Is it only relevant to the test?

Well, this test opens an IW on a too-old index... so if your app may do that....

bq. Can IW swallow those exceptions internally, and relieve the application from all this?

Whoa no way!

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924002#action_12924002 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

I see. It's just that you describe this exception as being thrown because close is called while optimize was running over an old index - but I don't understand why it has to be thrown in this case - namely, what's the connection between optimize + close and an old index? If my app knowingly opened a too old index, would it get this exception always, if it will call optimize followed by close? Or is it a special scenario hit by the test?

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924006#action_12924006 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

bq.  If my app knowingly opened a too old index, would it get this exception always, if it will call optimize followed by close? Or is it a special scenario hit by the test?

Not always.  It's only if the MP registered more than 1 merge for the optimize, and, you're using SMS.

But, really if your app has risk of opening a too-old index, it should be prepared for this exc...

bq. namely, what's the connection between optimize + close and an old index?

MP enrolled 2 merges for the optimize... the first one hits exc... then test calls close... and close lets MS run... and MS is SMS... and it runs the 2nd merge, which this the exc.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924008#action_12924008 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

I think there's a separate issue open (Uwe?) to have IW immediately throw this exc on open, instead of during optimize/close.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924150#action_12924150 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

OK Mike .I understood the sequence of operations that led to this exception before. What didn't add up is why is it thrown during optimize, and not say up front when IW is opened, or when the Directory was added through addIndexes.

We should fix the code to throw the exception immediately. Is there a way to check a Directory if it's old or not? If not, such exception could really throw you off your chair, when you hit it at a point in time not remotely related to when it was added to the index.

I don't mind if you continue w/ the fix to the test as you did, but IMO it just hides the real problem. I.e., allowing all merges caused by optimize() to finish is a correct fix. But catching that exception upon IW.close() is a bad one IMO - people who read the code learn how to use Lucene, and catching that exception on close() makes absolutely no sense, at least to me. Could you plz add a TODO there to get rid of that code when we fix IW to detect old indexes up front? That way, if someone reads the code, he'll at least understand that this is a temporary solution.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924163#action_12924163 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

bq. We should fix the code to throw the exception immediately. Is there a way to check a Directory if it's old or not?

I agree -- IW.open should fail immediately if any of the segments are too old.

Unfortunately, I don't see a simple way to do this.  We can't just look at the version of the segments_N file, for example, because one segment could be from 2.9, and [say] 3.1 had last opened the index and written the 3.x file format for segments_N.  See, IW does not go and open all SegmentReaders on open.  It's only on merge, applying deletes, or opening an NRT reader, that we go and open segments for reading.

I think to do this correctly we should modify segments_N format to record the oldest segment in the index?  Then IW can check this easily on open.

bq. I don't mind if you continue w/ the fix to the test as you did, but IMO it just hides the real problem. I.e., allowing all merges caused by optimize() to finish is a correct fix.

I agree.

There is already a pre-existing TODO in the test stating that we should fix IW to throw this exc on open.  I'll also add a TODO to IW's ctor and go open an issue...

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924164#action_12924164 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

OK I opened LUCENE-2720.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924167#action_12924167 ]

Shai Erera commented on LUCENE-2618:
------------------------------------

Thanks Mike.

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2618) Intermittent failure in 3.x's backwards TestThreadedOptimize

kuladeep (Jira)
In reply to this post by kuladeep (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924298#action_12924298 ]

Michael McCandless commented on LUCENE-2618:
--------------------------------------------

Ugh -- last night's 3.x build just failed again!  So this was not the [only] cause.  Hmm.  I'll leave this reopened....

> Intermittent failure in 3.x's backwards TestThreadedOptimize
> ------------------------------------------------------------
>
>                 Key: LUCENE-2618
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2618
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>             Fix For: 3.1, 4.0
>
>         Attachments: LUCENE-2618.patch, LUCENE-2618.patch
>
>
> Failure looks like this:
> {noformat}
>     [junit] Testsuite: org.apache.lucene.index.TestThreadedOptimize
>     [junit] Testcase: testThreadedOptimize(org.apache.lucene.index.TestThreadedOptimize): FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError: null
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.runTest(TestThreadedOptimize.java:125)
>     [junit] at org.apache.lucene.index.TestThreadedOptimize.testThreadedOptimize(TestThreadedOptimize.java:149)
>     [junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:253)
> {noformat}
> I just committed some verbosity so next time it strikes we'll have more details.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12