[jira] Created: (LUCENE-2095) Document not guaranteed to be found after write and commit

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
Document not guaranteed to be found after write and commit
----------------------------------------------------------

                 Key: LUCENE-2095
                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
    Affects Versions: 2.9.1, 2.4.1
         Environment: Linux 64bit
            Reporter: Sanne Grinovero


after same email on developer list:
"I developed a stress test to assert that a new document containing a
specific term "X" is always found after a commit on the IndexWriter.
This works most of the time, but it fails under load in rare occasions.

I'm testing with 40 Threads, both with a SerialMergeScheduler and a
ConcurrentMergeScheduler, all sharing a common IndexWriter.
Attached testcase is using a RAMDirectory only, but I verified a
FSDirectory behaves in the same way so I don't believe it's the
Directory implementation or the MergeScheduler.
This test is slow, so I don't consider it a functional or unit test.
It might give false positives: it doesn't always fail, sorry I
couldn't find out how to make it more likely to happen, besides
scheduling it to run for a longer time."

I tested this to affect versions 2.4.1 and 2.9.1;


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanne Grinovero updated LUCENE-2095:
------------------------------------

    Attachment: lucene-stresstest.patch

attaching the testcase, apply to version 2.9.1.
It's slow, please be patient.

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless reassigned LUCENE-2095:
------------------------------------------

    Assignee: Michael McCandless

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782644#action_12782644 ]

Michael McCandless commented on LUCENE-2095:
--------------------------------------------

I can get this to fail fairly quickly, using 2 threads.

It's a thread safety issue of commit itself.  If you only allow 1 thread into commit at a time I believe the issue won't happen.

It has to do with what thread #2 does when it enters commit and thread #1 is already in the process of committing.  In this case thread #2 basically waits for thread #1 to finish and then returns, whereas it should start its own new commit.

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2095:
---------------------------------------

    Fix Version/s: 3.1
                   3.0.1
                   2.9.2

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782659#action_12782659 ]

Jason Rutherglen commented on LUCENE-2095:
------------------------------------------

Why do we allow un-synchronized commit if doFlush is
synchronized? The main cost of commit is most likely in the
flush as it doesn't wait for merges? There's a todo above
doFlush indicating we may want to make it un-synchronized in the
future to not block merges, how tenable is this?

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782715#action_12782715 ]

Michael McCandless commented on LUCENE-2095:
--------------------------------------------

I think the main cost is often fsync'ing the new files.

But I agree we should simply lock so only one thread can be in commit
at once.  Concurrency inside commit (when fscyning) seems silly (it
used to be slightly more interesting when we had autoCommit=true).

We shouldn't use IW's lock.  First, it's overkill (doing so would
unnecessarily block other sync'd ops from running).  Second, it leads
to deadlock, at least in CMS (if it waits because too many merges are
running).  I'll use a separate lock.

I'm not adding locking around the separate calls to prepareCommit then
commit.  An app must ensure these two calls are always balanced.

Making flush unsynchronized would be great but I think wouldn't be
that big a gain in practice, unless we could make it truly unsync'd
even with adding new docs.  Ie, if while we are flushing the last
segment, we can accept adding/deleting docs into a new ram buffer in
DW, that'd be quite interesting.  But that's a big change!


> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2095:
---------------------------------------

    Attachment: LUCENE-2095.patch

Patch attached.  I think it's ready to commit:

  * Added a simple monitor lock during commit() to serialize threads.

  * Boiled down the test case to a new test case in TestIndexWriter.

  * Found & fixed the [unrelated] deadlock in TestCrash, and cutover
    RAMDir's tracking of sizeInBytes to an AtomicLong, as discussed on
    java-dev.


> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783222#action_12783222 ]

Uwe Schindler commented on LUCENE-2095:
---------------------------------------

Your patch only works for trunk and 3.0. If you want to fix it in 2.9 you must remove the j.u.concurrent* class usage.

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783223#action_12783223 ]

Michael McCandless commented on LUCENE-2095:
--------------------------------------------

Thanks for reminding me -- I'll remove it on backport.

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-2095.
----------------------------------------

    Resolution: Fixed

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784974#action_12784974 ]

Michael McCandless commented on LUCENE-2095:
--------------------------------------------

Thanks Sanne!

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785943#action_12785943 ]

Sanne Grinovero commented on LUCENE-2095:
-----------------------------------------

Thanks a lot Michael, this makes my distributed testing reliable again :-)

I see you didn't apply my testcase, do you think it's not needed to have such a test?
If you need I could change it as you wish.

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2095) Document not guaranteed to be found after write and commit

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785965#action_12785965 ]

Michael McCandless commented on LUCENE-2095:
--------------------------------------------

bq. Thanks a lot Michael,

Thank you!

bq. this makes my distributed testing reliable again

I'm glad to hear it :)  Thanks for bringing closure...

bq. I see you didn't apply my testcase

Actually, I did: I boiled your testcase down and added it, in TestIndexWriter.java (testCommitThreadSafety).  The more tests the better!

> Document not guaranteed to be found after write and commit
> ----------------------------------------------------------
>
>                 Key: LUCENE-2095
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2095
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.4.1, 2.9.1
>         Environment: Linux 64bit
>            Reporter: Sanne Grinovero
>            Assignee: Michael McCandless
>             Fix For: 2.9.2, 3.0.1, 3.1
>
>         Attachments: LUCENE-2095.patch, lucene-stresstest.patch
>
>
> after same email on developer list:
> "I developed a stress test to assert that a new document containing a
> specific term "X" is always found after a commit on the IndexWriter.
> This works most of the time, but it fails under load in rare occasions.
> I'm testing with 40 Threads, both with a SerialMergeScheduler and a
> ConcurrentMergeScheduler, all sharing a common IndexWriter.
> Attached testcase is using a RAMDirectory only, but I verified a
> FSDirectory behaves in the same way so I don't believe it's the
> Directory implementation or the MergeScheduler.
> This test is slow, so I don't consider it a functional or unit test.
> It might give false positives: it doesn't always fail, sorry I
> couldn't find out how to make it more likely to happen, besides
> scheduling it to run for a longer time."
> I tested this to affect versions 2.4.1 and 2.9.1;

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]