[jira] Created: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
TestIndexModifier.testIndexWithThreads is not valid?
----------------------------------------------------

         Key: LUCENE-442
         URL: http://issues.apache.org/jira/browse/LUCENE-442
     Project: Lucene - Java
        Type: Bug
  Components: Search  
    Versions: 1.9    
    Reporter: Hoss Man


I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.

The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.

Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
   a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
   b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.

If I'm understnading that correctly, I see no reason to assume this test will pass.  
It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
   1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
   2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
   3) the IndexThread instances from #2 do nothing but add documents
...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.

Am I missing something, or should this test be removed?



--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12344241 ]

Daniel Naber commented on LUCENE-442:
-------------------------------------

Hoss,

how exactly did the test fail, what was the real number and what was the expected number?

I must admit I don't understand your scenario. The test happens only at the end, on an optimized index. We start two threads on an empty index. Both threads randomly add and delete documents and remember the number of documents added and deleted. At the end, when both threads are finished, the index should contain this many documents:

# added by thread 1 + # added by thread 2 - # deleted by thread 1 - # deleted by thread 2

This is how I understand it. Maybe you could rephrase your scenario?

Regards
 Daniel


> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12344567 ]

Daniel Naber commented on LUCENE-442:
-------------------------------------

Okay, I see now that there are problems with the test. I will try to fix them, maybe that way it becomes clear (to me) if that test case makes sense at all.


> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12349910 ]

Yonik Seeley commented on LUCENE-442:
-------------------------------------

I see a few issues:

1) A constant seed is used for reproducability:
  private Random random = new Random(101); // constant seed for reproducability
But, it won't have that effect since the same random number generator is used across threads.

2) race condition between checking idStack.size() and calling pop()

3) non-atomic increment of the document id could lead to duplicates

IMO, the ideal multithreaded test would be designed to not use any synchronization at all, making it easier to test if IndexModifier actually does all the synchronization it needs to.

> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12355652 ]

Daniel Naber commented on LUCENE-442:
-------------------------------------

I just committed some updates, could you please check whether it's okay now?

> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12355911 ]

Hoss Man commented on LUCENE-442:
---------------------------------

Daniel: I used the following script to run through 50 iterations of the test...

#!/bin/sh
for ((i=0; i < 50; i++)); do {
  ant test -Dtestcase=TestIndexModifier;
  mv build/test/TEST-org.apache.lucene.index.TestIndexModifier.xml build/test/TEST-org.apache.lucene.index.TestIndexModifier.xml.$i;
};
done
grep -l "failure message" build/test/TEST-org.apache.lucene.index.TestIndexModifier.xml.*

...prior to upgrading to the changes you made, 30 of the 50 test runs failed (with the values being off by one or two in each case).  After upgrading to TestIndexModifier.java v328151, 0 of the 50 runs failed.  

It's not a proof of correctness, but it certainly bodes well.

I also want to clarify the scenerio I described earlier: I'm an idiot.  I completely missed that each call to testIndexInternal(int) blocks waiting for the threads to finish -- for some reason I thought all three calls were happening concurrently.  So the types of scenerios I was initially thinking could cause problems were completely non-existent -- and it looks like you fixed all the ones Yonik noticed.

(I do think Yonik has a good point however regarding a test with no synchronization at all -- this should be possible if each instant of IndexThread maintaining it's own id and idStack variables, using  'threadNumber + "-" id++' to generate new doc identifiers, and only ever deleted documents it added).

> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/LUCENE-442?page=comments#action_12355988 ]

Yonik Seeley commented on LUCENE-442:
-------------------------------------

Yep, it looks good.  I'd go ahead and close this issue (if I knew how...)

The multithreaded tests w/o any synchronization is just a good general guideline.  I briefly browsed the TestIndexModifier, and there shouldn't be any issues in that regard since  the synchronization is trivial (all public methods synchronized on the same object)

> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (LUCENE-442) TestIndexModifier.testIndexWithThreads is not valid?

Hudson (Jira)
In reply to this post by Hudson (Jira)
     [ http://issues.apache.org/jira/browse/LUCENE-442?page=all ]
     
Daniel Naber closed LUCENE-442:
-------------------------------

    Fix Version: 1.9
     Resolution: Fixed

Yonik, thanks for your review. I'm closing this issue now. You'll need to ask Erik to give you committer's permissions in Jira, then you'll be able to close issues with the "Workflow actions" on the left.


> TestIndexModifier.testIndexWithThreads is not valid?
> ----------------------------------------------------
>
>          Key: LUCENE-442
>          URL: http://issues.apache.org/jira/browse/LUCENE-442
>      Project: Lucene - Java
>         Type: Bug
>   Components: Search
>     Versions: 1.9
>     Reporter: Hoss Man
>      Fix For: 1.9

>
> I recently started playing with the trunk of SVN, and noticed that intermitently, TestIndexModifier.testIndexWithThreads (revision 292010) would fail.
> The basic premise of the test seems to be that 3 pairs of IndexThread instances can be started in parallel, each pair using the same instance of IndexModifier to concurrently and randomly add/delete/optimize a single FSDirectory index.  
> The test is considered a success if the sum of additions-deletions recorded by each pair of threads equals the final docCount() for the IndexModifier instance used by that pair of threads.
> Now I freely admit that I'm not 100% familiar with the code for IndexModifier, but at a glance, the basic premise seems to be:
>    a) If method for IndexWriter is called, open it if needed, close the IndexReader first if needed.
>    b) if method for IndexReader is called, open it if needed, close the IndexWriter first if needed.
> If I'm understnading that correctly, I see no reason to assume this test will pass.  
> It seems like there could be plenty of scenerios in which the number of additions-deletions != docCount(). The most trivial example I can think of is:
>    1) the first IndexThread instance which has a chance to run adds a document, and optimizes before any other IndexThreads ever open the Directory.
>    2) a subsequent pair of IndexThread instances open their IndexModifier instance before any documents are deleted.
>    3) the IndexThread instances from #2 do nothing but add documents
> ...that pair of IndexThreads is now garunteed to have recorded a differnet number of additions then the docCount returned by their IndexModifier.
> Am I missing something, or should this test be removed?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]