gradle + beasting

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

gradle + beasting

Michael Sokolov-4
Hi, I've been using gradlew a lot last few days, and first I want to
give a huge thanks to those who did the work to make this possible.
I'm not naming names since I probably don't know everyone who chipped
in, but this is really great. I'm especially loving the `gradlew
tasks` and the excellent documentation for the tasks!

Now my beasting beef :) I did some beasting and got a bunch of
failures, and it looks as if they were all run with the same seed;; eg
here's some output from the test showing (I think) two failures with
the same seed. I had 43 failures out of 1400, all with the same (first
part of) the random seed

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII


I tried poking around in randomization.gradle to see if I could verify
and/or fix, but I'm not sure how the tests.iters mechanism works - is
it built into gradle? Something we added on top?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gradle + beasting

Dawid Weiss-2

Short because I am on a phone only. The first part of the seed.is the "static" part that applies to static context - constructors, beforeclass etc. The second part would apply to methods. Test.iters is a.property recognized by the runner (so it works from the ide etc.) - if your random failure is at method level then the second part is relevant. 

Gradle build currently does not have "beasting" in the form of different static seeds. This would have to be implemented as a series of recursive calls with different master seeds...
 Complicated but not impossible.



On Wed, Feb 5, 2020, 18:49 Michael Sokolov <[hidden email]> wrote:
Hi, I've been using gradlew a lot last few days, and first I want to
give a huge thanks to those who did the work to make this possible.
I'm not naming names since I probably don't know everyone who chipped
in, but this is really great. I'm especially loving the `gradlew
tasks` and the excellent documentation for the tasks!

Now my beasting beef :) I did some beasting and got a bunch of
failures, and it looks as if they were all run with the same seed;; eg
here's some output from the test showing (I think) two failures with
the same seed. I had 43 failures out of 1400, all with the same (first
part of) the random seed

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII


I tried poking around in randomization.gradle to see if I could verify
and/or fix, but I'm not sure how the tests.iters mechanism works - is
it built into gradle? Something we added on top?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gradle + beasting

Dawid Weiss-2

In case I was not clear: in case of most tests the tests.iters _is_ the right way to repeat the same test with different seeds. The only difference is when you would like to randomize static stuff - then tests.iters would still apply only to tests, not the static test context ( which would remain the same for all repetitions). 


On Wed, Feb 5, 2020, 19:23 Dawid Weiss <[hidden email]> wrote:

Short because I am on a phone only. The first part of the seed.is the "static" part that applies to static context - constructors, beforeclass etc. The second part would apply to methods. Test.iters is a.property recognized by the runner (so it works from the ide etc.) - if your random failure is at method level then the second part is relevant. 

Gradle build currently does not have "beasting" in the form of different static seeds. This would have to be implemented as a series of recursive calls with different master seeds...
 Complicated but not impossible.



On Wed, Feb 5, 2020, 18:49 Michael Sokolov <[hidden email]> wrote:
Hi, I've been using gradlew a lot last few days, and first I want to
give a huge thanks to those who did the work to make this possible.
I'm not naming names since I probably don't know everyone who chipped
in, but this is really great. I'm especially loving the `gradlew
tasks` and the excellent documentation for the tasks!

Now my beasting beef :) I did some beasting and got a bunch of
failures, and it looks as if they were all run with the same seed;; eg
here's some output from the test showing (I think) two failures with
the same seed. I had 43 failures out of 1400, all with the same (first
part of) the random seed

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII

  - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
{seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
    Test output:
/home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
    Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.search.TestTopFieldCollector"
-Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
-Ptests.file.encoding=US-ASCII


I tried poking around in randomization.gradle to see if I could verify
and/or fix, but I'm not sure how the tests.iters mechanism works - is
it built into gradle? Something we added on top?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gradle + beasting

Michael Sokolov-4
ah thanks I never did understand the split seed before. I'm still a
little confused because I had previously thought that to reproduce a
test failure I would define tests.seed= to be the first part of the
seed (like it says in the repro line), but if I would still get a
different seed applied to calls to random() within the test method,
then I don't see how that works to make tests reproducible? Maybe I
was just wrong, but it always seemed to work that way - I would often,
usually, be able to repro random failures using only the first half of
the seed.

On Wed, Feb 5, 2020 at 1:27 PM Dawid Weiss <[hidden email]> wrote:

>
>
> In case I was not clear: in case of most tests the tests.iters _is_ the right way to repeat the same test with different seeds. The only difference is when you would like to randomize static stuff - then tests.iters would still apply only to tests, not the static test context ( which would remain the same for all repetitions).
>
>
> On Wed, Feb 5, 2020, 19:23 Dawid Weiss <[hidden email]> wrote:
>>
>>
>> Short because I am on a phone only. The first part of the seed.is the "static" part that applies to static context - constructors, beforeclass etc. The second part would apply to methods. Test.iters is a.property recognized by the runner (so it works from the ide etc.) - if your random failure is at method level then the second part is relevant.
>>
>> Gradle build currently does not have "beasting" in the form of different static seeds. This would have to be implemented as a series of recursive calls with different master seeds...
>>  Complicated but not impossible.
>>
>>
>>
>> On Wed, Feb 5, 2020, 18:49 Michael Sokolov <[hidden email]> wrote:
>>>
>>> Hi, I've been using gradlew a lot last few days, and first I want to
>>> give a huge thanks to those who did the work to make this possible.
>>> I'm not naming names since I probably don't know everyone who chipped
>>> in, but this is really great. I'm especially loving the `gradlew
>>> tasks` and the excellent documentation for the tasks!
>>>
>>> Now my beasting beef :) I did some beasting and got a bunch of
>>> failures, and it looks as if they were all run with the same seed;; eg
>>> here's some output from the test showing (I think) two failures with
>>> the same seed. I had 43 failures out of 1400, all with the same (first
>>> part of) the random seed
>>>
>>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>>> {seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
>>>     Test output:
>>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>>>     Reproduce with: gradlew :lucene:core:test --tests
>>> "org.apache.lucene.search.TestTopFieldCollector"
>>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>>> -Ptests.file.encoding=US-ASCII
>>>
>>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>>> {seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
>>>     Test output:
>>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>>>     Reproduce with: gradlew :lucene:core:test --tests
>>> "org.apache.lucene.search.TestTopFieldCollector"
>>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>>> -Ptests.file.encoding=US-ASCII
>>>
>>>
>>> I tried poking around in randomization.gradle to see if I could verify
>>> and/or fix, but I'm not sure how the tests.iters mechanism works - is
>>> it built into gradle? Something we added on top?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gradle + beasting

Dawid Weiss-2

The second part of the seed is derived from the first (and iteration number, test name etc) so typically the first would be sufficient. 

D.

On Wed, Feb 5, 2020, 19:59 Michael Sokolov <[hidden email]> wrote:
ah thanks I never did understand the split seed before. I'm still a
little confused because I had previously thought that to reproduce a
test failure I would define tests.seed= to be the first part of the
seed (like it says in the repro line), but if I would still get a
different seed applied to calls to random() within the test method,
then I don't see how that works to make tests reproducible? Maybe I
was just wrong, but it always seemed to work that way - I would often,
usually, be able to repro random failures using only the first half of
the seed.

On Wed, Feb 5, 2020 at 1:27 PM Dawid Weiss <[hidden email]> wrote:
>
>
> In case I was not clear: in case of most tests the tests.iters _is_ the right way to repeat the same test with different seeds. The only difference is when you would like to randomize static stuff - then tests.iters would still apply only to tests, not the static test context ( which would remain the same for all repetitions).
>
>
> On Wed, Feb 5, 2020, 19:23 Dawid Weiss <[hidden email]> wrote:
>>
>>
>> Short because I am on a phone only. The first part of the seed.is the "static" part that applies to static context - constructors, beforeclass etc. The second part would apply to methods. Test.iters is a.property recognized by the runner (so it works from the ide etc.) - if your random failure is at method level then the second part is relevant.
>>
>> Gradle build currently does not have "beasting" in the form of different static seeds. This would have to be implemented as a series of recursive calls with different master seeds...
>>  Complicated but not impossible.
>>
>>
>>
>> On Wed, Feb 5, 2020, 18:49 Michael Sokolov <[hidden email]> wrote:
>>>
>>> Hi, I've been using gradlew a lot last few days, and first I want to
>>> give a huge thanks to those who did the work to make this possible.
>>> I'm not naming names since I probably don't know everyone who chipped
>>> in, but this is really great. I'm especially loving the `gradlew
>>> tasks` and the excellent documentation for the tasks!
>>>
>>> Now my beasting beef :) I did some beasting and got a bunch of
>>> failures, and it looks as if they were all run with the same seed;; eg
>>> here's some output from the test showing (I think) two failures with
>>> the same seed. I had 43 failures out of 1400, all with the same (first
>>> part of) the random seed
>>>
>>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>>> {seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
>>>     Test output:
>>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>>>     Reproduce with: gradlew :lucene:core:test --tests
>>> "org.apache.lucene.search.TestTopFieldCollector"
>>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>>> -Ptests.file.encoding=US-ASCII
>>>
>>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>>> {seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
>>>     Test output:
>>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>>>     Reproduce with: gradlew :lucene:core:test --tests
>>> "org.apache.lucene.search.TestTopFieldCollector"
>>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>>> -Ptests.file.encoding=US-ASCII
>>>
>>>
>>> I tried poking around in randomization.gradle to see if I could verify
>>> and/or fix, but I'm not sure how the tests.iters mechanism works - is
>>> it built into gradle? Something we added on top?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: gradle + beasting

Michael Sokolov-4
Ahh. Well in this case, I did need the full seed to repro for whatever
reason, then I easily found my small bug. Thanks for the help!

On Wed, Feb 5, 2020 at 3:59 PM Dawid Weiss <[hidden email]> wrote:

>
>
> The second part of the seed is derived from the first (and iteration number, test name etc) so typically the first would be sufficient.
>
> D.
>
> On Wed, Feb 5, 2020, 19:59 Michael Sokolov <[hidden email]> wrote:
>>
>> ah thanks I never did understand the split seed before. I'm still a
>> little confused because I had previously thought that to reproduce a
>> test failure I would define tests.seed= to be the first part of the
>> seed (like it says in the repro line), but if I would still get a
>> different seed applied to calls to random() within the test method,
>> then I don't see how that works to make tests reproducible? Maybe I
>> was just wrong, but it always seemed to work that way - I would often,
>> usually, be able to repro random failures using only the first half of
>> the seed.
>>
>> On Wed, Feb 5, 2020 at 1:27 PM Dawid Weiss <[hidden email]> wrote:
>> >
>> >
>> > In case I was not clear: in case of most tests the tests.iters _is_ the right way to repeat the same test with different seeds. The only difference is when you would like to randomize static stuff - then tests.iters would still apply only to tests, not the static test context ( which would remain the same for all repetitions).
>> >
>> >
>> > On Wed, Feb 5, 2020, 19:23 Dawid Weiss <[hidden email]> wrote:
>> >>
>> >>
>> >> Short because I am on a phone only. The first part of the seed.is the "static" part that applies to static context - constructors, beforeclass etc. The second part would apply to methods. Test.iters is a.property recognized by the runner (so it works from the ide etc.) - if your random failure is at method level then the second part is relevant.
>> >>
>> >> Gradle build currently does not have "beasting" in the form of different static seeds. This would have to be implemented as a series of recursive calls with different master seeds...
>> >>  Complicated but not impossible.
>> >>
>> >>
>> >>
>> >> On Wed, Feb 5, 2020, 18:49 Michael Sokolov <[hidden email]> wrote:
>> >>>
>> >>> Hi, I've been using gradlew a lot last few days, and first I want to
>> >>> give a huge thanks to those who did the work to make this possible.
>> >>> I'm not naming names since I probably don't know everyone who chipped
>> >>> in, but this is really great. I'm especially loving the `gradlew
>> >>> tasks` and the excellent documentation for the tasks!
>> >>>
>> >>> Now my beasting beef :) I did some beasting and got a bunch of
>> >>> failures, and it looks as if they were all run with the same seed;; eg
>> >>> here's some output from the test showing (I think) two failures with
>> >>> the same seed. I had 43 failures out of 1400, all with the same (first
>> >>> part of) the random seed
>> >>>
>> >>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>> >>> {seed=[C95A2DAEAA6E2219:1FAD1846BF530502]} (:lucene:core)
>> >>>     Test output:
>> >>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>> >>>     Reproduce with: gradlew :lucene:core:test --tests
>> >>> "org.apache.lucene.search.TestTopFieldCollector"
>> >>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>> >>> -Ptests.file.encoding=US-ASCII
>> >>>
>> >>>   - org.apache.lucene.search.TestTopFieldCollector.testRandomMaxScoreTermination
>> >>> {seed=[C95A2DAEAA6E2219:11046F532841C712]} (:lucene:core)
>> >>>     Test output:
>> >>> /home/ANT.AMAZON.COM/sokolovm/workspace/lucene-solr/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.search.TestTopFieldCollector.txt
>> >>>     Reproduce with: gradlew :lucene:core:test --tests
>> >>> "org.apache.lucene.search.TestTopFieldCollector"
>> >>> -Ptests.seed=C95A2DAEAA6E2219 -Ptests.iters=100
>> >>> -Ptests.file.encoding=US-ASCII
>> >>>
>> >>>
>> >>> I tried poking around in randomization.gradle to see if I could verify
>> >>> and/or fix, but I'm not sure how the tests.iters mechanism works - is
>> >>> it built into gradle? Something we added on top?
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [hidden email]
>> >>> For additional commands, e-mail: [hidden email]
>> >>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]