[Important] New test framework, differences, features, etc.

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Important] New test framework, differences, features, etc.

Dawid Weiss-2
Ah... such a relief to finally have it committed even if this means
the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene
and make it available as a separate project for reuse in other
projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for
randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
with an aggregation of events, balancing, logs, different (from
standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on
github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
RandomizedRunner. It also modifies build scripts to run junit4-ant
instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is
gone. This change was motivated by the fact that the field's value was
read from places where it shouldn't be read, passed to places where it
shouldn't be passed, etc. Instead, the Random instance for a given
scope (see below) can be acquired from a static method in
LuceneTestCase called random(). In the essence, you can just add
brackets around your previous random field references and it _should_
work out of the box. There are differences though: Random object
returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random
instance in a static scope (@BeforeClass) to a field and reusing it in
a test, passing a Random instance from one thread to another, saving a
Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the
scope (test method, hook) it was acquired for. You _can_ call random()
from non-test threads -- they will be given their own thread-local
Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master
seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed
to 'tests.iters' (you'll get a build failure and a message if you try
to use the former one). I could add a fallback but I'd rather not
introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a
duplicate of a given test (or tests), including any @Before/@After
hooks and setups. In theory, this means it is now possible to
reiterate ANY test, no matter how complex. If it doesn't depend on
static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3
-Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in
TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each
executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three
times, but they all count individually (so the reported total is 9
tests). What's also clearly seen is that the master seed is constant
for all tests but each repetition gets a (predictable) derivative of
the random seed. This way you can reiterate N times over a test, each
time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3
-Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2 seed=[DEADBEEF:CAFEBABE]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s

We now fixed both the master and per-method seeds. Every repetition
will have the same randomness. Test failures will report test error
seeds as "master:test" so you will be able to use both scenarios,
depending on the use case.

- "tests.iter.min" has been temporarily removed. I will think of a way
to reintegrate it.

- The console output (sysout/systerrs) is only emitted to ANT's output
if a test/suite fails. If a test/suite passes, its output is
redirected in full to "tests-report.txt" file for inspection if
needed.

3) WHAT ARE THE OTHER GOODIES?

There are many... For one thing, the console output will be much less
chatty. Typically, one will see something like this:

   [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
   [junit4] Expected execution time on JVM J0:    93.93s
   [junit4] Expected execution time on JVM J1:    93.93s
   [junit4] Expected execution time on JVM J2:    93.93s
   [junit4] Expected execution time on JVM J3:    93.93s
   [junit4] Executing 296 suites with 4 JVMs.
   [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
   [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.31s
   [junit4]
   [junit4] Running org.apache.lucene.index.TestStressIndexing2
   [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.89s
...

The number of JVMs used depends on your number of cores (and is not
equal to their number to leave some room for infrastructure). ant
test-help contains instructions how to override the number of JVMs to
make the executions sequential, for example.

Another thing is that tests (their suite, output, etc.) that _failed_
are appended to tests-failures.txt. I remember that somebody once
complained that it's impossible to find a test failure in ANT's log.
Now it should be easier. I made a test fail; the output is like this
(console):

   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at
__randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:0)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
   [junit4]    >        at
org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorerFreqs.java:156)
....[snip]
   [junit4]    >
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.class=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A
-Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.filter=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A
-Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
   [junit4]    > (@AfterClass output)
   [junit4]   2> NOTE: test params are: codec=Lucene40:
{f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da,
timezone=Asia/Bishkek
   [junit4]   2> NOTE: all tests run in this JVM:
   [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
(64-bit)/cpus=8,threads=1,free=97723464,total=128647168
   [junit4]   2>
   [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
Time:  0.45s <<< FAILURES!

Note "<<< FAILURES!" -- this is something to grep for. The " >" marker
is a stack trace from the throwable, "2>" is standard error, "1>" is
standard output. The same error is also appended to
tests-failures.txt. The stack trace contains a synthetic entry with
the master seed and the test's seed.

For those who prefer more visual (color) reports -- look for
"tests-report-{module-name}" folder under the build folder. This
contains index.html with a test report that should be easier to
navigate around. An example of what such a report looks like is here:

http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-Test-Report/index.html

Yet another thing is that test suites are load-balanced across forked
JVMs (with job stealing in place). For the initial balancing there are
precomputed statistics (which should be updated from time to time!).
Any performance improvement this yields is currently eaten away by the
cost of random() calls from tight loops in certain tests (it's better
to just cache the result in a local variable). I'll try to pinpoint
these later on. The timings shouldn't be much different to what it was
before so it's not a performance drop.

I don't know what else... I'm sure there are things that I haven't
covered. Ask if something is not clear or something is not working.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Yonik Seeley-2-2
Great stuff Dawid!
The tests.iters functionality sounds like it will be particularly useful.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10



On Sun, Apr 15, 2012 at 10:45 AM, Dawid Weiss <[hidden email]> wrote:

> Ah... such a relief to finally have it committed even if this means
> the work has just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene
> and make it available as a separate project for reuse in other
> projects. This kind of grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for
> randomization and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
> with an aggregation of events, balancing, logs, different (from
> standard ANT) reporting and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on
> github here: https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant
> instead of ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is
> gone. This change was motivated by the fact that the field's value was
> read from places where it shouldn't be read, passed to places where it
> shouldn't be passed, etc. Instead, the Random instance for a given
> scope (see below) can be acquired from a static method in
> LuceneTestCase called random(). In the essence, you can just add
> brackets around your previous random field references and it _should_
> work out of the box. There are differences though: Random object
> returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random
> instance in a static scope (@BeforeClass) to a field and reusing it in
> a test, passing a Random instance from one thread to another, saving a
> Random instance to a field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the
> scope (test method, hook) it was acquired for. You _can_ call random()
> from non-test threads -- they will be given their own thread-local
> Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master
> seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed
> to 'tests.iters' (you'll get a build failure and a message if you try
> to use the former one). I could add a fallback but I'd rather not
> introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a
> duplicate of a given test (or tests), including any @Before/@After
> hooks and setups. In theory, this means it is now possible to
> reiterate ANY test, no matter how complex. If it doesn't depend on
> static state, it can be repeated. This also links to how seed is used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3
> -Dtestcase=TestSubScorerFreqs test-core
>
> the above will result in 3 executions of every test in
> TestSubScorerFreqs. What you'll get on the console is:
>
>   [junit4] <JUnit4> says hello. Random seed: deadbeef
>   [junit4] Expected execution time on JVM J0:     0.02s
>   [junit4] Executing 1 suite with 1 JVM.
>   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
> Time:  0.45s
>   [junit4]
>   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>   [junit4] Execution time total: 0.98 sec.
>   [junit4] Tests summary: 1 suite, 9 tests
>
> and if you peek at the log file with test results
> (build/core/test/tests-report.txt) you'll see the details of each
> executed test:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
> seed=[DEADBEEF:BAE943E3CC27A0F]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
> seed=[DEADBEEF:BFF828C20800B123]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
> seed=[DEADBEEF:3111BE1E59C4F9E8]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:3DDFB93BCC712A41]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:898905C7F8B3E16D]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:760931BA977A9A6]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
> seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
> seed=[DEADBEEF:13FB542229750127]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
> seed=[DEADBEEF:9D12C2FE78B149EC]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s
>
> Note that tests.iters=3 resulted in every test case executed three
> times, but they all count individually (so the reported total is 9
> tests). What's also clearly seen is that the master seed is constant
> for all tests but each repetition gets a (predictable) derivative of
> the random seed. This way you can reiterate N times over a test, each
> time with a different seed. Now, compare this to:
>
> ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3
> -Dtestcase=TestSubScorerFreqs test-core
>
> The log file now will be:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0 seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1 seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2 seed=[DEADBEEF:CAFEBABE]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s
>
> We now fixed both the master and per-method seeds. Every repetition
> will have the same randomness. Test failures will report test error
> seeds as "master:test" so you will be able to use both scenarios,
> depending on the use case.
>
> - "tests.iter.min" has been temporarily removed. I will think of a way
> to reintegrate it.
>
> - The console output (sysout/systerrs) is only emitted to ANT's output
> if a test/suite fails. If a test/suite passes, its output is
> redirected in full to "tests-report.txt" file for inspection if
> needed.
>
> 3) WHAT ARE THE OTHER GOODIES?
>
> There are many... For one thing, the console output will be much less
> chatty. Typically, one will see something like this:
>
>   [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
>   [junit4] Expected execution time on JVM J0:    93.93s
>   [junit4] Expected execution time on JVM J1:    93.93s
>   [junit4] Expected execution time on JVM J2:    93.93s
>   [junit4] Expected execution time on JVM J3:    93.93s
>   [junit4] Executing 296 suites with 4 JVMs.
>   [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
>   [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.31s
>   [junit4]
>   [junit4] Running org.apache.lucene.index.TestStressIndexing2
>   [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.89s
> ...
>
> The number of JVMs used depends on your number of cores (and is not
> equal to their number to leave some room for infrastructure). ant
> test-help contains instructions how to override the number of JVMs to
> make the executions sequential, for example.
>
> Another thing is that tests (their suite, output, etc.) that _failed_
> are appended to tests-failures.txt. I remember that somebody once
> complained that it's impossible to find a test failure in ANT's log.
> Now it should be easier. I made a test fail; the output is like this
> (console):
>
>   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>   [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
>   [junit4]    > Throwable #1: java.lang.AssertionError
>   [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:0)
>   [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
>   [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
>   [junit4]    >        at
> org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorerFreqs.java:156)
> ....[snip]
>   [junit4]    >
>   [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.class=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A
> -Dargs="-Dfile.encoding=Cp1252"
>   [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.filter=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A
> -Dargs="-Dfile.encoding=Cp1252"
>   [junit4]   2>
>   [junit4]    > (@AfterClass output)
>   [junit4]   2> NOTE: test params are: codec=Lucene40:
> {f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da,
> timezone=Asia/Bishkek
>   [junit4]   2> NOTE: all tests run in this JVM:
>   [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
>   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
> (64-bit)/cpus=8,threads=1,free=97723464,total=128647168
>   [junit4]   2>
>   [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
> Time:  0.45s <<< FAILURES!
>
> Note "<<< FAILURES!" -- this is something to grep for. The " >" marker
> is a stack trace from the throwable, "2>" is standard error, "1>" is
> standard output. The same error is also appended to
> tests-failures.txt. The stack trace contains a synthetic entry with
> the master seed and the test's seed.
>
> For those who prefer more visual (color) reports -- look for
> "tests-report-{module-name}" folder under the build folder. This
> contains index.html with a test report that should be easier to
> navigate around. An example of what such a report looks like is here:
>
> http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-Test-Report/index.html
>
> Yet another thing is that test suites are load-balanced across forked
> JVMs (with job stealing in place). For the initial balancing there are
> precomputed statistics (which should be updated from time to time!).
> Any performance improvement this yields is currently eaten away by the
> cost of random() calls from tight loops in certain tests (it's better
> to just cache the result in a local variable). I'll try to pinpoint
> these later on. The timings shouldn't be much different to what it was
> before so it's not a performance drop.
>
> I don't know what else... I'm sure there are things that I haven't
> covered. Ask if something is not clear or something is not working.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
> Great stuff Dawid!

Thanks! This stuff is based on concepts that were already present in
Lucene so I can't take the credit for it, but I admit I really like it
being a separate project. We've been using it for some time in our own
codebase and it's worked great. In particular tests isolation, groups
and parameterized factories...

> The tests.iters functionality sounds like it will be particularly useful.

...which are the other goodies that I think will be useful as well.
I'll introduce them gradually so that it's not too overwhelming.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
In reply to this post by Dawid Weiss-2
Hi Dawid,

I've been running the new test setup, so far without any problems - nice! - and I noticed a couple of things:

- tests.threadspercpu seems to behave like a maximum JVM count rather a core multiplier - is that right?  If so, maybe this param should be renamed?

- How does "ant -verbose" differ from/replace "ant -Dtests.verbose" ?  Are these complementary?

Thanks for all your work on this!

Would you like me to take a crack at modifying the RunningTests wiki page?

Steve

-----Original Message-----
From: Dawid Weiss [mailto:[hidden email]]
Sent: Sunday, April 15, 2012 10:45 AM
To: [hidden email]
Subject: [Important] New test framework, differences, features, etc.

Ah... such a relief to finally have it committed even if this means the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene and make it available as a separate project for reuse in other projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an aggregation of events, balancing, logs, different (from standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with RandomizedRunner. It also modifies build scripts to run junit4-ant instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is gone. This change was motivated by the fact that the field's value was read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc. Instead, the Random instance for a given scope (see below) can be acquired from a static method in LuceneTestCase called random(). In the essence, you can just add brackets around your previous random field references and it _should_ work out of the box. There are differences though: Random object returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random instance in a static scope (@BeforeClass) to a field and reusing it in a test, passing a Random instance from one thread to another, saving a Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the scope (test method, hook) it was acquired for. You _can_ call random() from non-test threads -- they will be given their own thread-local Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed to 'tests.iters' (you'll get a build failure and a message if you try to use the former one). I could add a fallback but I'd rather not introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a given test (or tests), including any @Before/@After hooks and setups. In theory, this means it is now possible to reiterate ANY test, no matter how complex. If it doesn't depend on static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three times, but they all count individually (so the reported total is 9 tests). What's also clearly seen is that the master seed is constant for all tests but each repetition gets a (predictable) derivative of the random seed. This way you can reiterate N times over a test, each time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2 seed=[DEADBEEF:CAFEBABE]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s

We now fixed both the master and per-method seeds. Every repetition will have the same randomness. Test failures will report test error seeds as "master:test" so you will be able to use both scenarios, depending on the use case.

- "tests.iter.min" has been temporarily removed. I will think of a way to reintegrate it.

- The console output (sysout/systerrs) is only emitted to ANT's output if a test/suite fails. If a test/suite passes, its output is redirected in full to "tests-report.txt" file for inspection if needed.

3) WHAT ARE THE OTHER GOODIES?

There are many... For one thing, the console output will be much less chatty. Typically, one will see something like this:

   [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
   [junit4] Expected execution time on JVM J0:    93.93s
   [junit4] Expected execution time on JVM J1:    93.93s
   [junit4] Expected execution time on JVM J2:    93.93s
   [junit4] Expected execution time on JVM J3:    93.93s
   [junit4] Executing 296 suites with 4 JVMs.
   [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
   [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.31s
   [junit4]
   [junit4] Running org.apache.lucene.index.TestStressIndexing2
   [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.89s
...

The number of JVMs used depends on your number of cores (and is not equal to their number to leave some room for infrastructure). ant test-help contains instructions how to override the number of JVMs to make the executions sequential, for example.

Another thing is that tests (their suite, output, etc.) that _failed_ are appended to tests-failures.txt. I remember that somebody once complained that it's impossible to find a test failure in ANT's log.
Now it should be easier. I made a test fail; the output is like this
(console):

   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at
__randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:0)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
   [junit4]    >        at
org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorerFreqs.java:156)
....[snip]
   [junit4]    >
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.class=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.filter=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
   [junit4]    > (@AfterClass output)
   [junit4]   2> NOTE: test params are: codec=Lucene40:
{f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da, timezone=Asia/Bishkek
   [junit4]   2> NOTE: all tests run in this JVM:
   [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
(64-bit)/cpus=8,threads=1,free=97723464,total=128647168
   [junit4]   2>
   [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
Time:  0.45s <<< FAILURES!

Note "<<< FAILURES!" -- this is something to grep for. The " >" marker is a stack trace from the throwable, "2>" is standard error, "1>" is standard output. The same error is also appended to tests-failures.txt. The stack trace contains a synthetic entry with the master seed and the test's seed.

For those who prefer more visual (color) reports -- look for "tests-report-{module-name}" folder under the build folder. This contains index.html with a test report that should be easier to navigate around. An example of what such a report looks like is here:

http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-Test-Report/index.html

Yet another thing is that test suites are load-balanced across forked JVMs (with job stealing in place). For the initial balancing there are precomputed statistics (which should be updated from time to time!).
Any performance improvement this yields is currently eaten away by the cost of random() calls from tight loops in certain tests (it's better to just cache the result in a local variable). I'll try to pinpoint these later on. The timings shouldn't be much different to what it was before so it's not a performance drop.

I don't know what else... I'm sure there are things that I haven't covered. Ask if something is not clear or something is not working.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
Hi Steve,

> - tests.threadspercpu seems to behave like a maximum JVM count rather a core multiplier - is that right?  If so, maybe this param should be renamed?

You're right, I don't know how it came to be threadspercpu... it is
actually the number of forked JVMs -- it should be "tests.jvms"
(sounds good?). There is no reliable way to tell if a cpu is running
in hyperthreading mode or with real threads (multicpu motherboards,
for example) and from my experiments it seems that going into
hyperthreading does not speed up things (possibly because Java has its
own concurrent threads anyway so everything's saturated already).

I'll rename this attribute -- thanks for spotting this.

> - How does "ant -verbose" differ from/replace "ant -Dtests.verbose" ?  Are these complementary?

"ant -verbose" is a global switch to run ant's logger in verbose mode.
So you'll get tons of logs from tasks other than junit4 runner, but
including it. I just reused ant's infrastructure here. -Dtests.verbose
is project (lucene/solr) specific.

> Would you like me to take a crack at modifying the RunningTests wiki page?

Oh, this would be awesome if you could. I am wiki challenged...

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
On 4/15/2012 at 1:35 PM, Dawid Weiss wrote:
> > - tests.threadspercpu seems to behave like a maximum JVM
> > count rather a core multiplier - is that right?  If so,
> > maybe this param should be renamed?
>
> You're right, I don't know how it came to be threadspercpu...
> it is actually the number of forked JVMs -- it should be
> "tests.jvms" (sounds good?).

+1

> There is no reliable way to tell if a cpu is running in
> hyperthreading mode or with real threads (multicpu
> motherboards, for example) and from my experiments it seems
> that going into hyperthreading does not speed up things
> (possibly because Java has its own concurrent threads anyway
> so everything's saturated already).

On my machine (Win7 64-bit, 4 real cores + Hyperthreading -> 8 "cores"), leaving tests.threadspercu at the default "auto" maxes out at 4 JVMs.  

Running all tests from the top level, the sweet spot for me seems to be tests.threadspercpu=5 - this is about 8-10% faster than "auto".  tests.threadspercpu=6 is also faster than "auto", but only by about 3-4%.  (I have only done a couple trials, and this is on a workstation where I'm doing other stuff...)

> > Would you like me to take a crack at modifying the RunningTests wiki page?
>
> Oh, this would be awesome if you could. I am wiki challenged...

OK, I'll work on it later today.

Steve

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
>> "tests.jvms" (sounds good?).
>
> +1

It'll be in in the next batch of changes.

> On my machine (Win7 64-bit, 4 real cores + Hyperthreading -> 8 "cores"), leaving tests.threadspercu at the default "auto" maxes out at 4 JVMs.

I have a similar machine. I was afraid to overspecify this tuning
though. Like if you want to do anything else in the background it does
make a difference. I guess if somebody wishes to commit more resoures
they can tune it manually?

> OK, I'll work on it later today.

Thanks.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Mark Miller-3

On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:

> I have a similar machine. I was afraid to overspecify this tuning
> though. Like if you want to do anything else in the background it does
> make a difference. I guess if somebody wishes to commit more resoures
> they can tune it manually?

+1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.

- Mark Miller
lucidimagination.com












---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
The heuristic is here if anybody wants to contribute their ideas.

https://github.com/carrotsearch/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L850-878

The maximum for "auto" mode is 4 jvms if somebody has >= 8 logical
processors. I didn't want to increase it because folks like Mike (24
cores if I remember) would quickly run out of memory with so many jvms
forked. It would be a nice touch to try to sense -Xmx options passed
to the forked JVMs and pick the number of forks (or warn the user
about memory limit) based on that but I haven't done it yet.

Dawid

On Mon, Apr 16, 2012 at 4:42 AM, Mark Miller <[hidden email]> wrote:

>
> On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:
>
>> I have a similar machine. I was afraid to overspecify this tuning
>> though. Like if you want to do anything else in the background it does
>> make a difference. I guess if somebody wishes to commit more resoures
>> they can tune it manually?
>
> +1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
I ran 5 trials on my machine (Win7/64, 4 cores+HT->8 "cores") of testing under lucene/ (including the contribs there) without compilation, setting tests.jvms/threadspercpu to auto (4), 5, 6, and 7:

        || jvms || min || max || avg ||
        |     4  |  92s | 109s |  97s |
        |     5  |  80s |  96s |  92s |
        |     6  |  90s |  98s |  98s |
        |     7  |  91s | 110s | 100s |

So the benefit of overriding "auto" is barely noticeable, in both relative and absolute terms.

Steve

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 2:56 AM
To: [hidden email]
Subject: Re: [Important] New test framework, differences, features, etc.

The heuristic is here if anybody wants to contribute their ideas.

https://github.com/carrotsearch/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L850-878

The maximum for "auto" mode is 4 jvms if somebody has >= 8 logical processors. I didn't want to increase it because folks like Mike (24 cores if I remember) would quickly run out of memory with so many jvms forked. It would be a nice touch to try to sense -Xmx options passed to the forked JVMs and pick the number of forks (or warn the user about memory limit) based on that but I haven't done it yet.

Dawid

On Mon, Apr 16, 2012 at 4:42 AM, Mark Miller <[hidden email]> wrote:

>
> On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:
>
>> I have a similar machine. I was afraid to overspecify this tuning
>> though. Like if you want to do anything else in the background it
>> does make a difference. I guess if somebody wishes to commit more
>> resoures they can tune it manually?
>
> +1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For
> additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
In reply to this post by Dawid Weiss-2
Hi Dawid,

-Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Steve

-----Original Message-----
From: Dawid Weiss [mailto:[hidden email]]
Sent: Sunday, April 15, 2012 10:45 AM
To: [hidden email]
Subject: [Important] New test framework, differences, features, etc.

Ah... such a relief to finally have it committed even if this means the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene and make it available as a separate project for reuse in other projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an aggregation of events, balancing, logs, different (from standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with RandomizedRunner. It also modifies build scripts to run junit4-ant instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is gone. This change was motivated by the fact that the field's value was read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc. Instead, the Random instance for a given scope (see below) can be acquired from a static method in LuceneTestCase called random(). In the essence, you can just add brackets around your previous random field references and it _should_ work out of the box. There are differences though: Random object returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random instance in a static scope (@BeforeClass) to a field and reusing it in a test, passing a Random instance from one thread to another, saving a Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the scope (test method, hook) it was acquired for. You _can_ call random() from non-test threads -- they will be given their own thread-local Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed to 'tests.iters' (you'll get a build failure and a message if you try to use the former one). I could add a fallback but I'd rather not introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a given test (or tests), including any @Before/@After hooks and setups. In theory, this means it is now possible to reiterate ANY test, no matter how complex. If it doesn't depend on static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three times, but they all count individually (so the reported total is 9 tests). What's also clearly seen is that the master seed is constant for all tests but each repetition gets a (predictable) derivative of the random seed. This way you can reiterate N times over a test, each time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2 seed=[DEADBEEF:CAFEBABE]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s

We now fixed both the master and per-method seeds. Every repetition will have the same randomness. Test failures will report test error seeds as "master:test" so you will be able to use both scenarios, depending on the use case.

- "tests.iter.min" has been temporarily removed. I will think of a way to reintegrate it.

- The console output (sysout/systerrs) is only emitted to ANT's output if a test/suite fails. If a test/suite passes, its output is redirected in full to "tests-report.txt" file for inspection if needed.

3) WHAT ARE THE OTHER GOODIES?

There are many... For one thing, the console output will be much less chatty. Typically, one will see something like this:

   [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
   [junit4] Expected execution time on JVM J0:    93.93s
   [junit4] Expected execution time on JVM J1:    93.93s
   [junit4] Expected execution time on JVM J2:    93.93s
   [junit4] Expected execution time on JVM J3:    93.93s
   [junit4] Executing 296 suites with 4 JVMs.
   [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
   [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.31s
   [junit4]
   [junit4] Running org.apache.lucene.index.TestStressIndexing2
   [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
Time:  1.89s
...

The number of JVMs used depends on your number of cores (and is not equal to their number to leave some room for infrastructure). ant test-help contains instructions how to override the number of JVMs to make the executions sequential, for example.

Another thing is that tests (their suite, output, etc.) that _failed_ are appended to tests-failures.txt. I remember that somebody once complained that it's impossible to find a test failure in ANT's log.
Now it should be easier. I made a test fail; the output is like this
(console):

   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at
__randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:0)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
   [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
   [junit4]    >        at
org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorerFreqs.java:156)
....[snip]
   [junit4]    >
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.class=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2> NOTE: reproduce with: ant test
-Dtests.filter=*.TestSubScorerFreqs
-Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
   [junit4]    > (@AfterClass output)
   [junit4]   2> NOTE: test params are: codec=Lucene40:
{f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da, timezone=Asia/Bishkek
   [junit4]   2> NOTE: all tests run in this JVM:
   [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
(64-bit)/cpus=8,threads=1,free=97723464,total=128647168
   [junit4]   2>
   [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
Time:  0.45s <<< FAILURES!

Note "<<< FAILURES!" -- this is something to grep for. The " >" marker is a stack trace from the throwable, "2>" is standard error, "1>" is standard output. The same error is also appended to tests-failures.txt. The stack trace contains a synthetic entry with the master seed and the test's seed.

For those who prefer more visual (color) reports -- look for "tests-report-{module-name}" folder under the build folder. This contains index.html with a test report that should be easier to navigate around. An example of what such a report looks like is here:

http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-Test-Report/index.html

Yet another thing is that test suites are load-balanced across forked JVMs (with job stealing in place). For the initial balancing there are precomputed statistics (which should be updated from time to time!).
Any performance improvement this yields is currently eaten away by the cost of random() calls from tight loops in certain tests (it's better to just cache the result in a local variable). I'll try to pinpoint these later on. The timings shouldn't be much different to what it was before so it's not a performance drop.

I don't know what else... I'm sure there are things that I haven't covered. Ask if something is not clear or something is not working.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
> -Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Will this work for you (ant test-help):

     [echo] # Run all tests in a package and sub-packages
     [echo] ant test "-Dtests.class=org.apache.lucene.package.*"

This is a globbing pattern so you can also do:

ant test "-Dtests.class=*.packagename.*"

We could simulate previous options but globbing seems more versatile?
If you really need it then we can re-add it, sure.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
I don't really need it - I was just asking about the absence of an alias for a previously advertised feature.  If you think it best not to continue to support that, it's fine with me.  (I'm updating the RunningTests wiki now, where this feature is described.)

Steve

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 9:03 AM
To: [hidden email]
Subject: Re: [Important] New test framework, differences, features, etc.

> -Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Will this work for you (ant test-help):

     [echo] # Run all tests in a package and sub-packages
     [echo] ant test "-Dtests.class=org.apache.lucene.package.*"

This is a globbing pattern so you can also do:

ant test "-Dtests.class=*.packagename.*"

We could simulate previous options but globbing seems more versatile?
If you really need it then we can re-add it, sure.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
In reply to this post by steve_rowe
>        || jvms || min || max || avg ||
>        |     4  |  92s | 109s |  97s |
>        |     5  |  80s |  96s |  92s |
>        |     6  |  90s |  98s |  98s |
>        |     7  |  91s | 110s | 100s |
>
> So the benefit of overriding "auto" is barely noticeable, in both relative and absolute terms.

If you go "down" from 4 you'll see the difference real cores make -- 4
is pretty much saturated I7 (I have the same). If somebody has more
than 4 physical cores they will see a difference (I think, I don't
have access to such a machine). Memory bandwidth also comes into play
here so it won't scale perfectly I bet.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
In reply to this post by steve_rowe
> I don't really need it - I was just asking about the absence of an alias for a previously advertised feature.  If you think it best not to continue to support that, it's fine with me.  (I'm updating the RunningTests wiki now, where this feature is described.)

Oh, ok. I'd rather remove package and packageroot and replace it with
an example of globbing?

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

Uwe Schindler
In reply to this post by Dawid Weiss-2
One thing I disagree:

[junit4] Suite: TestTopDocsMerge
[junit4] Completed in 3.73s, 1 test

Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]


> -----Original Message-----
> From: Dawid Weiss [mailto:[hidden email]]
> Sent: Sunday, April 15, 2012 4:45 PM
> To: [hidden email]
> Subject: [Important] New test framework, differences, features, etc.
>
> Ah... such a relief to finally have it committed even if this means the work has
> just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene and
> make it available as a separate project for reuse in other projects. This kind of
> grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for randomization
> and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an
> aggregation of events, balancing, logs, different (from standard ANT) reporting
> and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on github here:
> https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant instead of
> ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is gone. This
> change was motivated by the fact that the field's value was read from places
> where it shouldn't be read, passed to places where it shouldn't be passed, etc.
> Instead, the Random instance for a given scope (see below) can be acquired
> from a static method in LuceneTestCase called random(). In the essence, you
> can just add brackets around your previous random field references and it
> _should_ work out of the box. There are differences though: Random object
> returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random instance in
> a static scope (@BeforeClass) to a field and reusing it in a test, passing a
> Random instance from one thread to another, saving a Random instance to a
> field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the scope (test
> method, hook) it was acquired for. You _can_ call random() from non-test
> threads -- they will be given their own thread-local Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed to
> 'tests.iters' (you'll get a build failure and a message if you try to use the former
> one). I could add a fallback but I'd rather not introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a
> given test (or tests), including any @Before/@After hooks and setups. In theory,
> this means it is now possible to reiterate ANY test, no matter how complex. If it
> doesn't depend on static state, it can be repeated. This also links to how seed is
> used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-
> core
>
> the above will result in 3 executions of every test in TestSubScorerFreqs. What
> you'll get on the console is:
>
>    [junit4] <JUnit4> says hello. Random seed: deadbeef
>    [junit4] Expected execution time on JVM J0:     0.02s
>    [junit4] Executing 1 suite with 1 JVM.
>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>    [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
> Time:  0.45s
>    [junit4]
>    [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>    [junit4] Execution time total: 0.98 sec.
>    [junit4] Tests summary: 1 suite, 9 tests
>
> and if you peek at the log file with test results
> (build/core/test/tests-report.txt) you'll see the details of each executed test:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
> seed=[DEADBEEF:BAE943E3CC27A0F]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
> seed=[DEADBEEF:BFF828C20800B123]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
> seed=[DEADBEEF:3111BE1E59C4F9E8]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:3DDFB93BCC712A41]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:898905C7F8B3E16D]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:760931BA977A9A6]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
> seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
> seed=[DEADBEEF:13FB542229750127]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
> seed=[DEADBEEF:9D12C2FE78B149EC]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s
>
> Note that tests.iters=3 resulted in every test case executed three times, but
> they all count individually (so the reported total is 9 tests). What's also clearly
> seen is that the master seed is constant for all tests but each repetition gets a
> (predictable) derivative of the random seed. This way you can reiterate N times
> over a test, each time with a different seed. Now, compare this to:
>
> ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -
> Dtestcase=TestSubScorerFreqs test-core
>
> The log file now will be:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s
>
> We now fixed both the master and per-method seeds. Every repetition will
> have the same randomness. Test failures will report test error seeds as
> "master:test" so you will be able to use both scenarios, depending on the use
> case.
>
> - "tests.iter.min" has been temporarily removed. I will think of a way to
> reintegrate it.
>
> - The console output (sysout/systerrs) is only emitted to ANT's output if a
> test/suite fails. If a test/suite passes, its output is redirected in full to "tests-
> report.txt" file for inspection if needed.
>
> 3) WHAT ARE THE OTHER GOODIES?
>
> There are many... For one thing, the console output will be much less chatty.
> Typically, one will see something like this:
>
>    [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
>    [junit4] Expected execution time on JVM J0:    93.93s
>    [junit4] Expected execution time on JVM J1:    93.93s
>    [junit4] Expected execution time on JVM J2:    93.93s
>    [junit4] Expected execution time on JVM J3:    93.93s
>    [junit4] Executing 296 suites with 4 JVMs.
>    [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
>    [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.31s
>    [junit4]
>    [junit4] Running org.apache.lucene.index.TestStressIndexing2
>    [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.89s
> ...
>
> The number of JVMs used depends on your number of cores (and is not equal to
> their number to leave some room for infrastructure). ant test-help contains
> instructions how to override the number of JVMs to make the executions
> sequential, for example.
>
> Another thing is that tests (their suite, output, etc.) that _failed_ are appended
> to tests-failures.txt. I remember that somebody once complained that it's
> impossible to find a test failure in ANT's log.
> Now it should be easier. I made a test fail; the output is like this
> (console):
>
>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>    [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:
> 0)
>    [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
>    [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
>    [junit4]    >        at
> org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorer
> Freqs.java:156)
> ....[snip]
>    [junit4]    >
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.class=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
> Dargs="-Dfile.encoding=Cp1252"
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.filter=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
> Dargs="-Dfile.encoding=Cp1252"
>    [junit4]   2>
>    [junit4]    > (@AfterClass output)
>    [junit4]   2> NOTE: test params are: codec=Lucene40:
> {f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da,
> timezone=Asia/Bishkek
>    [junit4]   2> NOTE: all tests run in this JVM:
>    [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
>    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
> (64-bit)/cpus=8,threads=1,free=97723464,total=128647168
>    [junit4]   2>
>    [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
> Time:  0.45s <<< FAILURES!
>
> Note "<<< FAILURES!" -- this is something to grep for. The " >" marker is a stack
> trace from the throwable, "2>" is standard error, "1>" is standard output. The
> same error is also appended to tests-failures.txt. The stack trace contains a
> synthetic entry with the master seed and the test's seed.
>
> For those who prefer more visual (color) reports -- look for "tests-report-
> {module-name}" folder under the build folder. This contains index.html with a
> test report that should be easier to navigate around. An example of what such a
> report looks like is here:
>
> http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-
> Test-Report/index.html
>
> Yet another thing is that test suites are load-balanced across forked JVMs (with
> job stealing in place). For the initial balancing there are precomputed statistics
> (which should be updated from time to time!).
> Any performance improvement this yields is currently eaten away by the cost
> of random() calls from tight loops in certain tests (it's better to just cache the
> result in a local variable). I'll try to pinpoint these later on. The timings
> shouldn't be much different to what it was before so it's not a performance
> drop.
>
> I don't know what else... I'm sure there are things that I haven't covered. Ask if
> something is not clear or something is not working.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional
> commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
I ordinarily run 80-column terminals.

But I agree with Uwe - package absence looks bad.

Maybe a compression scheme like people use would work here?  E.g. o.a.l.whatever.Class, or maybe even oal.whatever.Class ?

Also, does Java have access to the column width of the running terminal? If so, the output width would not have to have a fixed width.

Steve

-----Original Message-----
From: Uwe Schindler [mailto:[hidden email]]
Sent: Monday, April 16, 2012 9:10 AM
To: [hidden email]
Subject: RE: [Important] New test framework, differences, features, etc.

One thing I disagree:

[junit4] Suite: TestTopDocsMerge
[junit4] Completed in 3.73s, 1 test

Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]


> -----Original Message-----
> From: Dawid Weiss [mailto:[hidden email]]
> Sent: Sunday, April 15, 2012 4:45 PM
> To: [hidden email]
> Subject: [Important] New test framework, differences, features, etc.
>
> Ah... such a relief to finally have it committed even if this means
> the work has just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene
> and make it available as a separate project for reuse in other
> projects. This kind of grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for
> randomization and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
> with an aggregation of events, balancing, logs, different (from
> standard ANT) reporting and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on github here:
> https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant
> instead of ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is
> gone. This change was motivated by the fact that the field's value was
> read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc.
> Instead, the Random instance for a given scope (see below) can be
> acquired from a static method in LuceneTestCase called random(). In
> the essence, you can just add brackets around your previous random
> field references and it _should_ work out of the box. There are
> differences though: Random object returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random
> instance in a static scope (@BeforeClass) to a field and reusing it in
> a test, passing a Random instance from one thread to another, saving a
> Random instance to a field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the
> scope (test method, hook) it was acquired for. You _can_ call random()
> from non-test threads -- they will be given their own thread-local Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed
> to 'tests.iters' (you'll get a build failure and a message if you try
> to use the former one). I could add a fallback but I'd rather not introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a
> duplicate of a given test (or tests), including any @Before/@After
> hooks and setups. In theory, this means it is now possible to
> reiterate ANY test, no matter how complex. If it doesn't depend on
> static state, it can be repeated. This also links to how seed is used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3
> -Dtestcase=TestSubScorerFreqs test- core
>
> the above will result in 3 executions of every test in
> TestSubScorerFreqs. What you'll get on the console is:
>
>    [junit4] <JUnit4> says hello. Random seed: deadbeef
>    [junit4] Expected execution time on JVM J0:     0.02s
>    [junit4] Executing 1 suite with 1 JVM.
>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>    [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
> Time:  0.45s
>    [junit4]
>    [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>    [junit4] Execution time total: 0.98 sec.
>    [junit4] Tests summary: 1 suite, 9 tests
>
> and if you peek at the log file with test results
> (build/core/test/tests-report.txt) you'll see the details of each executed test:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
> seed=[DEADBEEF:BAE943E3CC27A0F]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
> seed=[DEADBEEF:BFF828C20800B123]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
> seed=[DEADBEEF:3111BE1E59C4F9E8]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:3DDFB93BCC712A41]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:898905C7F8B3E16D]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:760931BA977A9A6]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
> seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
> seed=[DEADBEEF:13FB542229750127]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
> seed=[DEADBEEF:9D12C2FE78B149EC]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s
>
> Note that tests.iters=3 resulted in every test case executed three
> times, but they all count individually (so the reported total is 9
> tests). What's also clearly seen is that the master seed is constant
> for all tests but each repetition gets a
> (predictable) derivative of the random seed. This way you can
> reiterate N times over a test, each time with a different seed. Now, compare this to:
>
> ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -
> Dtestcase=TestSubScorerFreqs test-core
>
> The log file now will be:
>
> Executing 1 suite with 1 JVM.
> Running org.apache.lucene.search.TestSubScorerFreqs
> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
> seed=[DEADBEEF:CAFEBABE]}
> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
> seed=[DEADBEEF:CAFEBABE]}
> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s
>
> We now fixed both the master and per-method seeds. Every repetition
> will have the same randomness. Test failures will report test error
> seeds as "master:test" so you will be able to use both scenarios,
> depending on the use case.
>
> - "tests.iter.min" has been temporarily removed. I will think of a way
> to reintegrate it.
>
> - The console output (sysout/systerrs) is only emitted to ANT's output
> if a test/suite fails. If a test/suite passes, its output is
> redirected in full to "tests- report.txt" file for inspection if needed.
>
> 3) WHAT ARE THE OTHER GOODIES?
>
> There are many... For one thing, the console output will be much less chatty.
> Typically, one will see something like this:
>
>    [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
>    [junit4] Expected execution time on JVM J0:    93.93s
>    [junit4] Expected execution time on JVM J1:    93.93s
>    [junit4] Expected execution time on JVM J2:    93.93s
>    [junit4] Expected execution time on JVM J3:    93.93s
>    [junit4] Executing 296 suites with 4 JVMs.
>    [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
>    [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.31s
>    [junit4]
>    [junit4] Running org.apache.lucene.index.TestStressIndexing2
>    [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
> Time:  1.89s
> ...
>
> The number of JVMs used depends on your number of cores (and is not
> equal to their number to leave some room for infrastructure). ant
> test-help contains instructions how to override the number of JVMs to
> make the executions sequential, for example.
>
> Another thing is that tests (their suite, output, etc.) that _failed_
> are appended to tests-failures.txt. I remember that somebody once
> complained that it's impossible to find a test failure in ANT's log.
> Now it should be easier. I made a test fail; the output is like this
> (console):
>
>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>    [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:
> 0)
>    [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
>    [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
>    [junit4]    >        at
> org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubSc
> orer
> Freqs.java:156)
> ....[snip]
>    [junit4]    >
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.class=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
> Dargs="-Dfile.encoding=Cp1252"
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtests.filter=*.TestSubScorerFreqs
> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
> Dargs="-Dfile.encoding=Cp1252"
>    [junit4]   2>
>    [junit4]    > (@AfterClass output)
>    [junit4]   2> NOTE: test params are: codec=Lucene40:
> {f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da,
> timezone=Asia/Bishkek
>    [junit4]   2> NOTE: all tests run in this JVM:
>    [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
>    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
> (64-bit)/cpus=8,threads=1,free=97723464,total=128647168
>    [junit4]   2>
>    [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
> Time:  0.45s <<< FAILURES!
>
> Note "<<< FAILURES!" -- this is something to grep for. The " >" marker
> is a stack trace from the throwable, "2>" is standard error, "1>" is
> standard output. The same error is also appended to
> tests-failures.txt. The stack trace contains a synthetic entry with the master seed and the test's seed.
>
> For those who prefer more visual (color) reports -- look for
> "tests-report- {module-name}" folder under the build folder. This
> contains index.html with a test report that should be easier to
> navigate around. An example of what such a report looks like is here:
>
> http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/J
> Unit-
> Test-Report/index.html
>
> Yet another thing is that test suites are load-balanced across forked
> JVMs (with job stealing in place). For the initial balancing there are
> precomputed statistics (which should be updated from time to time!).
> Any performance improvement this yields is currently eaten away by the
> cost of random() calls from tight loops in certain tests (it's better
> to just cache the result in a local variable). I'll try to pinpoint
> these later on. The timings shouldn't be much different to what it was
> before so it's not a performance drop.
>
> I don't know what else... I'm sure there are things that I haven't
> covered. Ask if something is not clear or something is not working.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For
> additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
In reply to this post by Uwe Schindler
I am indifferent about it, really. If there are identicl class names
in the code perhaps they should be made more specific though? This
will matter not only for the output but also for other things
(ambiguous imports etc).

Anyway, feel free to play with these attributes --

                    useSimpleNames="true"
                    maxClassNameColumns="70"

if you change useSimpleNames to 'false' and keep the limit there will
be an ellipsis before max number of columns is reached.

Dawid

On Mon, Apr 16, 2012 at 3:09 PM, Uwe Schindler <[hidden email]> wrote:

> One thing I disagree:
>
> [junit4] Suite: TestTopDocsMerge
> [junit4] Completed in 3.73s, 1 test
>
> Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
>
>> -----Original Message-----
>> From: Dawid Weiss [mailto:[hidden email]]
>> Sent: Sunday, April 15, 2012 4:45 PM
>> To: [hidden email]
>> Subject: [Important] New test framework, differences, features, etc.
>>
>> Ah... such a relief to finally have it committed even if this means the work has
>> just begun on straightening it all out. So.
>>
>> 1) WHAT'S BEEN DONE?
>>
>> The primary objective was to pull out "randomized" runner from Lucene and
>> make it available as a separate project for reuse in other projects. This kind of
>> grew into two things:
>>
>> a) RandomizedRunner - a runner that has built-in support for randomization
>> and other interesting features,
>> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an
>> aggregation of events, balancing, logs, different (from standard ANT) reporting
>> and forked JVM crash resilience.
>>
>> Everything above is heavily covered with unit tests (the code is on github here:
>> https://github.com/carrotsearch/randomizedtesting).
>>
>> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
>> RandomizedRunner. It also modifies build scripts to run junit4-ant instead of
>> ANT's default <junit>.
>>
>> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>>
>> - The most visible change is that 'random' field in LuceneTestCase is gone. This
>> change was motivated by the fact that the field's value was read from places
>> where it shouldn't be read, passed to places where it shouldn't be passed, etc.
>> Instead, the Random instance for a given scope (see below) can be acquired
>> from a static method in LuceneTestCase called random(). In the essence, you
>> can just add brackets around your previous random field references and it
>> _should_ work out of the box. There are differences though: Random object
>> returned by random() is valid only for the scope it was created for.
>> So any of the following will end up in an exception: saving a Random instance in
>> a static scope (@BeforeClass) to a field and reusing it in a test, passing a
>> Random instance from one thread to another, saving a Random instance to a
>> field in one test, using it in another, etc. In
>> short: the result of random() is per-thread and only valid for the scope (test
>> method, hook) it was acquired for. You _can_ call random() from non-test
>> threads -- they will be given their own thread-local Random instances.
>>
>> - The 'random seed' is a single HEX-encoded long. The "three seeds"
>> from before are gone. Everything is a derivative of the initial master seed.
>>
>> - I provided a 'help on syntax' for test properties. Type:
>>
>> ant test-help
>>
>> and the most common use case scenarios will be dumped to your console.
>>
>> - A notable difference is that 'tests.iter' property has been renamed to
>> 'tests.iters' (you'll get a build failure and a message if you try to use the former
>> one). I could add a fallback but I'd rather not introduce any more aliases.
>>
>> - "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a
>> given test (or tests), including any @Before/@After hooks and setups. In theory,
>> this means it is now possible to reiterate ANY test, no matter how complex. If it
>> doesn't depend on static state, it can be repeated. This also links to how seed is
>> used.
>> It's probably best explained on an example:
>>
>> ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-
>> core
>>
>> the above will result in 3 executions of every test in TestSubScorerFreqs. What
>> you'll get on the console is:
>>
>>    [junit4] <JUnit4> says hello. Random seed: deadbeef
>>    [junit4] Expected execution time on JVM J0:     0.02s
>>    [junit4] Executing 1 suite with 1 JVM.
>>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>>    [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
>> Time:  0.45s
>>    [junit4]
>>    [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>>    [junit4] Execution time total: 0.98 sec.
>>    [junit4] Tests summary: 1 suite, 9 tests
>>
>> and if you peek at the log file with test results
>> (build/core/test/tests-report.txt) you'll see the details of each executed test:
>>
>> Executing 1 suite with 1 JVM.
>> Running org.apache.lucene.search.TestSubScorerFreqs
>> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
>> seed=[DEADBEEF:BAE943E3CC27A0F]}
>> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
>> seed=[DEADBEEF:BFF828C20800B123]}
>> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
>> seed=[DEADBEEF:3111BE1E59C4F9E8]}
>> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
>> seed=[DEADBEEF:3DDFB93BCC712A41]}
>> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
>> seed=[DEADBEEF:898905C7F8B3E16D]}
>> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
>> seed=[DEADBEEF:760931BA977A9A6]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
>> seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
>> seed=[DEADBEEF:13FB542229750127]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
>> seed=[DEADBEEF:9D12C2FE78B149EC]}
>> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s
>>
>> Note that tests.iters=3 resulted in every test case executed three times, but
>> they all count individually (so the reported total is 9 tests). What's also clearly
>> seen is that the master seed is constant for all tests but each repetition gets a
>> (predictable) derivative of the random seed. This way you can reiterate N times
>> over a test, each time with a different seed. Now, compare this to:
>>
>> ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -
>> Dtestcase=TestSubScorerFreqs test-core
>>
>> The log file now will be:
>>
>> Executing 1 suite with 1 JVM.
>> Running org.apache.lucene.search.TestSubScorerFreqs
>> OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
>> seed=[DEADBEEF:CAFEBABE]}
>> OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
>> seed=[DEADBEEF:CAFEBABE]}
>> Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.46s
>>
>> We now fixed both the master and per-method seeds. Every repetition will
>> have the same randomness. Test failures will report test error seeds as
>> "master:test" so you will be able to use both scenarios, depending on the use
>> case.
>>
>> - "tests.iter.min" has been temporarily removed. I will think of a way to
>> reintegrate it.
>>
>> - The console output (sysout/systerrs) is only emitted to ANT's output if a
>> test/suite fails. If a test/suite passes, its output is redirected in full to "tests-
>> report.txt" file for inspection if needed.
>>
>> 3) WHAT ARE THE OTHER GOODIES?
>>
>> There are many... For one thing, the console output will be much less chatty.
>> Typically, one will see something like this:
>>
>>    [junit4] <JUnit4> says hello. Random seed: 71F222160952E78A
>>    [junit4] Expected execution time on JVM J0:    93.93s
>>    [junit4] Expected execution time on JVM J1:    93.93s
>>    [junit4] Expected execution time on JVM J2:    93.93s
>>    [junit4] Expected execution time on JVM J3:    93.93s
>>    [junit4] Executing 296 suites with 4 JVMs.
>>    [junit4] Running org.apache.lucene.index.TestTermVectorsWriter
>>    [junit4] Tests run:  13, Failures:   0, Errors:   0, Skipped:   0,
>> Time:  1.31s
>>    [junit4]
>>    [junit4] Running org.apache.lucene.index.TestStressIndexing2
>>    [junit4] Tests run:   3, Failures:   0, Errors:   0, Skipped:   0,
>> Time:  1.89s
>> ...
>>
>> The number of JVMs used depends on your number of cores (and is not equal to
>> their number to leave some room for infrastructure). ant test-help contains
>> instructions how to override the number of JVMs to make the executions
>> sequential, for example.
>>
>> Another thing is that tests (their suite, output, etc.) that _failed_ are appended
>> to tests-failures.txt. I remember that somebody once complained that it's
>> impossible to find a test failure in ANT's log.
>> Now it should be easier. I made a test fail; the output is like this
>> (console):
>>
>>    [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>>    [junit4] FAILURE 0.06s | TestSubScorerFreqs.testBooleanQuery
>>    [junit4]    > Throwable #1: java.lang.AssertionError
>>    [junit4]    >        at
>> __randomizedtesting.SeedInfo.seed([B2258B2DE68D190A:8FFA3216F4518DA4]:
>> 0)
>>    [junit4]    >        at org.junit.Assert.fail(Assert.java:92)
>>    [junit4]    >        at org.junit.Assert.fail(Assert.java:100)
>>    [junit4]    >        at
>> org.apache.lucene.search.TestSubScorerFreqs.testBooleanQuery(TestSubScorer
>> Freqs.java:156)
>> ....[snip]
>>    [junit4]    >
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtests.class=*.TestSubScorerFreqs
>> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
>> Dargs="-Dfile.encoding=Cp1252"
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtests.filter=*.TestSubScorerFreqs
>> -Dtests.filter.method=testBooleanQuery -Dtests.seed=B2258B2DE68D190A -
>> Dargs="-Dfile.encoding=Cp1252"
>>    [junit4]   2>
>>    [junit4]    > (@AfterClass output)
>>    [junit4]   2> NOTE: test params are: codec=Lucene40:
>> {f=PostingsFormat(name=MockSep)}, sim=DefaultSimilarity, locale=da,
>> timezone=Asia/Bishkek
>>    [junit4]   2> NOTE: all tests run in this JVM:
>>    [junit4]   2> [TestSubScorerFreqs, TestSubScorerFreqs]
>>    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_03
>> (64-bit)/cpus=8,threads=1,free=97723464,total=128647168
>>    [junit4]   2>
>>    [junit4] Tests run:   3, Failures:   1, Errors:   0, Skipped:   0,
>> Time:  0.45s <<< FAILURES!
>>
>> Note "<<< FAILURES!" -- this is something to grep for. The " >" marker is a stack
>> trace from the throwable, "2>" is standard error, "1>" is standard output. The
>> same error is also appended to tests-failures.txt. The stack trace contains a
>> synthetic entry with the master seed and the test's seed.
>>
>> For those who prefer more visual (color) reports -- look for "tests-report-
>> {module-name}" folder under the build folder. This contains index.html with a
>> test report that should be easier to navigate around. An example of what such a
>> report looks like is here:
>>
>> http://builds.carrot2.org/browse/C2HEAD-SOURCES/latest/artifact/JOB1/JUnit-
>> Test-Report/index.html
>>
>> Yet another thing is that test suites are load-balanced across forked JVMs (with
>> job stealing in place). For the initial balancing there are precomputed statistics
>> (which should be updated from time to time!).
>> Any performance improvement this yields is currently eaten away by the cost
>> of random() calls from tight loops in certain tests (it's better to just cache the
>> result in a local variable). I'll try to pinpoint these later on. The timings
>> shouldn't be much different to what it was before so it's not a performance
>> drop.
>>
>> I don't know what else... I'm sure there are things that I haven't covered. Ask if
>> something is not clear or something is not working.
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email] For additional
>> commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Important] New test framework, differences, features, etc.

Dawid Weiss
In reply to this post by steve_rowe
> But I agree with Uwe - package absence looks bad.

Again -- play with these attributes to find what you like.

> Maybe a compression scheme like people use would work here?  E.g. o.a.l.whatever.Class, or maybe even oal.whatever.Class ?

I don't particularly like such compressions, they're misleading (some
obfuscators do emit o.a.l.... :).

> Also, does Java have access to the column width of the running terminal? If so, the output width would not have to have a fixed width.

I wish it had, but unfortunately the answer is no. This would make
several things easier...

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [Important] New test framework, differences, features, etc.

steve_rowe
In reply to this post by Dawid Weiss
Okay, that's what I did on the RunningTests wiki page.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 9:09 AM
To: [hidden email]
Subject: Re: [Important] New test framework, differences, features, etc.

> I don't really need it - I was just asking about the absence of an
> alias for a previously advertised feature.  If you think it best not
> to continue to support that, it's fine with me.  (I'm updating the
> RunningTests wiki now, where this feature is described.)

Oh, ok. I'd rather remove package and packageroot and replace it with an example of globbing?

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12