Speaking of tests

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Speaking of tests

Grant Ingersoll-2
Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console?  It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.  

Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.

-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Erik Hatcher-4
The JUnit report task will generate a nice HTML version that illustrates with red what tests failed and you can easily drill into it.   We have that wired into our build, right?

Granted, it'd be nice to see a summary at the end of a JUnit run in the console, but I don't know of a way to make that happen.

        Erik

On Oct 14, 2010, at 13:58 , Grant Ingersoll wrote:

> Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console?  It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.  
>
> Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Grant Ingersoll-2

On Oct 14, 2010, at 2:18 PM, Erik Hatcher wrote:

> The JUnit report task will generate a nice HTML version that illustrates with red what tests failed and you can easily drill into it.   We have that wired into our build, right?

Yeah, I can look at the reports, etc.

>
> Granted, it'd be nice to see a summary at the end of a JUnit run in the console, but I don't know of a way to make that happen.

but, I just want a simple print out of the class names, as in:

Tests failed:
FooTest.testFoo
BarTest.testBar

Patches welcome, I suppose!

>
> Erik
>
> On Oct 14, 2010, at 13:58 , Grant Ingersoll wrote:
>
>> Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console?  It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.  
>>
>> Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.
>>
>> -Grant
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Robert Muir
On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <[hidden email]> wrote:

> but, I just want a simple print out of the class names, as in:
>
> Tests failed:
> FooTest.testFoo
> BarTest.testBar
>
> Patches welcome, I suppose!
>

maybe open an issue?
I think its possible we can improve this.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Mark Miller-3
We could, at the least, make

TEST org.apache.solr.analysis.TestTypeAsFieldFilterFactory FAILED

stand out a bit more - prefix with a bunch of ***** or whatever.

It would be nice if the red (in eclipse) error stream printed the TEST
FAILED correctly, but I don't think that would be very easy to sync up
with std out...

- Mark

On 10/14/10 3:00 PM, Robert Muir wrote:

> On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <[hidden email]> wrote:
>
>> but, I just want a simple print out of the class names, as in:
>>
>> Tests failed:
>> FooTest.testFoo
>> BarTest.testBar
>>
>> Patches welcome, I suppose!
>>
>
> maybe open an issue?
> I think its possible we can improve this.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Shai Erera
What we use to do is run 'ant test' with a redirection of standard output to a file. That way, the errors stand out - they appear in the console, and the rest of the test output goes to the file.

Don't know if that's what you want, but it sure makes it much easier to spot the failing tests.

Shai

On Thu, Oct 14, 2010 at 9:16 PM, Mark Miller <[hidden email]> wrote:
We could, at the least, make

TEST org.apache.solr.analysis.TestTypeAsFieldFilterFactory FAILED

stand out a bit more - prefix with a bunch of ***** or whatever.

It would be nice if the red (in eclipse) error stream printed the TEST
FAILED correctly, but I don't think that would be very easy to sync up
with std out...

- Mark

On 10/14/10 3:00 PM, Robert Muir wrote:
> On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <[hidden email]> wrote:
>
>> but, I just want a simple print out of the class names, as in:
>>
>> Tests failed:
>> FooTest.testFoo
>> BarTest.testBar
>>
>> Patches welcome, I suppose!
>>
>
> maybe open an issue?
> I think its possible we can improve this.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Chris Hostetter-3
In reply to this post by Grant Ingersoll-2

: Anyone know how to print out, at the end of a long run of tests, the
: names of the actual tests that fail to the console?  It's a pain to

I typically use...

        grep -rL 'errors="0" failures="0"' test_output_dir

(where test_output_dir depends on which set of tests failed)

however two recent changes seem to have made this problematic...

1) all of the "temp" files for tetss now seem to live in the
test_output_dir ... i know thta the threading issues made it important to
start splitting up the working dirs for each test, but i'm relaly not sure
while they really need to be in the same dir as the output files .. in any
case adding a "*.xml" was my work arround when that start happening.

2) Solr tests recently started using the "Plain text" formatter by
default instead of the xml formatter.  this seems like a mistake to me,
but i'm not certain.


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Erick Erickson
In Junit4, there's a TestWatchman class that has a failed method that's 
called whenever a test fails. It *seems* like it would be possible to
gather them all up and print them at the end, but I confess I have no
clue how, and don't have time to look now.

Not much help, but...

Erick

On Thu, Oct 14, 2010 at 6:35 PM, Chris Hostetter <[hidden email]> wrote:

: Anyone know how to print out, at the end of a long run of tests, the
: names of the actual tests that fail to the console?  It's a pain to

I typically use...

       grep -rL 'errors="0" failures="0"' test_output_dir

(where test_output_dir depends on which set of tests failed)

however two recent changes seem to have made this problematic...

1) all of the "temp" files for tetss now seem to live in the
test_output_dir ... i know thta the threading issues made it important to
start splitting up the working dirs for each test, but i'm relaly not sure
while they really need to be in the same dir as the output files .. in any
case adding a "*.xml" was my work arround when that start happening.

2) Solr tests recently started using the "Plain text" formatter by
default instead of the xml formatter.  this seems like a mistake to me,
but i'm not certain.


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Chris Hostetter-3

: In Junit4, there's a TestWatchman class that has a failed method that's
: called whenever a test fails. It *seems* like it would be possible to
: gather them all up and print them at the end, but I confess I have no
: clue how, and don't have time to look now.

interesting...

the intent of TestWatchmen seems to be so that individual test classes can
configure certain things to happen when *any* of the tests fail (or have
some interesting condition happen) ... we could put that in our JUnit base
class ... but ...

It might be more reliable to modify our recently add
LuceneJUnitResultFormatter to do tis aggregation as it collects failure
info ... i'm not seeing any sort of "tear down" or "close" method though,
so i'm not sure if it's really possible.

Hmmm...

I do note that in Ant 1.8.0 there is a "failure" formatter that looks kind
of cool ... it's purpose is to generate a new TestCase that contains calls
to all of the tests that failed, but the lifecyle concepts could be
borrowed to generate the same kind of info in a summary report...

http://ant.apache.org/manual/Tasks/junit.html
http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/ant/taskdefs/optional/junit/FailureRecorder.java


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Robert Muir
On Thu, Oct 14, 2010 at 9:54 PM, Chris Hostetter
<[hidden email]> wrote:

> It might be more reliable to modify our recently add
> LuceneJUnitResultFormatter to do tis aggregation as it collects failure
> info ... i'm not seeing any sort of "tear down" or "close" method though,
> so i'm not sure if it's really possible.
>

+1, this is the place to do it. you can get the test stats in
endTestSuite(JUnitTest suite)
and append it to a file (set from a sysprop).

then later in the build, when the "tests.failed" sysprop is set, we
cat the file.

i just added a patch to SOLR-2002 so that we only check the
"tests.failed" once so you wont see it a ton of times... or the BUILD
FAILED! over and over and over again.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Chris Hostetter-3

: +1, this is the place to do it. you can get the test stats in
: endTestSuite(JUnitTest suite)
: and append it to a file (set from a sysprop).
:
: then later in the build, when the "tests.failed" sysprop is set, we
: cat the file.

I hadn't even though of that ... i was assuming we'd need to implement the
"BuildListener" interface and use the "taskFinished" method (like
FailureRecorder does) to write the info to the console.

-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Speaking of tests

Chris Hostetter-3
In reply to this post by Robert Muir

: i just added a patch to SOLR-2002 so that we only check the
: "tests.failed" once so you wont see it a ton of times... or the BUILD
: FAILED! over and over and over again.

I actually thought that was intentional ... the a test didn't just "FAIL"
several tests "FAILED FAILED FAILED!!!!!!"

(kind of like: you aren't just "under arrest" for a misdomenier, you are
"UNDER ARREST FOR A FEDERAL FELONY!!!")


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]