test buffering causes OOM errors

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

test buffering causes OOM errors

Yonik Seeley-2-2
Is there a way we can write to a file instead of buffering test output
in memory?
This has been causing exceptions for me lately.

[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4] at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4] at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)

-Yonik
http://lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Dawid Weiss-2

Full stack trace would me more helpful but I think the problem is that solr tests emit megabytes of data to the console and this has to be buffered and passed to report listeners.

It is possible to spill to disk of course but not a trivial refactoring.

An easy workaround for now would be to redirect java logging to a file and keep the console output smaller.

There is an issue for those Ooms on github but I admit it isn't of super high priority for me given the workaround above?

On Jul 21, 2012 11:05 AM, "Yonik Seeley" <[hidden email]> wrote:
Is there a way we can write to a file instead of buffering test output
in memory?
This has been causing exceptions for me lately.

[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4]         at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4]         at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4]         at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)

-Yonik
http://lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Yonik Seeley-2-2
On Sat, Jul 21, 2012 at 5:36 AM, Dawid Weiss <[hidden email]> wrote:
> Full stack trace would me more helpful

Full stack trace below.

> but I think the problem is that solr
> tests emit megabytes of data to the console and this has to be buffered and
> passed to report listeners.

I've not looked into the test framework architecture and don't
understand why things can't just be streamed - but from my perspective
the most important thing is just getting the logs when a test fails.
Logs are most useful in big stress tests where things are not easily
reproducible or debuggable with a debugger - and these also tend to
produce a lot of logging.

A month or so ago, I also hit issues with tests.iters (I think I got
OOM errors when I cranked it too high?)
so I went back to looping the test myself in a shell script.
I don't know what the exact behavior here is, but what would be most
useful is if the output were reset for each run so that when things
fail you are left with the full log for just the run that failed.

> It is possible to spill to disk of course but not a trivial refactoring.
>
> An easy workaround for now would be to redirect java logging to a file and
> keep the console output smaller.
>
> There is an issue for those Ooms on github but I admit it isn't of super
> high priority for me given the workaround above?

-Yonik
http://lucidimagination.com


[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
[junit4:junit4] at java.lang.StringBuffer.append(StringBuffer.java:224)
[junit4:junit4] at java.io.StringWriter.write(StringWriter.java:84)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.LineBufferWriter.write(LineBufferWriter.java:26)
[junit4:junit4] at java.io.Writer.write(Writer.java:96)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:32)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:41)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.decodeStreamEvents(TextReport.java:480)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.format(TextReport.java:443)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(TextReport.java:249)
[junit4:junit4] at
sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveSuiteEnd(AggregatingListener.java:150)
[junit4:junit4] at
sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4] at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4] at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.buildModel(AntXmlReport.java:149)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.onSuiteResult(AntXmlReport.java:95)
[junit4:junit4] at
sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4] at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler.pumpEvents(LocalSlaveStreamHandler.java:174)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler$1.run(LocalSlaveStreamHandler.java:94)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4] at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4] at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4] at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4] at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.processInput(WriterOutputStream.java:283)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:214)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serializeEvents(JsonAggregatedSuiteResultEventAdapter.java:75)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:34)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:20)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:97)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonReport.onSuiteResult(JsonReport.java:198)
[junit4:junit4] at
sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4] at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Dawid Weiss-2

I am on mobile but I will try to justify the design decisions I made later on.

As for now, quickly, tests.iters has been fixed and should not oom unless you run with many thousands of repetitions (if so it is better to loop manually).

As for the oom on large sysouts - incease ant memory or redirect tests logging (java logging subsystrm) to a file. Standard junit target from ant suffers from the same issue really so this is not a regression over what was available previously.

I think the easiest way, even for many repetitions, would be to redirect the logging sink to a file and add a logger.info before each test so that you can match failures with the log stream.

Sorry this comes so short but I'm on holidays.

On Jul 21, 2012 11:49 AM, "Yonik Seeley" <[hidden email]> wrote:
On Sat, Jul 21, 2012 at 5:36 AM, Dawid Weiss <[hidden email]> wrote:
> Full stack trace would me more helpful

Full stack trace below.

> but I think the problem is that solr
> tests emit megabytes of data to the console and this has to be buffered and
> passed to report listeners.

I've not looked into the test framework architecture and don't
understand why things can't just be streamed - but from my perspective
the most important thing is just getting the logs when a test fails.
Logs are most useful in big stress tests where things are not easily
reproducible or debuggable with a debugger - and these also tend to
produce a lot of logging.

A month or so ago, I also hit issues with tests.iters (I think I got
OOM errors when I cranked it too high?)
so I went back to looping the test myself in a shell script.
I don't know what the exact behavior here is, but what would be most
useful is if the output were reset for each run so that when things
fail you are left with the full log for just the run that failed.

> It is possible to spill to disk of course but not a trivial refactoring.
>
> An easy workaround for now would be to redirect java logging to a file and
> keep the console output smaller.
>
> There is an issue for those Ooms on github but I admit it isn't of super
> high priority for me given the workaround above?

-Yonik
http://lucidimagination.com


[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4]         at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
[junit4:junit4]         at java.lang.StringBuffer.append(StringBuffer.java:224)
[junit4:junit4]         at java.io.StringWriter.write(StringWriter.java:84)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.LineBufferWriter.write(LineBufferWriter.java:26)
[junit4:junit4]         at java.io.Writer.write(Writer.java:96)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:32)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.PrefixedWriter.write(PrefixedWriter.java:41)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.decodeStreamEvents(TextReport.java:480)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.format(TextReport.java:443)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(TextReport.java:249)
[junit4:junit4]         at
sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveSuiteEnd(AggregatingListener.java:150)
[junit4:junit4]         at
sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4]         at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4]         at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4]         at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:219)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.SlaveInfo.decodeStreams(SlaveInfo.java:129)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.buildModel(AntXmlReport.java:149)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.antxml.AntXmlReport.onSuiteResult(AntXmlReport.java:95)
[junit4:junit4]         at
sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4]         at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler.pumpEvents(LocalSlaveStreamHandler.java:174)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.LocalSlaveStreamHandler$1.run(LocalSlaveStreamHandler.java:94)
[junit4:junit4] Event bus dispatch error: java.lang.OutOfMemoryError:
Java heap space
[junit4:junit4] java.lang.OutOfMemoryError: Java heap space
[junit4:junit4]         at java.util.Arrays.copyOf(Arrays.java:2882)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
[junit4:junit4]         at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
[junit4:junit4]         at java.lang.StringBuffer.append(StringBuffer.java:306)
[junit4:junit4]         at java.io.StringWriter.write(StringWriter.java:77)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.flushOutput(WriterOutputStream.java:303)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.processInput(WriterOutputStream.java:283)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:214)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.org.apache.commons.io.output.WriterOutputStream.write(WriterOutputStream.java:231)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serializeEvents(JsonAggregatedSuiteResultEventAdapter.java:75)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:34)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonAggregatedSuiteResultEventAdapter.serialize(JsonAggregatedSuiteResultEventAdapter.java:20)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:97)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.listeners.json.JsonReport.onSuiteResult(JsonReport.java:198)
[junit4:junit4]         at
sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatingListener.receiveTestEnd(AggregatingListener.java:145)
[junit4:junit4]         at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
[junit4:junit4]         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]         at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:68)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatch(EventBus.java:313)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
[junit4:junit4]         at
com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus.post(EventBus.java:264)
Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Michael McCandless-2
In reply to this post by Yonik Seeley-2-2
On Sat, Jul 21, 2012 at 5:49 AM, Yonik Seeley
<[hidden email]> wrote:

> Logs are most useful in big stress tests where things are not easily
> reproducible or debuggable with a debugger - and these also tend to
> produce a lot of logging.

+1, I also miss simple streamed output option when running a test.

And it's fine if it only works if you run one test, one JVM, etc.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Dawid Weiss
> +1, I also miss simple streamed output option when running a test.
>
> And it's fine if it only works if you run one test, one JVM, etc.

There is an issue for 1-jvm streaming on github. Like I said -- it is
possible to decrease memory consumption (stdout buffering)  although
there will always be limitations (like if you run
-Dtests.iters=10000000 this results in 10000000 physical instances of
a Description object that need to be at once in memory; JUnit
architecture -- nothing I can do about it).

I will get back to this when I come back from vacation. In the
meantime I think I provided enough workarounds so that you can
continue working? I mean Solr mostly uses java logging and redirecting
this to a file is trivial. Mike -- you can do the same thing if you
need tons of logs. There is also an option of doing:

ANT_OPTS="-Xmx100g"

or whatever amount of memory you have -- this is crappy in the sense
that you'll see the output only after the test finishes but it also
should work in the mean time until I do that refactoring.

I admit harnessing threading was my priority and I am pretty happy
with the result although time will tell how useful it really is in
practice (in the sense of providing sensible output for debugging).
There are nuances and blockers from doing this right that are mind
boggling. These stem both from JUnit architecture (which is really
nice but was thought for simpler test cases I think) and from JVM
internals (like a livelock in J9 when you're trying to get a stack
trace from certain threads).

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Dawid Weiss
In reply to this post by Michael McCandless-2
> +1, I also miss simple streamed output option when running a test.
>
> And it's fine if it only works if you run one test, one JVM, etc.

There is an issue for 1-jvm streaming on github. Like I said -- it is
possible to decrease memory consumption (stdout buffering)  although
there will always be limitations (like if you run
-Dtests.iters=10000000 this results in 10000000 physical instances of
a Description object that need to be at once in memory; JUnit
architecture -- nothing I can do about it).

I will get back to this when I come back from vacation. In the
meantime I think I provided enough workarounds so that you can
continue working? I mean Solr mostly uses java logging and redirecting
this to a file is trivial. Mike -- you can do the same thing if you
need tons of logs. There is also an option of doing:

ANT_OPTS="-Xmx100g"

or whatever amount of memory you have -- this is crappy in the sense
that you'll see the output only after the test finishes but it also
should work in the mean time until I do that refactoring.

I admit harnessing threading was my priority and I am pretty happy
with the result although time will tell how usefl There are nuances
and blockers from doing this right resulting from JUnit infrastructure
that are mind boggling, eh, long story.

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: test buffering causes OOM errors

Dawid Weiss
Just for completeness -- something I forgot about.

There is one more option that I forgot about -- that <junit4> ANT
target is meant to simplify splitting suites into multiple JVMs and
providing nicer integration with the randomized runner (seed passing
etc). You can still use a regular ANT's <junit> to run the tests
(sequentially) and this will provide you with an immediate pipe of the
test's output to the console. This doesn't mean that it won't OOM on
large outputs because test's outputs are passed as Strings to report
instances (so it will OOM sooner or later).

D.

On Sun, Jul 22, 2012 at 8:53 AM, Dawid Weiss
<[hidden email]> wrote:

>> +1, I also miss simple streamed output option when running a test.
>>
>> And it's fine if it only works if you run one test, one JVM, etc.
>
> There is an issue for 1-jvm streaming on github. Like I said -- it is
> possible to decrease memory consumption (stdout buffering)  although
> there will always be limitations (like if you run
> -Dtests.iters=10000000 this results in 10000000 physical instances of
> a Description object that need to be at once in memory; JUnit
> architecture -- nothing I can do about it).
>
> I will get back to this when I come back from vacation. In the
> meantime I think I provided enough workarounds so that you can
> continue working? I mean Solr mostly uses java logging and redirecting
> this to a file is trivial. Mike -- you can do the same thing if you
> need tons of logs. There is also an option of doing:
>
> ANT_OPTS="-Xmx100g"
>
> or whatever amount of memory you have -- this is crappy in the sense
> that you'll see the output only after the test finishes but it also
> should work in the mean time until I do that refactoring.
>
> I admit harnessing threading was my priority and I am pretty happy
> with the result although time will tell how usefl There are nuances
> and blockers from doing this right resulting from JUnit infrastructure
> that are mind boggling, eh, long story.
>
> Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]