Concurrent builds

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Concurrent builds

Tilman Hausherr

I can't do concurrent builds of trunk and 1.* version on my pc. While I don't know if these are all the causes, I identified at least two reasons:

ForkParserIntegrationTest.java:

address=54321

TikaServerIntegrationTest.java:

INTEGRATION_TEST_PORT = "9999";

It still fails for me after I changed the port of the trunk build to 9997. The trunk build now has this:

org.junit.runners.model.TestTimedOutException: test timed out after 60000 milliseconds
    at org.apache.tika.server.TikaServerIntegrationTest.awaitServerStartup(TikaServerIntegrationTest.java:435)
    at org.apache.tika.server.TikaServerIntegrationTest.testBaseline(TikaServerIntegrationTest.java:517)
    at org.apache.tika.server.TikaServerIntegrationTest.testTimeout(TikaServerIntegrationTest.java:286)

I looked into the log, there is "tika test client failed to connect to server with status: 404" many times until it gives up.

I also tried with the browser, http://localhost:9997/tika  shows 404.

Could it be that both jetty instances use something in common?

Has this (not being able to run the builds on the same machine) been discussed before?

Tilman

Reply | Threaded
Open this post in threaded view
|

Re: Concurrent builds

Tim Allison
Hi Tilman,
  Thank you for raising this!  I don't think we have discussed this.  I've noticed it and build in sequence or at least let one build get far enough along that it will finish the tika-server module before the other one gets there.  As you point out, there is port contention in tika-server (all the tests) and in the one (?) ForkParser test.
  I've been finding that some of the tests in TikaServerIntegrationTest are flaky even on a single build, and that is a separate issue that I need to pin down and fix.
  I don't _think_ jetty is doing something hidden under the hood, but we definitely have port contention if you're running two builds in parallel.  Do you have any recommendations to fix this?
  Thank you for raising this!

       Best,

              Tim


On Sun, Nov 3, 2019 at 11:35 AM Tilman Hausherr <[hidden email]> wrote:

I can't do concurrent builds of trunk and 1.* version on my pc. While I don't know if these are all the causes, I identified at least two reasons:

ForkParserIntegrationTest.java:

address=54321

TikaServerIntegrationTest.java:

INTEGRATION_TEST_PORT = "9999";

It still fails for me after I changed the port of the trunk build to 9997. The trunk build now has this:

org.junit.runners.model.TestTimedOutException: test timed out after 60000 milliseconds
    at org.apache.tika.server.TikaServerIntegrationTest.awaitServerStartup(TikaServerIntegrationTest.java:435)
    at org.apache.tika.server.TikaServerIntegrationTest.testBaseline(TikaServerIntegrationTest.java:517)
    at org.apache.tika.server.TikaServerIntegrationTest.testTimeout(TikaServerIntegrationTest.java:286)

I looked into the log, there is "tika test client failed to connect to server with status: 404" many times until it gives up.

I also tried with the browser, http://localhost:9997/tika  shows 404.

Could it be that both jetty instances use something in common?

Has this (not being able to run the builds on the same machine) been discussed before?

Tilman

Reply | Threaded
Open this post in threaded view
|

Re: Concurrent builds

Tilman Hausherr
Am 06.11.2019 um 16:07 schrieb Tim Allison:

> Hi Tilman,
>   Thank you for raising this!  I don't think we have discussed this. 
> I've noticed it and build in sequence or at least let one build get
> far enough along that it will finish the tika-server module before the
> other one gets there.  As you point out, there is port contention in
> tika-server (all the tests) and in the one (?) ForkParser test.
>   I've been finding that some of the tests in
> TikaServerIntegrationTest are flaky even on a single build, and that
> is a separate issue that I need to pin down and fix.
>   I don't _think_ jetty is doing something hidden under the hood, but
> we definitely have port contention if you're running two builds in
> parallel.  Do you have any recommendations to fix this?


The port thing is the two occurances I mentioned below... but the
problem is that this isn't enough.

I found where the logs are... in the trunk log, the server doesn't start
because

Could not start Jetty server on port 9.997: Failed to bind to
localhost/127.0.0.1:9997

But this happens only when doing the 2 whole builds. When running both
TikaServerIntegrationTest at the same time there are no problems.

It also fails when just testing the two "server" subprojects. (This
takes less time)

I also tried starting the server from the command line, it works fine
and there is a text that appears. But in the "fail" case, there is an
"empty jetty" that is on that port.

Tilman


>   Thank you for raising this!
>
>        Best,
>
>               Tim
>
>
> On Sun, Nov 3, 2019 at 11:35 AM Tilman Hausherr <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I can't do concurrent builds of trunk and 1.* version on my pc.
>     While I don't know if these are all the causes, I identified at
>     least two reasons:
>
>     ForkParserIntegrationTest.java:
>
>     address=54321
>
>     TikaServerIntegrationTest.java:
>
>     INTEGRATION_TEST_PORT = "9999";
>
>     It still fails for me after I changed the port of the trunk build
>     to 9997. The trunk build now has this:
>
>     org.junit.runners.model.TestTimedOutException: test timed out
>     after 60000 milliseconds
>         at
>     org.apache.tika.server.TikaServerIntegrationTest.awaitServerStartup(TikaServerIntegrationTest.java:435)
>         at
>     org.apache.tika.server.TikaServerIntegrationTest.testBaseline(TikaServerIntegrationTest.java:517)
>         at
>     org.apache.tika.server.TikaServerIntegrationTest.testTimeout(TikaServerIntegrationTest.java:286)
>
>     I looked into the log, there is "tika test client failed to
>     connect to server with status: 404" many times until it gives up.
>
>     I also tried with the browser, http://localhost:9997/tika shows 404.
>
>     Could it be that both jetty instances use something in common?
>
>     Has this (not being able to run the builds on the same machine)
>     been discussed before?
>
>     Tilman
>

Reply | Threaded
Open this post in threaded view
|

Re: Concurrent builds

Tilman Hausherr
I've found one possible cause... I was able to stop bind() from failing
by setting

-Dsun.net.useExclusiveBind=false

as shown on

https://stackoverflow.com/questions/44269863/

however I am no longer able to run the server tests, so I suspect I
destroyed something in all the mess I added while debugging LOL. (build
tests did work a week or two ago, so it must be my fault)

Tilman


Reply | Threaded
Open this post in threaded view
|

Re: Concurrent builds

Tim Allison
Hahahaha!  Thank you for digging.  I may have some time this week to take a
look.  I can almost guarantee that failed builds are not your fault!

On Mon, Nov 11, 2019 at 8:45 PM Tilman Hausherr <[hidden email]>
wrote:

> I've found one possible cause... I was able to stop bind() from failing
> by setting
>
> -Dsun.net.useExclusiveBind=false
>
> as shown on
>
> https://stackoverflow.com/questions/44269863/
>
> however I am no longer able to run the server tests, so I suspect I
> destroyed something in all the mess I added while debugging LOL. (build
> tests did work a week or two ago, so it must be my fault)
>
> Tilman
>
>
>