DistributedSearch$Client.updateSegments() blocking other threads

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

DistributedSearch$Client.updateSegments() blocking other threads

Andrzej Białecki-2
Hi,

I was doing performance testing of a distributed search setup, with
JMeter, using the code from trunk/.

Whenever one of the backend Servers goes down, there is a hiccup on the
frontend, because all ParallelCalls started by the Client, which still
use that dead address, need to timeout. This is expected, and acceptable.

New calls being made in the meantime (before updateSegments() discovers
that the host is down) will also need to timeout - which is so so, I
think it could be improved by removing the offending address at the
first sign of trouble, i.e. not to wait for updateSegments() but
immediately remove the dead host from liveAddresses. Anyway, read on...

What was curious was that the same hiccup would then occur every 10
seconds, which is the hardcoded interval for calling
Client.updateSegments(). It was as if the call to updateSegments() was
synchronized on the whole class, so that all other calls are blocked
until updateSegments() completes. I modified the code, so that instead
of using DistributedSearch$Client itself as a Thread instance, a new
independent Thread instance is created.

The hiccups are gone now - the list of liveAddresses is still being
updated as it should whenever Servers go down/up, but now
updateSegments() doesn't interfere with other calls. I attach the patch
- but to be honest I'm still not quite sure what was happening...

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Index: DistributedSearch.java
===================================================================
--- DistributedSearch.java (revision 280515)
+++ DistributedSearch.java (working copy)
@@ -112,8 +112,9 @@
     public Client(InetSocketAddress[] addresses) throws IOException {
       this.defaultAddresses = addresses;
       updateSegments();
-      setDaemon(true);
-      start();
+      Thread t = new Thread(this);
+      t.setDaemon(true);
+      t.start();
     }
     
     private static final Method GET_SEGMENTS;
@@ -168,8 +169,10 @@
         liveSegments+=segments.length;
       }
 
-      this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
-        liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
+      synchronized(this.liveAddresses) {
+        this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
+          liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
+      }
 
       LOG.info("STATS: "+liveServers+" servers, "+liveSegments+" segments.");
     }
Reply | Threaded
Open this post in threaded view
|

Re: DistributedSearch$Client.updateSegments() blocking other threads

Piotr Kosiorowski
Hello Andrzej,
You can also try http://issues.apache.org/jira/browse/NUTCH-79
- I think it should also help here - it is a bit complicated as it
contain additional functionality but if you have any problems I am
willing to help. I am going to perform some test of it again and maybe
commit it in some time if others think it is worth it.
Regrads
Piotr

Andrzej Bialecki wrote:

> Hi,
>
> I was doing performance testing of a distributed search setup, with
> JMeter, using the code from trunk/.
>
> Whenever one of the backend Servers goes down, there is a hiccup on the
> frontend, because all ParallelCalls started by the Client, which still
> use that dead address, need to timeout. This is expected, and acceptable.
>
> New calls being made in the meantime (before updateSegments() discovers
> that the host is down) will also need to timeout - which is so so, I
> think it could be improved by removing the offending address at the
> first sign of trouble, i.e. not to wait for updateSegments() but
> immediately remove the dead host from liveAddresses. Anyway, read on...
>
> What was curious was that the same hiccup would then occur every 10
> seconds, which is the hardcoded interval for calling
> Client.updateSegments(). It was as if the call to updateSegments() was
> synchronized on the whole class, so that all other calls are blocked
> until updateSegments() completes. I modified the code, so that instead
> of using DistributedSearch$Client itself as a Thread instance, a new
> independent Thread instance is created.
>
> The hiccups are gone now - the list of liveAddresses is still being
> updated as it should whenever Servers go down/up, but now
> updateSegments() doesn't interfere with other calls. I attach the patch
> - but to be honest I'm still not quite sure what was happening...
>
>
> ------------------------------------------------------------------------
>
> Index: DistributedSearch.java
> ===================================================================
> --- DistributedSearch.java (revision 280515)
> +++ DistributedSearch.java (working copy)
> @@ -112,8 +112,9 @@
>      public Client(InetSocketAddress[] addresses) throws IOException {
>        this.defaultAddresses = addresses;
>        updateSegments();
> -      setDaemon(true);
> -      start();
> +      Thread t = new Thread(this);
> +      t.setDaemon(true);
> +      t.start();
>      }
>      
>      private static final Method GET_SEGMENTS;
> @@ -168,8 +169,10 @@
>          liveSegments+=segments.length;
>        }
>  
> -      this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> -        liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      synchronized(this.liveAddresses) {
> +        this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> +          liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      }
>  
>        LOG.info("STATS: "+liveServers+" servers, "+liveSegments+" segments.");
>      }