Hadoop RPC call response post processing

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop RPC call response post processing

Stefan Groschupf
Hi All,
I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor slot that allows me to handle a RPC call response writable after it was send over the wire.
Does anybody has an idea how break into the RPC code from outside. All the interesting methods are private. :(

Background:
Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems  that the garbage collect can get so busy that the jvm almost freezes for seconds. Things like zookeeper sessions time out in that cases.
My idea is to create an object pool for writables. Borrowing an object from the pool is simple since this happen in our custom code, though we do know when the writable return was send over the wire and can be returned into the pool.
A dirty hack would be to overwrite the write(out) method in the writable, assuming that is the last thing done with the writable, though turns out that this method is called in other cases too, e.g. to measure throughput.

Any ideas?

Thanks,
Stefan
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Todd Lipcon
Hi Stefan,

Sounds interesting.

Maybe you're looking for o.a.h.ipc.Server$Responder?

-Todd

On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:

> Hi All,
> I'm browsing the RPC code since quite a while now trying to find any entry
> point / interceptor slot that allows me to handle a RPC call response
> writable after it was send over the wire.
> Does anybody has an idea how break into the RPC code from outside. All the
> interesting methods are private. :(
>
> Background:
> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
> multiple systems  that the garbage collect can get so busy that the jvm
> almost freezes for seconds. Things like zookeeper sessions time out in that
> cases.
> My idea is to create an object pool for writables. Borrowing an object from
> the pool is simple since this happen in our custom code, though we do know
> when the writable return was send over the wire and can be returned into the
> pool.
> A dirty hack would be to overwrite the write(out) method in the writable,
> assuming that is the last thing done with the writable, though turns out
> that this method is called in other cases too, e.g. to measure throughput.
>
> Any ideas?
>
> Thanks,
> Stefan




--
Todd Lipcon
Software Engineer, Cloudera
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Ted Dunning-5
In reply to this post by Stefan Groschupf
I would be very surprised if allocation itself is the problem as opposed to
good old fashioned excess copying.

It is very hard to write an allocator faster than the java generational gc,
especially if you are talking about objects that are ephemeral.

Have you looked at the tenuring distribution?

On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:

> Hi All,
> I'm browsing the RPC code since quite a while now trying to find any entry
> point / interceptor slot that allows me to handle a RPC call response
> writable after it was send over the wire.
> Does anybody has an idea how break into the RPC code from outside. All the
> interesting methods are private. :(
>
> Background:
> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
> multiple systems  that the garbage collect can get so busy that the jvm
> almost freezes for seconds. Things like zookeeper sessions time out in that
> cases.
> My idea is to create an object pool for writables. Borrowing an object from
> the pool is simple since this happen in our custom code, though we do know
> when the writable return was send over the wire and can be returned into the
> pool.
> A dirty hack would be to overwrite the write(out) method in the writable,
> assuming that is the last thing done with the writable, though turns out
> that this method is called in other cases too, e.g. to measure throughput.
>
> Any ideas?
>
> Thanks,
> Stefan
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Stefan Groschupf
Hi Ted,
I don't think the problem is allocation but garbage collection.
When the gc kicks in everything freezes. Of course changing the gc algorithm helps a little.
Stefan



On Dec 27, 2010, at 11:21 PM, Ted Dunning wrote:

> I would be very surprised if allocation itself is the problem as opposed to
> good old fashioned excess copying.
>
> It is very hard to write an allocator faster than the java generational gc,
> especially if you are talking about objects that are ephemeral.
>
> Have you looked at the tenuring distribution?
>
> On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:
>
>> Hi All,
>> I'm browsing the RPC code since quite a while now trying to find any entry
>> point / interceptor slot that allows me to handle a RPC call response
>> writable after it was send over the wire.
>> Does anybody has an idea how break into the RPC code from outside. All the
>> interesting methods are private. :(
>>
>> Background:
>> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
>> multiple systems  that the garbage collect can get so busy that the jvm
>> almost freezes for seconds. Things like zookeeper sessions time out in that
>> cases.
>> My idea is to create an object pool for writables. Borrowing an object from
>> the pool is simple since this happen in our custom code, though we do know
>> when the writable return was send over the wire and can be returned into the
>> pool.
>> A dirty hack would be to overwrite the write(out) method in the writable,
>> assuming that is the last thing done with the writable, though turns out
>> that this method is called in other cases too, e.g. to measure throughput.
>>
>> Any ideas?
>>
>> Thanks,
>> Stefan

Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Stefan Groschupf
In reply to this post by Todd Lipcon
Hi Todd,
Right, that is the code I'm looking into. Though Responder is inner private class and is created  "responder = new Responder();"
It would be great if the Responder implementation could be configured.
Do you have any idea how to overwrite the Responder?
Thanks,
Stefan


On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:

> Hi Stefan,
>
> Sounds interesting.
>
> Maybe you're looking for o.a.h.ipc.Server$Responder?
>
> -Todd
>
> On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:
>
>> Hi All,
>> I'm browsing the RPC code since quite a while now trying to find any entry
>> point / interceptor slot that allows me to handle a RPC call response
>> writable after it was send over the wire.
>> Does anybody has an idea how break into the RPC code from outside. All the
>> interesting methods are private. :(
>>
>> Background:
>> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
>> multiple systems  that the garbage collect can get so busy that the jvm
>> almost freezes for seconds. Things like zookeeper sessions time out in that
>> cases.
>> My idea is to create an object pool for writables. Borrowing an object from
>> the pool is simple since this happen in our custom code, though we do know
>> when the writable return was send over the wire and can be returned into the
>> pool.
>> A dirty hack would be to overwrite the write(out) method in the writable,
>> assuming that is the last thing done with the writable, though turns out
>> that this method is called in other cases too, e.g. to measure throughput.
>>
>> Any ideas?
>>
>> Thanks,
>> Stefan
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Ted Dunning-5
In reply to this post by Stefan Groschupf
Knowing the tenuring distribution will tell a lot about that exact issue.
 Ephemeral collections take on average less than one instruction per
allocation and the allocation itself is generally only a single instruction.
 For ephemeral garbage, it is extremely unlikely that you can beat that.

So the real question is whether you are actually creating so much garbage
that you are over-whelming the collector or whether the data is much longer
lived than it should be. *That* can cause lots of collection costs.

To tell how long data lives, you need to get the tenuring distribution:

-XX:+PrintTenuringDistribution Prints details about the tenuring
distribution to standard out. It can be used to show this threshold and the
ages of objects in the new generation. It is also useful for observing the
lifetime distribution of an application.
On Tue, Dec 28, 2010 at 11:59 AM, Stefan Groschupf <[hidden email]> wrote:

> I don't think the problem is allocation but garbage collection.
>
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Todd Lipcon
In reply to this post by Stefan Groschupf
On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <[hidden email]> wrote:

> Hi Todd,
> Right, that is the code I'm looking into. Though Responder is inner private
> class and is created  "responder = new Responder();"
> It would be great if the Responder implementation could be configured.
> Do you have any idea how to overwrite the Responder?
>

Nope, it's not currently pluggable, nor do I think there's any compelling
reason to make it pluggable. It's coupled quite tightly to the
implementation right now.

Perhaps you can hack something in a git branch, and if it has good results
on something like NNBench it could be a general contribution?

-Todd


> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
>
> > Hi Stefan,
> >
> > Sounds interesting.
> >
> > Maybe you're looking for o.a.h.ipc.Server$Responder?
> >
> > -Todd
> >
> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:
> >
> >> Hi All,
> >> I'm browsing the RPC code since quite a while now trying to find any
> entry
> >> point / interceptor slot that allows me to handle a RPC call response
> >> writable after it was send over the wire.
> >> Does anybody has an idea how break into the RPC code from outside. All
> the
> >> interesting methods are private. :(
> >>
> >> Background:
> >> Heavy use of the RPC allocates hugh amount of Writable objects. We saw
> in
> >> multiple systems  that the garbage collect can get so busy that the jvm
> >> almost freezes for seconds. Things like zookeeper sessions time out in
> that
> >> cases.
> >> My idea is to create an object pool for writables. Borrowing an object
> from
> >> the pool is simple since this happen in our custom code, though we do
> know
> >> when the writable return was send over the wire and can be returned into
> the
> >> pool.
> >> A dirty hack would be to overwrite the write(out) method in the
> writable,
> >> assuming that is the last thing done with the writable, though turns out
> >> that this method is called in other cases too, e.g. to measure
> throughput.
> >>
> >> Any ideas?
> >>
> >> Thanks,
> >> Stefan
> >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


--
Todd Lipcon
Software Engineer, Cloudera
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Lance Norskog-2
Are you connecting to this JVM with RMI? RMI does a very nasty thing
with garbage collection: it forces a blocking collection every 60
seconds. Really. You have to change this with a system property.

On Tue, Dec 28, 2010 at 5:56 PM, Todd Lipcon <[hidden email]> wrote:

> On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <[hidden email]> wrote:
>
>> Hi Todd,
>> Right, that is the code I'm looking into. Though Responder is inner private
>> class and is created  "responder = new Responder();"
>> It would be great if the Responder implementation could be configured.
>> Do you have any idea how to overwrite the Responder?
>>
>
> Nope, it's not currently pluggable, nor do I think there's any compelling
> reason to make it pluggable. It's coupled quite tightly to the
> implementation right now.
>
> Perhaps you can hack something in a git branch, and if it has good results
> on something like NNBench it could be a general contribution?
>
> -Todd
>
>
>> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
>>
>> > Hi Stefan,
>> >
>> > Sounds interesting.
>> >
>> > Maybe you're looking for o.a.h.ipc.Server$Responder?
>> >
>> > -Todd
>> >
>> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]> wrote:
>> >
>> >> Hi All,
>> >> I'm browsing the RPC code since quite a while now trying to find any
>> entry
>> >> point / interceptor slot that allows me to handle a RPC call response
>> >> writable after it was send over the wire.
>> >> Does anybody has an idea how break into the RPC code from outside. All
>> the
>> >> interesting methods are private. :(
>> >>
>> >> Background:
>> >> Heavy use of the RPC allocates hugh amount of Writable objects. We saw
>> in
>> >> multiple systems  that the garbage collect can get so busy that the jvm
>> >> almost freezes for seconds. Things like zookeeper sessions time out in
>> that
>> >> cases.
>> >> My idea is to create an object pool for writables. Borrowing an object
>> from
>> >> the pool is simple since this happen in our custom code, though we do
>> know
>> >> when the writable return was send over the wire and can be returned into
>> the
>> >> pool.
>> >> A dirty hack would be to overwrite the write(out) method in the
>> writable,
>> >> assuming that is the last thing done with the writable, though turns out
>> >> that this method is called in other cases too, e.g. to measure
>> throughput.
>> >>
>> >> Any ideas?
>> >>
>> >> Thanks,
>> >> Stefan
>> >
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



--
Lance Norskog
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Stefan Groschupf
In reply to this post by Todd Lipcon
Hi Todd,
Thanks for the feedback.
> Nope, it's not currently pluggable, nor do I think there's any compelling
> reason to make it pluggable.

Well, one could argue with an interceptor / filter it would be very easy to add compression or encryption to RPC.
But since the nutch days the code base was never architected in extendable or modular way.

> Perhaps you can hack something in a git branch, and if it has good results
> on something like NNBench it could be a general contribution?

Thanks - I pass on that offer. The days waiting a half year to get a patch into the codebase are behind me. :)
I think I will just replace hadoop RPC with netty.

Cheers,
Stefan
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop RPC call response post processing

Ted Dunning-5
In reply to this post by Lance Norskog-2
Nah... this is hadoop RPC

On Tue, Dec 28, 2010 at 6:42 PM, Lance Norskog <[hidden email]> wrote:

> Are you connecting to this JVM with RMI? RMI does a very nasty thing
> with garbage collection: it forces a blocking collection every 60
> seconds. Really. You have to change this with a system property.
>
> On Tue, Dec 28, 2010 at 5:56 PM, Todd Lipcon <[hidden email]> wrote:
> > On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <[hidden email]> wrote:
> >
> >> Hi Todd,
> >> Right, that is the code I'm looking into. Though Responder is inner
> private
> >> class and is created  "responder = new Responder();"
> >> It would be great if the Responder implementation could be configured.
> >> Do you have any idea how to overwrite the Responder?
> >>
> >
> > Nope, it's not currently pluggable, nor do I think there's any compelling
> > reason to make it pluggable. It's coupled quite tightly to the
> > implementation right now.
> >
> > Perhaps you can hack something in a git branch, and if it has good
> results
> > on something like NNBench it could be a general contribution?
> >
> > -Todd
> >
> >
> >> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
> >>
> >> > Hi Stefan,
> >> >
> >> > Sounds interesting.
> >> >
> >> > Maybe you're looking for o.a.h.ipc.Server$Responder?
> >> >
> >> > -Todd
> >> >
> >> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <[hidden email]>
> wrote:
> >> >
> >> >> Hi All,
> >> >> I'm browsing the RPC code since quite a while now trying to find any
> >> entry
> >> >> point / interceptor slot that allows me to handle a RPC call response
> >> >> writable after it was send over the wire.
> >> >> Does anybody has an idea how break into the RPC code from outside.
> All
> >> the
> >> >> interesting methods are private. :(
> >> >>
> >> >> Background:
> >> >> Heavy use of the RPC allocates hugh amount of Writable objects. We
> saw
> >> in
> >> >> multiple systems  that the garbage collect can get so busy that the
> jvm
> >> >> almost freezes for seconds. Things like zookeeper sessions time out
> in
> >> that
> >> >> cases.
> >> >> My idea is to create an object pool for writables. Borrowing an
> object
> >> from
> >> >> the pool is simple since this happen in our custom code, though we do
> >> know
> >> >> when the writable return was send over the wire and can be returned
> into
> >> the
> >> >> pool.
> >> >> A dirty hack would be to overwrite the write(out) method in the
> >> writable,
> >> >> assuming that is the last thing done with the writable, though turns
> out
> >> >> that this method is called in other cases too, e.g. to measure
> >> throughput.
> >> >>
> >> >> Any ideas?
> >> >>
> >> >> Thanks,
> >> >> Stefan
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Todd Lipcon
> >> > Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Lance Norskog
> [hidden email]
>