[jira] [Created] (HADOOP-14198) Should have a way to let PingInputStream to abort

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Created] (HADOOP-14198) Should have a way to let PingInputStream to abort

JIRA jira@apache.org
Yongjun Zhang created HADOOP-14198:

             Summary: Should have a way to let PingInputStream to abort
                 Key: HADOOP-14198
                 URL: https://issues.apache.org/jira/browse/HADOOP-14198
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Yongjun Zhang

We observed a case that RPC call get stuck, since PingInputStream does the following

 /** This class sends a ping to the remote side when timeout on
     * reading. If no failure is detected, it retries until at least
     * a byte is read.
    private class PingInputStream extends FilterInputStream {

It seems that in this case no data is ever received, and it keeps pinging.

Should we ping forever here? Maybe we should introduce a config to stop the ping after pinging for certain number of times, and report back timeout, let the caller to retry the RPC?

Wonder if there is chance the RPC get dropped somehow by the server so no response is ever received.

Thread 16127: (state = BLOCKED)                                                                    
 - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled frame)                  
 - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 (Compiled frame)      
 - org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) @bci=5, line=57 (Compiled frame)
 - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=35, line=142 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=161 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=131 (Compiled frame)
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame)              
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame)              
 - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=521 (Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)                          
 - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)                            
 - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)                              
 - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, line=1081 (Compiled frame)
 - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled frame)        


This message was sent by Atlassian JIRA

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]