New cluster help

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

New cluster help

tombin
I am setting up a new hadoop cluster for the first time.  My setup currently looks as follows:

hdfs cluster:
1 namenode
2 datanodes

hbase:
1 hbase node

zookeeper cluster:
3 zookeeper nodes

I have enabled ssl on the hdfs cluster.  When trying to connect from base i see the following error:

2016-07-14 19:38:58,333 WARN  [Thread-73] hdfs.DFSClient: DataStreamer Exception

java.lang.NullPointerException

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferEncryptor.getEncryptedStreams(DataTransferEncryptor.java:191)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1335)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2016-07-14 19:39:04,341 INFO  [hb01:16000.activeMasterManager] hdfs.DFSClient: Could not complete /hbase/.tmp/hbase.version retrying...



this will repeat several time and then i'll throw the following exception:


2016-07-14 19:39:58,772 FATAL [hb01:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.

java.io.IOException: Unable to close file because the last block does not have enough number of replicas.

at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2151)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2119)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:730)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:705)

at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:662)

at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)

at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)

at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)

at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:652)

at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:185)

at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1750)

at java.lang.Thread.run(Thread.java:745)


hbase shuts down at this point.


on the datanode i side i see the following in the logs that looks like it may be related:


2016-07-14 19:38:23,132 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hd03.domain.com:50010:DataXceiver error processing unknown operation  src: /10.0.0.10:34893 dst: /10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)

        at java.lang.Thread.run(Thread.java:745)

2016-07-14 19:39:33,575 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hd03.domain.com:50010:DataXceiver error processing unknown operation  src: /10.0.0.10:34898 dst: /10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)


Is this in relation to my ssl configuration ?
I'm confused about whats going on here.  Thank you in advance for any help.
Reply | Threaded
Open this post in threaded view
|

Re: New cluster help

Ravi Prakash-3
Hi Tombin!

Is this the first cluster you're ever setting up? Are you able to run an "hdfs dfs -ls /" successfully? How about putting files into HDFS? I'd take it one step at a time if I were you. i.e.

1. Set up a simple HDFS cluster (without SSL)
2. Turn on SSL
3. Then try to run HBase.

Is step 1 working for you?

Ravi

On Thu, Jul 14, 2016 at 12:59 PM, tombin <[hidden email]> wrote:
I am setting up a new hadoop cluster for the first time.  My setup currently looks as follows:

hdfs cluster:
1 namenode
2 datanodes

hbase:
1 hbase node

zookeeper cluster:
3 zookeeper nodes

I have enabled ssl on the hdfs cluster.  When trying to connect from base i see the following error:

2016-07-14 19:38:58,333 WARN  [Thread-73] hdfs.DFSClient: DataStreamer Exception

java.lang.NullPointerException

at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferEncryptor.getEncryptedStreams(DataTransferEncryptor.java:191)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1335)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2016-07-14 19:39:04,341 INFO  [hb01:16000.activeMasterManager] hdfs.DFSClient: Could not complete /hbase/.tmp/hbase.version retrying...



this will repeat several time and then i'll throw the following exception:


2016-07-14 19:39:58,772 FATAL [hb01:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.

java.io.IOException: Unable to close file because the last block does not have enough number of replicas.

at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2151)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2119)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:730)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:705)

at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:662)

at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)

at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)

at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)

at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:652)

at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:185)

at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1750)

at java.lang.Thread.run(Thread.java:745)


hbase shuts down at this point.


on the datanode i side i see the following in the logs that looks like it may be related:


2016-07-14 19:38:23,132 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hd03.domain.com:50010:DataXceiver error processing unknown operation  src: /10.0.0.10:34893 dst: /10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)

        at java.lang.Thread.run(Thread.java:745)

2016-07-14 19:39:33,575 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hd03.domain.com:50010:DataXceiver error processing unknown operation  src: /10.0.0.10:34898 dst: /10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)


Is this in relation to my ssl configuration ?
I'm confused about whats going on here.  Thank you in advance for any help.