Fwd: datanode is unable to connect to namenode

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: datanode is unable to connect to namenode

Aneela Saleem


Sent from my iPhone

Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

I have following two Kerberos Principals:

nn/hadoop-master
dn/hadoop-slave

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:
Probably sharing both Name node and datanode logs may help. 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:
Following is the result of telnet 

Trying 192.168.23.206...
Connected to hadoop-master.
Escape character is '^]'.

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:
Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN
tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED
tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED
tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost
#127.0.1.1      vm6-VirtualBox
192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox
192.168.23.207  hadoop-slave
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters


Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:
Hi all,

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

Rest of the things look fine. Please help me in this regard, what could be the issue?








datanode logs.txt (4K) Download Attachment
namenode.txt (44K) Download Attachment
hdfs-site.xml (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: datanode is unable to connect to namenode

Vinayakumar B-4

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To: [hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: datanode is unable to connect to namenode

Aneela Saleem
Thanks Vinayakumar

Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode
can connect to namenode successfully. 

So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many 
tutorials that the convention is to have different principals for all services like 
dn/_HOST for datanode 
nn/_HOST for namenode
sn/_HOST for secondarynamenode etc

Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml

Thanks 

On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <[hidden email]> wrote:

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To: [hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 


Reply | Threaded
Open this post in threaded view
|

Re: datanode is unable to connect to namenode

Gurmukh Singh

What you are doing is correct, but datanodes, in addition to the dn/_HOST principal also needs nn/_HOST principal.

Follow my github for configs from workign cluster: https://github.com/netxillon/hadoop/tree/master/kerberos



On 30/06/16 5:54 PM, Aneela Saleem wrote:
Thanks Vinayakumar

Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode
can connect to namenode successfully. 

So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many 
tutorials that the convention is to have different principals for all services like 
dn/_HOST for datanode 
nn/_HOST for namenode
sn/_HOST for secondarynamenode etc

Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml

Thanks 

On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <[hidden email]> wrote:

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To: [hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 



-- 
--
Thanks and Regards

Gurmukh Singh
Reply | Threaded
Open this post in threaded view
|

RE: datanode is unable to connect to namenode

Vinayakumar B-4
In reply to this post by Aneela Saleem

Please note, there are two different configs.

 

“dfs.datanode.kerberos.principal” and “dfs.namenode.kerberos.principal”

 

Following configs can be set, as required.

 

dfs.datanode.kerberos.principal à dn/_HOST

dfs.namenode.kerberos.principal  à nn/_HOST

 

“nn/_HOST” will be used only in namenode side.

 

-Vinay

From: Aneela Saleem [mailto:[hidden email]]
Sent: 30 June 2016 13:24
To: Vinayakumar B <[hidden email]>
Cc: [hidden email]
Subject: Re: datanode is unable to connect to namenode

 

Thanks Vinayakumar

 

Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode

can connect to namenode successfully. 

 

So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many 

tutorials that the convention is to have different principals for all services like 

dn/_HOST for datanode 

nn/_HOST for namenode

sn/_HOST for secondarynamenode etc

 

Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml

 

Thanks 

 

On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <[hidden email]> wrote:

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To:
[hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: datanode is unable to connect to namenode

Aneela Saleem
Thanks Vinaykumar and Gurmukh,

I have made it working successfully through auth_to_local configs. But i faced much issues. 

Actually I have two nodes cluster, one being namenode and datanode, and second being datanode only. I faced some authentication from keytab related issues. for example:

I added nn/hadoop-master to both nn.keytab and dn.keytab and did the same with dn/hadoop-slave (following you github dn.keytab file). but when i start cluster i got following error:

Login failure for nn/hadoop-master@platalyticsrealm from keytab /etc/hadoop/conf/hdfs.keytab: javax.security.auth.login.LoginException: Checksum failed

i verified the authentication of nn/hadoop-master with keytab nn/hadoop-master through kinit, but couldn't do so. because of error like could not verify credentials

i then removed nn/hadoop-master from dn.keytab then it authenticated successfully. And i removed all the hadoop-master principals from dn.keytab and hadoop-slave principals from nn.keytab.  So does it mean that a principal can't belong to more than one keytabs? And please make some time to review the attached hdfs-site.xml for both namenode and datanode, and keytab files. And point me out if something wrong.

Thanks


On Thu, Jun 30, 2016 at 1:21 PM, Vinayakumar B <[hidden email]> wrote:

Please note, there are two different configs.

 

“dfs.datanode.kerberos.principal” and “dfs.namenode.kerberos.principal”

 

Following configs can be set, as required.

 

dfs.datanode.kerberos.principal à dn/_HOST

dfs.namenode.kerberos.principal  à nn/_HOST

 

“nn/_HOST” will be used only in namenode side.

 

-Vinay

From: Aneela Saleem [mailto:[hidden email]]
Sent: 30 June 2016 13:24
To: Vinayakumar B <[hidden email]>
Cc: [hidden email]


Subject: Re: datanode is unable to connect to namenode

 

Thanks Vinayakumar

 

Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode

can connect to namenode successfully. 

 

So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many 

tutorials that the convention is to have different principals for all services like 

dn/_HOST for datanode 

nn/_HOST for namenode

sn/_HOST for secondarynamenode etc

 

Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml

 

Thanks 

 

On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <[hidden email]> wrote:

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To:
[hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 

 




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dn.keytab (1K) Download Attachment
hdfs-site(master node).xml (5K) Download Attachment
hdfs-site(slave-node).xml (5K) Download Attachment
nn.keytab (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: datanode is unable to connect to namenode

Gurmukh Singh

Have you configured JCE ?


On 01/07/16 6:36 AM, Aneela Saleem wrote:
Thanks Vinaykumar and Gurmukh,

I have made it working successfully through auth_to_local configs. But i faced much issues. 

Actually I have two nodes cluster, one being namenode and datanode, and second being datanode only. I faced some authentication from keytab related issues. for example:

I added nn/hadoop-master to both nn.keytab and dn.keytab and did the same with dn/hadoop-slave (following you github dn.keytab file). but when i start cluster i got following error:

Login failure for nn/hadoop-master@platalyticsrealm from keytab /etc/hadoop/conf/hdfs.keytab: javax.security.auth.login.LoginException: Checksum failed

i verified the authentication of nn/hadoop-master with keytab nn/hadoop-master through kinit, but couldn't do so. because of error like could not verify credentials

i then removed nn/hadoop-master from dn.keytab then it authenticated successfully. And i removed all the hadoop-master principals from dn.keytab and hadoop-slave principals from nn.keytab.  So does it mean that a principal can't belong to more than one keytabs? And please make some time to review the attached hdfs-site.xml for both namenode and datanode, and keytab files. And point me out if something wrong.

Thanks


On Thu, Jun 30, 2016 at 1:21 PM, Vinayakumar B <[hidden email]> wrote:

Please note, there are two different configs.

 

“dfs.datanode.kerberos.principal” and “dfs.namenode.kerberos.principal”

 

Following configs can be set, as required.

 

dfs.datanode.kerberos.principal à dn/_HOST

dfs.namenode.kerberos.principal  à nn/_HOST

 

“nn/_HOST” will be used only in namenode side.

 

-Vinay

From: Aneela Saleem [mailto:[hidden email]]
Sent: 30 June 2016 13:24
To: Vinayakumar B <[hidden email]>
Cc: [hidden email]


Subject: Re: datanode is unable to connect to namenode

 

Thanks Vinayakumar

 

Yes you got it right i was using different principal names i.e., nn/_HOST for namenode and dn/_HOST for datanode. Setting the same principal name for both datanode and namenode i.e., hdfs/_HOST@platalyticsrealm solved the issue. Now datanode

can connect to namenode successfully. 

 

So my question is, is it mandatory to have same principal name on all hosts i.e., hdfs/_HOST@platalyticsrealm, because i found in many 

tutorials that the convention is to have different principals for all services like 

dn/_HOST for datanode 

nn/_HOST for namenode

sn/_HOST for secondarynamenode etc

 

Secondly for map reduce and yarn, would that mapred-site.xml and yarn-site.xml be same on all cluster nodes? just like for hdfs-site.xml

 

Thanks 

 

On Thu, Jun 30, 2016 at 10:51 AM, Vinayakumar B <[hidden email]> wrote:

Hi Aneela,

 

1. Looks like you have attached the hdfs-site.xml from 'hadoop-master' node. For this node datanode connection is successfull as mentioned in below logs.

 

         2016-06-29 10:01:35,700 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:35,744 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for nn/hadoop-master@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol

 

         2016-06-29 10:01:36,845 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.23.206:1004

 

 

2. For the other node, 'hadoop-slave' kerberos athentication is successfull, but ServiceAuthorizationManager check failed.

 

2016-06-29 10:01:37,474 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

2016-06-29 10:01:37,512 WARN SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization failed for dn/hadoop-slave@platalyticsrealm (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol, expected client Kerberos principal is nn/hadoop-slave@platalyticsrealm

2016-06-29 10:01:37,514 INFO org.apache.hadoop.ipc.Server: Connection from 192.168.23.207:32807 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/hadoop-slave@platalyticsrealm (auth:KERBEROS)

 

reason could be mostly, "dfs.datanode.kerberos.principal" configuration in both nodes differ. I can see that this configuration in hadoop-master's hdfs-site.xml set to 'nn/_HOST@platalyticsrealm' but it might have been set to 'dn/_HOST@platalyticsrealm' in hadoop-slave node's configurations.

 

Please change this configuration in all nodes to 'dn/_HOST@platalyticsrealm' and restart all NNs and DNs, and check again.

 

If this does not help, then please share the hdfs-site.xml of hadoop-slave node too.

 

-Vinay

 

From: Aneela Saleem [mailto:[hidden email]]
Sent: 29 June 2016 21:35
To:
[hidden email]
Subject: Fwd: datanode is unable to connect to namenode

 



Sent from my iPhone


Begin forwarded message:

From: Aneela Saleem <[hidden email]>
Date: 29 June 2016 at 10:16:36 GMT+5
To: "sreebalineni ." <[hidden email]>
Subject: Re: datanode is unable to connect to namenode

Attached are the log files for datanode and namenode. Also i have attached hdfs-site.xml for namenode please check if there are any issues in configuration file.

 

I have following two Kerberos Principals:

 

nn/hadoop-master

dn/hadoop-slave

 

i have copied kdc.conf and krb5.conf on both nodes. Also i copied keytab file on datanode. And i have starting services with principal nn/hadoop-master.

 

On Wed, Jun 29, 2016 at 9:35 AM, sreebalineni . <[hidden email]> wrote:

Probably sharing both Name node and datanode logs may help. 

 

On Wed, Jun 29, 2016 at 10:02 AM, Aneela Saleem <[hidden email]> wrote:

Following is the result of telnet 

 

Trying 192.168.23.206...

Connected to hadoop-master.

Escape character is '^]'.

 

On Wed, Jun 29, 2016 at 3:57 AM, Aneela Saleem <[hidden email]> wrote:

Thanks Sreebalineni for the response.

This is the result of the netstat -a | grep 8020 command

tcp        0      0 hadoop-master:8020      *:*                     LISTEN

tcp        0      0 hadoop-master:33356     hadoop-master:8020      ESTABLISHED

tcp        0      0 hadoop-master:8020      hadoop-master:33356     ESTABLISHED

tcp        0      0 hadoop-master:55135     hadoop-master:8020      TIME_WAIT

And this is my /etc/hosts file

#127.0.0.1      localhost

#127.0.1.1      vm6-VirtualBox

192.168.23.206  hadoop-master platalytics.com vm6-VirtualBox

192.168.23.207  hadoop-slave

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters



Can you please tell me what's wrong with above configuration and how can i check whether it is firewall issue?

Thanks

 

On Wed, Jun 29, 2016 at 12:11 AM, sreebalineni . <[hidden email]> wrote:

Are you able to telnet ping. Check the firewalls as well

On Jun 29, 2016 12:39 AM, "Aneela Saleem" <[hidden email]> wrote:

Hi all,

 

I have setup two nodes cluster with security enabled. I have everything running successful like namenode, datanode, resourcemanager, nodemanager, jobhistoryserver etc. But datanode is unable to connect to namenode, as i can see only one node on the web UI. checking logs of datanode gives following warning:

 

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/192.168.23.206:8020

 

Rest of the things look fine. Please help me in this regard, what could be the issue?

 

 

 

 

 




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

-- 
--
Thanks and Regards

Gurmukh Singh