Namenode Unable to Authenticate to QJM in Secure mode.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Namenode Unable to Authenticate to QJM in Secure mode.

Akash Mishra
Hi *, 

I am trying to run Hadoop cluster [ 2.7.1] in Secure mode. In my cluster Namenode is failing while restart with 

2016-08-19 10:34:49,754 DEBUG org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using fallback authenticator sequence.
2016-08-19 10:34:49,774 DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hdfs/[hidden email] (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
2016-08-19 10:34:49,775 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception initializing http://hadoopdev1:8480/getJournal?jid=hadoopdev&segmentTxId=2275460&storageInfo=-63%3A1455401088%3A1444912570574%3ACID-f748dfef-c174-4d19-8d18-43b74552c8e6
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)


I am using MIT 5 Kerberos. I am able to successfully kinit using keytab file. I have DEBUG log enabled and attaching log from Namenode [nn.log]  and one of QJM [ qjm.log] 



Thanks.




--

Regards,
Akash Mishra.


"It's not our abilities that make us, but our decisions."--Albus Dumbledore



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

qjm.log (61K) Download Attachment
nn.log (100K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Namenode Unable to Authenticate to QJM in Secure mode.

Rakesh Radhakrishnan-2
Hi Akash,

In general "GSSException: No valid credentials provided" means you don’t have valid Kerberos credentials. I'm suspecting some issues related to spnego, could you please revisit all of your kerb related configurations, probably you can start from the below configuration. Please share *-site.xml configurations of JN and NNs. Also, please check any unexpected exceptions in KDC server logs.

I've filtered out "REQUEST /getJournal on org.mortbay.jetty.HttpConnection"  in your "qjm.log" log file and I could see this has came immediately after your restart, few has succeeded and few others failed with this exception.

2016-08-19 10:34:14,345 DEBUG org.mortbay.log: RESPONSE /getJournal  401
2016-08-19 10:34:14,374 DEBUG org.mortbay.log: RESPONSE /getJournal  403
2016-08-19 10:34:14,382 DEBUG org.mortbay.log: RESPONSE /getJournal  401
2016-08-19 10:34:14,398 DEBUG org.mortbay.log: RESPONSE /getJournal  403
2016-08-19 10:34:49,679 DEBUG org.mortbay.log: RESPONSE /getJournal  401

<property>
  <name>dfs.journalnode.kerberos.internal.spnego.principal</name>
  <value></value>
  <description>
    The server principal used by the JournalNode HTTP Server for
    SPNEGO authentication when Kerberos security is enabled. This is
    typically set to HTTP/[hidden email]. The SPNEGO server principal
    begins with the prefix HTTP/ by convention.

    If the value is '*', the web server will attempt to login with
    every principal specified in the keytab file
    dfs.web.authentication.kerberos.keytab.

    For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}
    i.e use the value of dfs.web.authentication.kerberos.principal.
  </description>
</property>


Rakesh,
Intel

On Fri, Aug 19, 2016 at 4:15 PM, Akash Mishra <[hidden email]> wrote:
Hi *, 

I am trying to run Hadoop cluster [ 2.7.1] in Secure mode. In my cluster Namenode is failing while restart with 

2016-08-19 10:34:49,754 DEBUG org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using fallback authenticator sequence.
2016-08-19 10:34:49,774 DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hdfs/hadoopdev1.mlan@HADOOPDEV.MLAN (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
2016-08-19 10:34:49,775 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception initializing http://hadoopdev1:8480/getJournal?jid=hadoopdev&segmentTxId=2275460&storageInfo=-63%3A1455401088%3A1444912570574%3ACID-f748dfef-c174-4d19-8d18-43b74552c8e6
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)


I am using MIT 5 Kerberos. I am able to successfully kinit using keytab file. I have DEBUG log enabled and attaching log from Namenode [nn.log]  and one of QJM [ qjm.log] 



Thanks.




--

Regards,
Akash Mishra.


"It's not our abilities that make us, but our decisions."--Albus Dumbledore



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]