RE: NFS Gateway - Secure Cluster - Mount Failed

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

RE: NFS Gateway - Secure Cluster - Mount Failed

kumar, Senthil(AWF)

+ Hadoop Users.

 

--Senthil

From: Senthil Kumar [mailto:[hidden email]]
Sent: Monday, August 29, 2016 4:22 PM
To: [hidden email]
Cc: Senthil kumar <[hidden email]>; Maliakkal Padmanabhan, Aroop <[hidden email]>; kumar, Senthil(AWF) <[hidden email]>
Subject: Re: NFS Gateway - Secure Cluster - Mount Failed

 

Anybody facing the  issue in Secure Cluster ?? ..

 

added root directory in /etc/exports

 cat /etc/exports

/ *(rw,fsid=0,no_root_squash)

 

mount -vvv -t nfs -o nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,sync host:/ /hdfs_space

mount: fstab path: "/etc/fstab"

mount: mtab path:  "/etc/mtab"

mount: lock path:  "/etc/mtab~"

mount: temp path:  "/etc/mtab.tmp"

mount: UID:        0

mount: eUID:       0

mount: spec:  "phxdpehdc30dn0007.stratus.phx.ebay.com:/"

mount: node:  "/hdfs_space"

mount: types: "nfs"

mount: opts:  "nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,sync"

final mount options: 'nfsvers=3,sec=krb5,proto=tcp,nolock,noacl'

mount: external mount: argv[0] = "/sbin/mount.nfs"

mount: external mount: argv[1] = "host:/"

mount: external mount: argv[2] = "/hdfs_space"

mount: external mount: argv[3] = "-v"

mount: external mount: argv[4] = "-o"

mount: external mount: argv[5] = "rw,sync,nfsvers=3,sec=krb5,proto=tcp,nolock,noacl"

mount.nfs: timeout set for Mon Aug 29 03:51:30 2016

mount.nfs: trying text-based options 'nfsvers=3,sec=krb5,proto=tcp,nolock,noacl,addr=10.115.22.46'

mount.nfs: prog 100003, trying vers=3, prot=6

mount.nfs: trying 10.115.22.46 prog 100003 vers 3 prot TCP port 2049

mount.nfs: prog 100005, trying vers=3, prot=6

mount.nfs: trying 10.115.22.46 prog 100005 vers 3 prot TCP port 4242

mount.nfs: mount(2): Permission denied

mount.nfs: access denied by server while mounting host:/

 

 

--Senthil

 

On Thu, Aug 25, 2016 at 4:58 PM, Senthil Kumar <[hidden email]> wrote:

Started NFS Service in DEBUG mode and found below logs ...

 

2016-08-25 03:59:05,766 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS NULL

2016-08-25 03:59:05,768 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT NULLOP :  client: /IP_ADDR

2016-08-25 03:59:05,770 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT NULLOP :  client: /IP_ADDR

2016-08-25 03:59:05,771 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT MNT path: / client: /IP_ADDR

2016-08-25 03:59:05,771 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: Got host: gateway path: /

2016-08-25 03:59:05,783 INFO org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: Giving handle (fileId:16385) to client for export /

 

==== { Looks like mount operation done } ======

2016-08-25 03:59:05,784 DEBUG org.apache.hadoop.hdfs.nfs.mount.RpcProgramMountd: MOUNT UMNT path: / client: /IP_ADDR

==== { Why client is Sending UMNT request } ==== 

 

Here is the MNT CMD:

 mount -vvv -t nfs -o vers=3,sec=krb5,proto=tcp,nolock,sync IP_ADDR:/ /hdfs_space

 

Can someone help me here to understand the behavior ?? and how to solve this mnt issue ??

 

--Senthil

 

On Thu, Aug 25, 2016 at 12:07 PM, Senthil Kumar <[hidden email]> wrote:

Expected Client Kerberos Principle is null issue resolved now .. Added sec=krb5 option while mounting ..

 

mount -vvv -t nfs -o vers=3,sec=krb5,proto=tcp,nolock,noacl,sync gateway:/ hdfs_space/

 

 

mount: fstab path: "/etc/fstab"

mount: mtab path:  "/etc/mtab"

mount: lock path:  "/etc/mtab~"

mount: temp path:  "/etc/mtab.tmp"

mount: UID:        0

mount: eUID:       0

mount: spec:  "gatewaymachine:/"

mount: node:  "hdfs_space/"

mount: types: "nfs"

mount: opts:  "vers=3,sec=krb5,proto=tcp,nolock,noacl,sync"

final mount options: 'vers=3,sec=krb5,proto=tcp,nolock,noacl'

mount: external mount: argv[0] = "/sbin/mount.nfs"

mount: external mount: argv[1] = "gatewaymachine:/"

mount: external mount: argv[2] = "hdfs_space/"

mount: external mount: argv[3] = "-v"

mount: external mount: argv[4] = "-o"

mount: external mount: argv[5] = "rw,sync,vers=3,sec=krb5,proto=tcp,nolock,noacl"

mount.nfs: timeout set for Wed Aug 24 23:34:31 2016

mount.nfs: trying text-based options 'vers=3,sec=krb5,proto=tcp,nolock,noacl,addr=10.115.22.109'

mount.nfs: prog 100003, trying vers=3, prot=6

mount.nfs: trying 10.115.22.109 prog 100003 vers 3 prot TCP port 2049

mount.nfs: prog 100005, trying vers=3, prot=6

mount.nfs: trying 10.115.22.109 prog 100005 vers 3 prot TCP port 4242

mount.nfs: mount(2): Permission denied

mount.nfs: access denied by server while mounting gatewaymachine:/

 

 

Not sure why mount throwing permission issue .. Anybody faced this issue ?? 

 

 

--Senthil

 

On Thu, Aug 25, 2016 at 10:53 AM, Senthil Kumar <[hidden email]> wrote:

Hi Team ,  As part of NFS Evaluation , i have installed NFS Gateway Service in Secure Cluster ..

 

 

Config in Gateway Machine:

<property>

        <name>nfs.file.dump.dir</name>

        <value>/tmp/.hdfs-nfs</value>

   </property>

   <property>

        <name>nfs.keytab.file</name>

        <value>/etc/hadoop/hadoop.keytab</value>

   </property>

   <property>

        <name>nfs.kerberos.principal</name>

        <value>hadoop/_[hidden email]</value>

   </property>

 

 

NFS3 Service Started Successfully , but when i try to Mount the root / directory it failed with below error ..

 

WARN org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: Exception

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): 

User root (auth:PROXY) via hadoop/[hidden email] (auth:KERBEROS)

 is not authorized for protocol interface org.apache.hadoop.hdfs.protocol.ClientProtocol, expected client Kerberos principal is null

 

 

mount command:

mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync gatewaymachine:/ hdfs_space/

mount.nfs: mount system call failed

 

 

What could be the issue here ??  I followed https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html this documentation ..

 

 

--Senthil