NFSv3 Filesystem Connector

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

NFSv3 Filesystem Connector

Gokul Soundararajan
Hi,

We (Jingxin Feng, Xing Lin, and I) have been working on providing a
FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
server as a filesystem. It leverages code from hadoop-nfs project for all
the request/response handling. We would like your help to add it as part of
hadoop tools (similar to the way hadoop-aws and hadoop-azure).

In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
The NFS Connector can be run in two modes: (1) secondary filesystem - where
Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
runs entirely on a NFSv3 storage server.

The code is written in a way such that existing applications do not have to
change. All one has to do is to copy the connector jar into the lib/
directory of Hadoop/Spark. Then, modify core-site.xml to provide the
necessary details.

The current version can be seen at:
https://github.com/NetApp/NetApp-Hadoop-NFS-Connector

It is my first time contributing to the Hadoop codebase. It would be great
if someone on the Hadoop team can guide us through this process. I'm
willing to make the necessary changes to integrate the code. What are the
next steps? Should I create a JIRA entry?

Thanks,

Gokul
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Niels Basjes
Hi,

The main reason Hadoop scales so well is because all components try to
adhere to the idea around having Data Locality.
In general this means that you are running the processing/query software on
the system where the actual data is already present on the local disk.

To me this NFS solution sounds like hooking the processing nodes to a
shared storage solution.
This may work for small clusters (say 5 nodes or so) but for large clusters
this shared storage will be the main bottle neck in the processing/query
speed.

We currently have more than 20 nodes with 12 harddisks each resulting in
over 50GB/sec [1] of disk-to-queryengine speed and this means that our
setup already goes much faster than any network connection to any NFS
solution can provide. We can simply go to say 50 nodes and exceed the
100GB/sec speed easy.

So to me this sounds like hooking a scalable processing platform to a non
scalable storage system (mainly because the network to this storage doesn't
scale).

So far I have only seen vendors of legacy storage solutions going in this
direction ... oh wait ... you are NetApp ... that explains it.

I am no committer in any of the Hadoop tools but I vote against having such
a "core concept breaking" piece in the main codebase. New people may start
to think it is a good idea to do this.

So I say you should simply make this plugin available to your customers,
just not as a core part of Hadoop.

Niels Basjes

[1]  50 GB/sec = approx 20*12*200MB/sec
      This page shows max read speed in the 200MB/sec range:

http://www.tomshardware.com/charts/enterprise-hdd-charts/-02-Read-Throughput-Maximum-h2benchw-3.16,3372.html


On Tue, Jan 13, 2015 at 10:35 PM, Gokul Soundararajan <
[hidden email]> wrote:

> Hi,
>
> We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> server as a filesystem. It leverages code from hadoop-nfs project for all
> the request/response handling. We would like your help to add it as part of
> hadoop tools (similar to the way hadoop-aws and hadoop-azure).
>
> In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> The NFS Connector can be run in two modes: (1) secondary filesystem - where
> Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> runs entirely on a NFSv3 storage server.
>
> The code is written in a way such that existing applications do not have to
> change. All one has to do is to copy the connector jar into the lib/
> directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> necessary details.
>
> The current version can be seen at:
> https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
>
> It is my first time contributing to the Hadoop codebase. It would be great
> if someone on the Hadoop team can guide us through this process. I'm
> willing to make the necessary changes to integrate the code. What are the
> next steps? Should I create a JIRA entry?
>
> Thanks,
>
> Gokul
>



--
Best regards / Met vriendelijke groeten,

Niels Basjes
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Gokul Soundararajan
Hi Niels,

Thanks for your comments. My goal in designing the NFS connector is *not*
to replace HDFS. HDFS is ideally suited for Hadoop (otherwise why was it
built?).
The problem is that we have people who have PBs (10PB to 50PB) of data on
NFS storage that they would like process using Hadoop. Such amount of data
is both time-consuming and costly to move around; some have used Sqoop and
Flume but it is still painful. To help these folks, we built the connector
to enable Hadoop analytics on this data. As NFS is an open standard, we
believe that it would benefit everyone who have this use case.

Regarding the performance point, I hope you don't think a NFS storage
server is a box with several disks and a single network connection. The
latest generation storage servers are clustered storage systems that can
have 17000+ drives , holding 100PBs, and support 64 10GbE ports on each
cluster node. The NetApp spec sheet is here:
http://www.netapp.com/us/products/storage-systems/fas8000/fas8000-tech-specs.aspx

I hope this clarifies why we want to make this contribution. It is to
unlock additional data that can be processed by Hadoop.

Thanks,

Gokul


On Wed, Jan 14, 2015 at 3:14 AM, Niels Basjes <[hidden email]> wrote:

> Hi,
>
> The main reason Hadoop scales so well is because all components try to
> adhere to the idea around having Data Locality.
> In general this means that you are running the processing/query software on
> the system where the actual data is already present on the local disk.
>
> To me this NFS solution sounds like hooking the processing nodes to a
> shared storage solution.
> This may work for small clusters (say 5 nodes or so) but for large clusters
> this shared storage will be the main bottle neck in the processing/query
> speed.
>
> We currently have more than 20 nodes with 12 harddisks each resulting in
> over 50GB/sec [1] of disk-to-queryengine speed and this means that our
> setup already goes much faster than any network connection to any NFS
> solution can provide. We can simply go to say 50 nodes and exceed the
> 100GB/sec speed easy.
>
> So to me this sounds like hooking a scalable processing platform to a non
> scalable storage system (mainly because the network to this storage doesn't
> scale).
>
> So far I have only seen vendors of legacy storage solutions going in this
> direction ... oh wait ... you are NetApp ... that explains it.
>
> I am no committer in any of the Hadoop tools but I vote against having such
> a "core concept breaking" piece in the main codebase. New people may start
> to think it is a good idea to do this.
>
> So I say you should simply make this plugin available to your customers,
> just not as a core part of Hadoop.
>
> Niels Basjes
>
> [1]  50 GB/sec = approx 20*12*200MB/sec
>       This page shows max read speed in the 200MB/sec range:
>
>
> http://www.tomshardware.com/charts/enterprise-hdd-charts/-02-Read-Throughput-Maximum-h2benchw-3.16,3372.html
>
>
> On Tue, Jan 13, 2015 at 10:35 PM, Gokul Soundararajan <
> [hidden email]> wrote:
>
> > Hi,
> >
> > We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> > FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> > server as a filesystem. It leverages code from hadoop-nfs project for all
> > the request/response handling. We would like your help to add it as part
> of
> > hadoop tools (similar to the way hadoop-aws and hadoop-azure).
> >
> > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> > Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> > The NFS Connector can be run in two modes: (1) secondary filesystem -
> where
> > Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> > second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> > runs entirely on a NFSv3 storage server.
> >
> > The code is written in a way such that existing applications do not have
> to
> > change. All one has to do is to copy the connector jar into the lib/
> > directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> > necessary details.
> >
> > The current version can be seen at:
> > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
> >
> > It is my first time contributing to the Hadoop codebase. It would be
> great
> > if someone on the Hadoop team can guide us through this process. I'm
> > willing to make the necessary changes to integrate the code. What are the
> > next steps? Should I create a JIRA entry?
> >
> > Thanks,
> >
> > Gokul
> >
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Colin McCabe
In reply to this post by Gokul Soundararajan
Why not just use LocalFileSystem with an NFS mount (or several)?  I read
through the README but I didn't see that question answered anywhere.

best,
Colin

On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan <[hidden email]
> wrote:

> Hi,
>
> We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> server as a filesystem. It leverages code from hadoop-nfs project for all
> the request/response handling. We would like your help to add it as part of
> hadoop tools (similar to the way hadoop-aws and hadoop-azure).
>
> In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> The NFS Connector can be run in two modes: (1) secondary filesystem - where
> Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> runs entirely on a NFSv3 storage server.
>
> The code is written in a way such that existing applications do not have to
> change. All one has to do is to copy the connector jar into the lib/
> directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> necessary details.
>
> The current version can be seen at:
> https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
>
> It is my first time contributing to the Hadoop codebase. It would be great
> if someone on the Hadoop team can guide us through this process. I'm
> willing to make the necessary changes to integrate the code. What are the
> next steps? Should I create a JIRA entry?
>
> Thanks,
>
> Gokul
>
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Colin McCabe-3
In reply to this post by Niels Basjes
Hi Niels,

I agree that direct-attached storage seems more economical for many users.
As an HDFS developer, I certainly have a dog in this fight as well :)

But we should be respectful towards people trying to contribute code to
Hadoop and evaluate the code on its own merits.  It is up to our users to
make the choice between direct-attached and remote storage paradigms.  And
networking technology continues to evolve so the right choice for 2014 may
not be the right choice for later.

Hadoop already includes code for connecting with s3, Cloudstack, and Azure,
none of which are direct-attached storage systems.  If this contribution
makes sense then we should accept it.  Like I said earlier, the question in
my mind is whether this functionality is possible to achieve through
LocalFileSsytem.

cheers,
Colin

On Wed, Jan 14, 2015 at 3:14 AM, Niels Basjes <[hidden email]> wrote:

> Hi,
>
> The main reason Hadoop scales so well is because all components try to
> adhere to the idea around having Data Locality.
> In general this means that you are running the processing/query software on
> the system where the actual data is already present on the local disk.
>
> To me this NFS solution sounds like hooking the processing nodes to a
> shared storage solution.
> This may work for small clusters (say 5 nodes or so) but for large clusters
> this shared storage will be the main bottle neck in the processing/query
> speed.
>
> We currently have more than 20 nodes with 12 harddisks each resulting in
> over 50GB/sec [1] of disk-to-queryengine speed and this means that our
> setup already goes much faster than any network connection to any NFS
> solution can provide. We can simply go to say 50 nodes and exceed the
> 100GB/sec speed easy.
>
> So to me this sounds like hooking a scalable processing platform to a non
> scalable storage system (mainly because the network to this storage doesn't
> scale).
>
> So far I have only seen vendors of legacy storage solutions going in this
> direction ... oh wait ... you are NetApp ... that explains it.
>
> I am no committer in any of the Hadoop tools but I vote against having such
> a "core concept breaking" piece in the main codebase. New people may start
> to think it is a good idea to do this.
>
> So I say you should simply make this plugin available to your customers,
> just not as a core part of Hadoop.
>
> Niels Basjes
>
> [1]  50 GB/sec = approx 20*12*200MB/sec
>       This page shows max read speed in the 200MB/sec range:
>
>
> http://www.tomshardware.com/charts/enterprise-hdd-charts/-02-Read-Throughput-Maximum-h2benchw-3.16,3372.html
>
>
> On Tue, Jan 13, 2015 at 10:35 PM, Gokul Soundararajan <
> [hidden email]> wrote:
>
> > Hi,
> >
> > We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> > FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> > server as a filesystem. It leverages code from hadoop-nfs project for all
> > the request/response handling. We would like your help to add it as part
> of
> > hadoop tools (similar to the way hadoop-aws and hadoop-azure).
> >
> > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> > Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> > The NFS Connector can be run in two modes: (1) secondary filesystem -
> where
> > Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> > second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> > runs entirely on a NFSv3 storage server.
> >
> > The code is written in a way such that existing applications do not have
> to
> > change. All one has to do is to copy the connector jar into the lib/
> > directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> > necessary details.
> >
> > The current version can be seen at:
> > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
> >
> > It is my first time contributing to the Hadoop codebase. It would be
> great
> > if someone on the Hadoop team can guide us through this process. I'm
> > willing to make the necessary changes to integrate the code. What are the
> > next steps? Should I create a JIRA entry?
> >
> > Thanks,
> >
> > Gokul
> >
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Gokul Soundararajan
In reply to this post by Colin McCabe
Hi Colin,

Yeah, I should add the reasons to the README. We tried LocalFileSystem when
we started out but we think we can do tighter Hadoop integration if we
write a connector.

Some examples include:
1. Limit over-prefetching of data - MapReduce splits the jobs into 128MB
splits and standard NFS driver tends to over-prefetch from a file. We limit
the prefetching to the split size.
2. Lazy write commits - For writes, we can relax the guarantees for writes
(and making it faster) and commit just before when the task ends.
3. Provide for location awareness - Later, we can hook some NFS smarts into
getFileBlockLocations() (Have some ideas but not implemented them yet).

Hope this helps.

Gokul



On Wed, Jan 14, 2015 at 10:47 AM, Colin McCabe <[hidden email]>
wrote:

> Why not just use LocalFileSystem with an NFS mount (or several)?  I read
> through the README but I didn't see that question answered anywhere.
>
> best,
> Colin
>
> On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan <
> [hidden email]
> > wrote:
>
> > Hi,
> >
> > We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> > FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> > server as a filesystem. It leverages code from hadoop-nfs project for all
> > the request/response handling. We would like your help to add it as part
> of
> > hadoop tools (similar to the way hadoop-aws and hadoop-azure).
> >
> > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) and
> > Apache Spark (1.2+) to use a NFSv3 storage server as a storage endpoint.
> > The NFS Connector can be run in two modes: (1) secondary filesystem -
> where
> > Hadoop/Spark runs using HDFS as its primary storage and can use NFS as a
> > second storage endpoint, and (2) primary filesystem - where Hadoop/Spark
> > runs entirely on a NFSv3 storage server.
> >
> > The code is written in a way such that existing applications do not have
> to
> > change. All one has to do is to copy the connector jar into the lib/
> > directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> > necessary details.
> >
> > The current version can be seen at:
> > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
> >
> > It is my first time contributing to the Hadoop codebase. It would be
> great
> > if someone on the Hadoop team can guide us through this process. I'm
> > willing to make the necessary changes to integrate the code. What are the
> > next steps? Should I create a JIRA entry?
> >
> > Thanks,
> >
> > Gokul
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Steve Loughran-3
Gokul,

What we expect from a filesystem is defined in (a) the HDFS code , (b) the
filesystem spec as derived from (a), and (c) contract tests derived from
(a) and (b)

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html

There's a wiki page to go with this : https://wiki.apache.org/hadoop/HCFS

Current policy on 3rd party FS support is pretty much:


   1. Nothing in the hadoop codebase that isn't easy for anyone to test
   2. Nothing in the hadoop codebase that isn't straightforward for anyone
   to maintain
   3. No FS support bundled in the Hadoop distribution that doesn't match
   (1) or (2). Note that you are free to use Bigtop to build your own stack
   with your own FS, which is what Redhat do with GFS, intel did with
   LustreFS, etc.

for a definition of "anyone" as "anyone who knows roughly what they are
doing and is familiar with the hadoop codebase"

If your proposal had been for some 3rd party FS, the answer would be "keep
it in your own source tree".

Except this isn't quite an FS, is it? It's an NFS client, which should be
able to talk to any NFSv3 server, including the standard Linux and OSX
ones, as well as the NFS support that you get with Hadoop.

Is that right? That with this code I could run tests on my Linux box which
would verify the client works with NFS?





On 15 January 2015 at 02:15, Gokul Soundararajan <[hidden email]>
wrote:

> Hi Colin,
>
> Yeah, I should add the reasons to the README. We tried LocalFileSystem when
> we started out but we think we can do tighter Hadoop integration if we
> write a connector.
>
Some examples include:
> 1. Limit over-prefetching of data - MapReduce splits the jobs into 128MB
> splits and standard NFS driver tends to over-prefetch from a file. We limit
> the prefetching to the split size.
>

Its more the MR asks the FS for the block size, and partitions are based on
that value. You'd want to return the prefetch-optimised value


> 2. Lazy write commits - For writes, we can relax the guarantees for writes
> (and making it faster) and commit just before when the task ends.
>

We haven't actually specified (yet) what output streams do. A key
requirement is that flush() persists, at least as far as the Java code is
aware of..there's the risk that the underlying OS can be lazy, and in a VM,
the virtual hardware can be even lazier. Note that the s3n, s3a and swift
break this rule utterly -this isn't considered satisfactory.


> 3. Provide for location awareness - Later, we can hook some NFS smarts into
> getFileBlockLocations() (Have some ideas but not implemented them yet).
>
>
That could be good. SwiftFS does this with some extensions to OpenStack
swift that went in in sync with our code.

Like I said, a new FS client for an external FS wouldn't get in to the
hadoop codebase. We care about HDFS, somewhat about file://, though
primarily for testing, and for integration with cloud storage.

An NFS client though, one that can be tested and is general purpose to work
with implementations of NFSv3, and ideally integrates with the Hadoop
kerberos auth mechanism, that could be nice.



> Hope this helps.
>
> Gokul
>
>
>
> On Wed, Jan 14, 2015 at 10:47 AM, Colin McCabe <[hidden email]>
> wrote:
>
> > Why not just use LocalFileSystem with an NFS mount (or several)?  I read
> > through the README but I didn't see that question answered anywhere.
> >
> > best,
> > Colin
> >
> > On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan <
> > [hidden email]
> > > wrote:
> >
> > > Hi,
> > >
> > > We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> > > FileSystem implementation that allows Hadoop to utilize a NFSv3 storage
> > > server as a filesystem. It leverages code from hadoop-nfs project for
> all
> > > the request/response handling. We would like your help to add it as
> part
> > of
> > > hadoop tools (similar to the way hadoop-aws and hadoop-azure).
> > >
> > > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+)
> and
> > > Apache Spark (1.2+) to use a NFSv3 storage server as a storage
> endpoint.
> > > The NFS Connector can be run in two modes: (1) secondary filesystem -
> > where
> > > Hadoop/Spark runs using HDFS as its primary storage and can use NFS as
> a
> > > second storage endpoint, and (2) primary filesystem - where
> Hadoop/Spark
> > > runs entirely on a NFSv3 storage server.
> > >
> > > The code is written in a way such that existing applications do not
> have
> > to
> > > change. All one has to do is to copy the connector jar into the lib/
> > > directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> > > necessary details.
> > >
> > > The current version can be seen at:
> > > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
> > >
> > > It is my first time contributing to the Hadoop codebase. It would be
> > great
> > > if someone on the Hadoop team can guide us through this process. I'm
> > > willing to make the necessary changes to integrate the code. What are
> the
> > > next steps? Should I create a JIRA entry?
> > >
> > > Thanks,
> > >
> > > Gokul
> > >
> >
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Reply | Threaded
Open this post in threaded view
|

Re: NFSv3 Filesystem Connector

Gokul Soundararajan
Hi Steve,

Thanks for the pointers. I looked at the FileSystemContractBaseTest and
MainOperationsBaseTest and wrote the setUp() functions to setup NFS
underneath. The extension of these test classes aren't very clean and I
hope I can use and the rest of team's help in writing them correctly.
In my unit tests, I setup a Mock NFS server (a really simple in-memory one)
but it would be better if we can launch the HDFS+NFS one instead.

The idea behind lazy flush is that NFS has commands for WRITE and COMMIT.
The COMMIT will guarantee that any lazily written writes are actually
written to stable (disk) storage. The optimization trick we do is to do the
COMMIT when the stream is flushed or when it is closed.
So the data will be stable after the stream is closed.

The NFS implementation follows the NFSv3 standard. In fact, I have run it
with Linux, Mac (how I write and test my code), and NetApp storage.

I need to look into the Hadoop Kerberos stuff and it would be great if we
can add that. (This is a work in progress and may be added in a later
revision).

Thanks,

Gokul

On Thu, Jan 15, 2015 at 3:39 AM, Steve Loughran <[hidden email]>
wrote:

> Gokul,
>
> What we expect from a filesystem is defined in (a) the HDFS code , (b) the
> filesystem spec as derived from (a), and (c) contract tests derived from
> (a) and (b)
>
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html
>
> There's a wiki page to go with this : https://wiki.apache.org/hadoop/HCFS
>
> Current policy on 3rd party FS support is pretty much:
>
>
>    1. Nothing in the hadoop codebase that isn't easy for anyone to test
>    2. Nothing in the hadoop codebase that isn't straightforward for anyone
>    to maintain
>    3. No FS support bundled in the Hadoop distribution that doesn't match
>    (1) or (2). Note that you are free to use Bigtop to build your own stack
>    with your own FS, which is what Redhat do with GFS, intel did with
>    LustreFS, etc.
>
> for a definition of "anyone" as "anyone who knows roughly what they are
> doing and is familiar with the hadoop codebase"
>
> If your proposal had been for some 3rd party FS, the answer would be "keep
> it in your own source tree".
>
> Except this isn't quite an FS, is it? It's an NFS client, which should be
> able to talk to any NFSv3 server, including the standard Linux and OSX
> ones, as well as the NFS support that you get with Hadoop.
>
> Is that right? That with this code I could run tests on my Linux box which
> would verify the client works with NFS?
>
>
>
>
>
> On 15 January 2015 at 02:15, Gokul Soundararajan <[hidden email]>
> wrote:
>
> > Hi Colin,
> >
> > Yeah, I should add the reasons to the README. We tried LocalFileSystem
> when
> > we started out but we think we can do tighter Hadoop integration if we
> > write a connector.
> >
> Some examples include:
> > 1. Limit over-prefetching of data - MapReduce splits the jobs into 128MB
> > splits and standard NFS driver tends to over-prefetch from a file. We
> limit
> > the prefetching to the split size.
> >
>
> Its more the MR asks the FS for the block size, and partitions are based on
> that value. You'd want to return the prefetch-optimised value
>
>
> > 2. Lazy write commits - For writes, we can relax the guarantees for
> writes
> > (and making it faster) and commit just before when the task ends.
> >
>
> We haven't actually specified (yet) what output streams do. A key
> requirement is that flush() persists, at least as far as the Java code is
> aware of..there's the risk that the underlying OS can be lazy, and in a VM,
> the virtual hardware can be even lazier. Note that the s3n, s3a and swift
> break this rule utterly -this isn't considered satisfactory.
>
>
> > 3. Provide for location awareness - Later, we can hook some NFS smarts
> into
> > getFileBlockLocations() (Have some ideas but not implemented them yet).
> >
> >
> That could be good. SwiftFS does this with some extensions to OpenStack
> swift that went in in sync with our code.
>
> Like I said, a new FS client for an external FS wouldn't get in to the
> hadoop codebase. We care about HDFS, somewhat about file://, though
> primarily for testing, and for integration with cloud storage.
>
> An NFS client though, one that can be tested and is general purpose to work
> with implementations of NFSv3, and ideally integrates with the Hadoop
> kerberos auth mechanism, that could be nice.
>
>
>
> > Hope this helps.
> >
> > Gokul
> >
> >
> >
> > On Wed, Jan 14, 2015 at 10:47 AM, Colin McCabe <[hidden email]>
> > wrote:
> >
> > > Why not just use LocalFileSystem with an NFS mount (or several)?  I
> read
> > > through the README but I didn't see that question answered anywhere.
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan <
> > > [hidden email]
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We (Jingxin Feng, Xing Lin, and I) have been working on providing a
> > > > FileSystem implementation that allows Hadoop to utilize a NFSv3
> storage
> > > > server as a filesystem. It leverages code from hadoop-nfs project for
> > all
> > > > the request/response handling. We would like your help to add it as
> > part
> > > of
> > > > hadoop tools (similar to the way hadoop-aws and hadoop-azure).
> > > >
> > > > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+)
> > and
> > > > Apache Spark (1.2+) to use a NFSv3 storage server as a storage
> > endpoint.
> > > > The NFS Connector can be run in two modes: (1) secondary filesystem -
> > > where
> > > > Hadoop/Spark runs using HDFS as its primary storage and can use NFS
> as
> > a
> > > > second storage endpoint, and (2) primary filesystem - where
> > Hadoop/Spark
> > > > runs entirely on a NFSv3 storage server.
> > > >
> > > > The code is written in a way such that existing applications do not
> > have
> > > to
> > > > change. All one has to do is to copy the connector jar into the lib/
> > > > directory of Hadoop/Spark. Then, modify core-site.xml to provide the
> > > > necessary details.
> > > >
> > > > The current version can be seen at:
> > > > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
> > > >
> > > > It is my first time contributing to the Hadoop codebase. It would be
> > > great
> > > > if someone on the Hadoop team can guide us through this process. I'm
> > > > willing to make the necessary changes to integrate the code. What are
> > the
> > > > next steps? Should I create a JIRA entry?
> > > >
> > > > Thanks,
> > > >
> > > > Gokul
> > > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>