Fastest HDFS loader

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Fastest HDFS loader

Edmon Begoli
Hi,

We are going to be loading 4-5 GB text, delimited file from a RHEL file
system into HDFS to be managed
as external table by Hive.

What is the recommended, fastest loading mechanism?

Thank you,
Edmon
Reply | Threaded
Open this post in threaded view
|

Re: Fastest HDFS loader

Konstantin Boudnik-2
Do you have some strict performance requirement or something? Cause 5Gb is
pretty much nothing, really. I'd say copyFromLocal will do just fine.

Cos

On Tue, Dec 20, 2011 at 10:32PM, Edmon Begoli wrote:

> Hi,
>
> We are going to be loading 4-5 GB text, delimited file from a RHEL file
> system into HDFS to be managed
> as external table by Hive.
>
> What is the recommended, fastest loading mechanism?
>
> Thank you,
> Edmon

signature.asc (237 bytes) Download Attachment