Quantcast

Re: How to load raw log file into HDFS?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

Harsh J-2
Hi,

Please read http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html
to get started with using HDFS.

To answer your question specifically though, use the "copyFromLocal"
or the "put" commands with the "hadoop" (0.20/1.0+) or "hdfs"
(0.22/0.23/2.0+) executables, as detailed here:
http://hadoop.apache.org/common/docs/current/file_system_shell.html#copyFromLocal

hadoop fs -put <local log file path> <path it should go to on the HDFS>

On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]> wrote:
>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context: http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>



--
Harsh J
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

samir.helpdoc@gmail.com
Hi
To load any file from local
Command:
 syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
   Example hadoop fs -copyFromLocal input/logs
hdfs://localhost/user/dataset/

  More Commans: http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html


On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]> wrote:

>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context:
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: How to load raw log file into HDFS?

Amith D K
U can even use put/copyFromLocal

both are similar and does the job via terminal.

Or u can write a simple client program to do the job :)

Amith


________________________________________
From: samir das mohapatra [[hidden email]]
Sent: Sunday, May 13, 2012 9:13 PM
To: [hidden email]
Subject: Re: How to load raw log file into HDFS?

Hi
To load any file from local
Command:
 syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
   Example hadoop fs -copyFromLocal input/logs
hdfs://localhost/user/dataset/

  More Commans: http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html


On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]> wrote:

>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs.
> Please suggest the command and steps.
> Thanks in advance!!
> --
> View this message in context:
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

Alexander Fahlke
Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org


On Mon, May 14, 2012 at 7:24 AM, Amith D K <[hidden email]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[hidden email]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [hidden email]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

rdaley
In reply to this post by Harsh J-2

If you are a novice I'd suggest using a visual design tool like Pentaho
Kettle   http://wiki.pentaho.com/display/BAD/Loading+Data+into+HDFS  How To
Load Data into HDFS

AnExplorer wrote:
>
> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> hdfs. Please suggest the command and steps.
> Thanks in advance!!
>

--
View this message in context: http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33832683.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: How to load raw log file into HDFS?

Michael Wang
In reply to this post by Alexander Fahlke
I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
How do I know what <HDFS_FILE_PATH> are available?

-----Original Message-----
From: Alexander Fahlke [mailto:[hidden email]]
Sent: Monday, May 14, 2012 1:53 AM
To: [hidden email]
Subject: Re: How to load raw log file into HDFS?

Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org


On Mon, May 14, 2012 at 7:24 AM, Amith D K <[hidden email]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[hidden email]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [hidden email]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>

This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

Manish Bhoge
You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS. If you want to run(test) PIG in local then in that case you don't to copy data to HDFS
Sent from my BlackBerry, pls excuse typo

-----Original Message-----
From: Michael Wang <[hidden email]>
Date: Mon, 14 May 2012 18:43:47
To: [hidden email]<[hidden email]>
Reply-To: [hidden email]
Subject: RE: How to load raw log file into HDFS?

I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
How do I know what <HDFS_FILE_PATH> are available?

-----Original Message-----
From: Alexander Fahlke [mailto:[hidden email]]
Sent: Monday, May 14, 2012 1:53 AM
To: [hidden email]
Subject: Re: How to load raw log file into HDFS?

Hi,

the best would be to read the documentation and some books to get familar
with Hadoop.

One of my favourite books is "Hadoop in Action" from Manning (
http://www.manning.com/lam/)
This book has an exmple for putting (log)-files into HDFS. Check out the
source "listing-3-1"

Later you can also check out Cloudera's Flume:
https://github.com/cloudera/flume/wiki

--
BR

Alexander Fahlke
Java Developer
www.nurago.com | www.fahlke.org


On Mon, May 14, 2012 at 7:24 AM, Amith D K <[hidden email]> wrote:

> U can even use put/copyFromLocal
>
> both are similar and does the job via terminal.
>
> Or u can write a simple client program to do the job :)
>
> Amith
>
>
> ________________________________________
> From: samir das mohapatra [[hidden email]]
> Sent: Sunday, May 13, 2012 9:13 PM
> To: [hidden email]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi
> To load any file from local
> Command:
>  syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>   Example hadoop fs -copyFromLocal input/logs
> hdfs://localhost/user/dataset/
>
>  More Commans:
> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>
>
> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]>
> wrote:
>
> >
> > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
> > hdfs.
> > Please suggest the command and steps.
> > Thanks in advance!!
> > --
> > View this message in context:
> >
> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>

This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to load raw log file into HDFS?

ranjith raghunath
You can load data directly into a hive table(external and internal) directly from the local file system. The same stands for pig. To Manish's point you can do the same using hadoop fs commands. I have tried it both ways and have seen a difference in performance. I would be interested to hear from the rest of the community about this to see it is consistent with what they have seen.

Thanks,
Ranjith

On May 14, 2012, at 8:45 PM, "Manish Bhoge" <[hidden email]> wrote:

> You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS. If you want to run(test) PIG in local then in that case you don't to copy data to HDFS
> Sent from my BlackBerry, pls excuse typo
>
> -----Original Message-----
> From: Michael Wang <[hidden email]>
> Date: Mon, 14 May 2012 18:43:47
> To: [hidden email]<[hidden email]>
> Reply-To: [hidden email]
> Subject: RE: How to load raw log file into HDFS?
>
> I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :)
> I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>.
> Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS?
> How do I know what <HDFS_FILE_PATH> are available?
>
> -----Original Message-----
> From: Alexander Fahlke [mailto:[hidden email]]
> Sent: Monday, May 14, 2012 1:53 AM
> To: [hidden email]
> Subject: Re: How to load raw log file into HDFS?
>
> Hi,
>
> the best would be to read the documentation and some books to get familar
> with Hadoop.
>
> One of my favourite books is "Hadoop in Action" from Manning (
> http://www.manning.com/lam/)
> This book has an exmple for putting (log)-files into HDFS. Check out the
> source "listing-3-1"
>
> Later you can also check out Cloudera's Flume:
> https://github.com/cloudera/flume/wiki
>
> --
> BR
>
> Alexander Fahlke
> Java Developer
> www.nurago.com | www.fahlke.org
>
>
> On Mon, May 14, 2012 at 7:24 AM, Amith D K <[hidden email]> wrote:
>
>> U can even use put/copyFromLocal
>>
>> both are similar and does the job via terminal.
>>
>> Or u can write a simple client program to do the job :)
>>
>> Amith
>>
>>
>> ________________________________________
>> From: samir das mohapatra [[hidden email]]
>> Sent: Sunday, May 13, 2012 9:13 PM
>> To: [hidden email]
>> Subject: Re: How to load raw log file into HDFS?
>>
>> Hi
>> To load any file from local
>> Command:
>> syntax:     hadoop fs -copyFromLocal  <LOCAL_FILE_PATH>  <HDFS_FILE_PATH>
>>  Example hadoop fs -copyFromLocal input/logs
>> hdfs://localhost/user/dataset/
>>
>> More Commans:
>> http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html
>>
>>
>> On Sun, May 13, 2012 at 9:53 AM, AnExplorer <[hidden email]>
>> wrote:
>>
>>>
>>> Hi, I am novice in Hadoop. Kindly suggest how do we load log files into
>>> hdfs.
>>> Please suggest the command and steps.
>>> Thanks in advance!!
>>> --
>>> View this message in context:
>>>
>> http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>>
>>>
>>
>
> This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.
>
Loading...