Quantcast

Job History files in Hadoop 2.0

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Job History files in Hadoop 2.0

sangroya
Hi,

I recently migrated to Hadoop 2.0 from Hadoop 1.0 (0.20.2 before).

I am able to successfully launch example applications.

Could anyone please suggest where are the MapReduce job history files available, after running jobs in Hadoop 2.0.

I need the statistics after running the jobs. Of course, the web UI gives me the information. But I need the history files that were available in the previous versions of hadoop job_ID etc.

I can see a directory with application and container patterns but this does not have specific information about job submit time, Map Start time, finish time, reduce start time, finish time, job finish time etc.

In the previous version of hadoop i.e. 1.0 or 0.20, it was stored under logs/history.

Can anyone suggest if the pattern of storing job history files is also changed in the new architecture?


Thanks in advance,
Amit
Sangroya
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Job History files in Hadoop 2.0

Ravi Prakash-4
Hi Amit,

The job history files are stored in HDFS. eg. /mapred/history/done/2013/02/06/000000

I would think that there would be some changes in the format.

Thanks
Ravi




________________________________
 From: sangroya <[hidden email]>
To: [hidden email]
Sent: Tuesday, February 5, 2013 9:42 AM
Subject: Job History files in Hadoop 2.0
 
Hi,

I recently migrated to Hadoop 2.0 from Hadoop 1.0 (0.20.2 before).

I am able to successfully launch example applications.

/Could anyone please suggest where are the MapReduce job history files
available, after running jobs in Hadoop 2.0./

I need the statistics after running the jobs. Of course, the web UI gives me
the information. But I need the history files that were available in the
previous versions of hadoop job_ID etc.

I can see a directory with application and container patterns but this does
not have specific information about job submit time, Map Start time, finish
time, reduce start time, finish time, job finish time etc.

In the previous version of hadoop i.e. 1.0 or 0.20, it was stored under
logs/history.

Can anyone suggest if the pattern of storing job history files is also
changed in the new architecture?


Thanks in advance,
Amit



-----
Sangroya
--
View this message in context: http://lucene.472066.n3.nabble.com/Job-History-files-in-Hadoop-2-0-tp4038599.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Loading...