Hadoop File system performance counters

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop File system performance counters

abhishek sharma
Hi,

What do the following two File Sytem counters associated with a job
(and printed at the end of a job's execution) represent?

FILE_BYTES_READ and FILE_BYTES_WRITTEN

How are they different from the HDFS_BYTES_READ and HDFS_BYTES_WRITTEN?

Thanks,
Abhishek
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop File system performance counters

James Seigel
They represent the amount data written to the physical disk on the slaves, as intermediate files before or during the shuffle phase.  Where HDFS bytes are the files written back into hdfs containing the data you wish to see.

J

On 2010-12-15, at 10:37 AM, abhishek sharma wrote:

> Hi,
>
> What do the following two File Sytem counters associated with a job
> (and printed at the end of a job's execution) represent?
>
> FILE_BYTES_READ and FILE_BYTES_WRITTEN
>
> How are they different from the HDFS_BYTES_READ and HDFS_BYTES_WRITTEN?
>
> Thanks,
> Abhishek