Commented: (HADOOP-492) Global counters

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Commented: (HADOOP-492) Global counters

Sebastian Nagel (Jira)

    [ ]

Doug Cutting commented on HADOOP-492:

Overall this looks good to me.  A couple of minor comments:

1. You've changed Reporter from an interface to an abstract class.  That's a significant enough change that I'd like to understand its motivation.  I'd like to see an analysis of the tradeoffs of that before we make such a change to a core public API.

2. You've commented out some code rather than deleted it.  We generally try to avoid that.

> Global counters
> ---------------
>                 Key: HADOOP-492
>                 URL:
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: arkady borkovsky
>         Assigned To: David Bowen
>         Attachments: counters1.patch
> It would be nice to have map / reduce job keep aggregated counts for arbitrary events occuring in its tasks -- the numer of records processed, the numer of exceptions of a specific type, the number of sentences in passive voice, whatever the jobs finds useful.
> This can be implemented by tasks periodically sending <name, value> pairs to the jobtracker (in some implementations such messages are piggy-backed on the heartbeats), so that the job tracker stores all the latests values from each task and aggregates them on a request.  It should also make the aggregated values available at the job end.  The value for a task would be flushed when the task fails.
> #491 and #490 may be related to this one.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.