[jira] Created: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
Metrics based on Map-Reduce Counters are not cleaned up
-------------------------------------------------------

                 Key: HADOOP-1406
                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
             Project: Hadoop
          Issue Type: Bug
            Reporter: David Bowen
         Assigned To: David Bowen


When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).

The fix is for JobInProgress to clean up the metrics it created when the job is complete.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497667 ]

David Bowen commented on HADOOP-1406:
-------------------------------------


As part of fixing this, I would like to propose two small API changes in org.apache.hadoop.metrics:

[1] Currently, to remove metrics records from the metrics library's internal table you have to iterate over them one by one, since removal requires the record name and all of its tags (name/value pairs).  I propose to generalize this so that you can specify the name and a subset of the tags - saving the client code the burden of having to remember all the tag/values that it has used.  This doesn't change any API signatures, but it changes the meaning of MetricsRecord.remove().

[2] Also, as a convenience, I propose to add a method MetricsRecord.removeTag(String tagName).  This method would be added to the MetricsRecordImpl class in the spi package also.  This enables the use pattern:

   On module initialization:
       create a metrics record named foo
       set long-lived tag (e.g. job id)

   Then repeatedly:
       set short-lived tags (e.g. counter name)
       set metric values
       call MetricsRecord.update

   And finally, on cleanup:
       remove the short-lived tags
       call MetricsRecord.remove to clean up all the metric data created

Both of these should be binary compatible changes for people using the public package o.a.h.metrics.



> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
In reply to this post by Nick Burch (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Bowen updated HADOOP-1406:
--------------------------------

    Attachment: 1406.patch


This patch contains the API changes described above, and code which uses the modified API to clean up the counter metric data after a job is completed.

Also, it adds an extra "jobId" tag to the counter metric data, so as to avoid problems that would arise in the case where a user has two jobs with the same name running at the same time (on the same jobtracker).





> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>         Attachments: 1406.patch
>
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
In reply to this post by Nick Burch (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Bowen updated HADOOP-1406:
--------------------------------

    Status: Patch Available  (was: Open)

> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>         Attachments: 1406.patch
>
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
In reply to this post by Nick Burch (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497681 ]

Hadoop QA commented on HADOOP-1406:
-----------------------------------

+1

http://issues.apache.org/jira/secure/attachment/12357825/1406.patch applied and successfully tested against trunk revision r540359.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/174/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/174/console

> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>         Attachments: 1406.patch
>
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
In reply to this post by Nick Burch (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-1406:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.14.0
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, David!

> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>             Fix For: 0.14.0
>
>         Attachments: 1406.patch
>
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1406) Metrics based on Map-Reduce Counters are not cleaned up

Nick Burch (Jira)
In reply to this post by Nick Burch (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498192 ]

Hadoop QA commented on HADOOP-1406:
-----------------------------------

Integrated in Hadoop-Nightly #98 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/98/)

> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
>                 Key: HADOOP-1406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1406
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: David Bowen
>         Assigned To: David Bowen
>             Fix For: 0.14.0
>
>         Attachments: 1406.patch
>
>
> When map-reduce jobs are finished, the metrics corresponding to their counters are not cleaned up.  This is a memory leak, but worse it means that an ever-increasing amount of metric data is sent to the metrics system (if one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job is complete.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.