[jira] Created: (HADOOP-1514) Progress reporting not handled for the case of empty map outputs

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-1514) Progress reporting not handled for the case of empty map outputs

Hudson (Jira)
Progress reporting not handled for the case of empty map outputs
----------------------------------------------------------------

                 Key: HADOOP-1514
                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.14.0
            Reporter: Devaraj Das
            Assignee: Vivek Ratan
            Priority: Blocker
             Fix For: 0.14.0


For the randomwriter run, the maps actually don't generate any output, and the reducer is just waiting for the maps to complete. During that time, progress reporting is not done and hence the process gets killed. Also the case of empty map outputs needs to be handled more generally.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-1514:
--------------------------------

    Description: In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.  (was: For the randomwriter run, the maps actually don't generate any output, and the reducer is just waiting for the maps to complete. During that time, progress reporting is not done and hence the process gets killed. Also the case of empty map outputs needs to be handled more generally.)
        Summary: Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch  (was: Progress reporting not handled for the case of empty map outputs)

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vivek Ratan updated HADOOP-1514:
--------------------------------

    Attachment: 1514_01.patch

The fix is to report progress in the Reducer if there are no downloads of map output in progress or none scheduled. See 1514_01.patch.

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508067 ]

Devaraj Das commented on HADOOP-1514:
-------------------------------------

+1

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vivek Ratan updated HADOOP-1514:
--------------------------------

    Status: Patch Available  (was: Open)

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508078 ]

Hadoop QA commented on HADOOP-1514:
-----------------------------------

+1

http://issues.apache.org/jira/secure/attachment/12360538/1514_01.patch applied and successfully tested against trunk revision r550635.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/331/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/331/console

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-1514:
---------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Vivek!

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1514) Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch

Hudson (Jira)
In reply to this post by Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508510 ]

Hudson commented on HADOOP-1514:
--------------------------------

Integrated in Hadoop-Nightly #137 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/137/])

> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1514
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Devaraj Das
>            Assignee: Vivek Ratan
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.