[jira] Created: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
Progress report is not sent during the intermediate sorts in the map phase
--------------------------------------------------------------------------

                 Key: HADOOP-846
                 URL: http://issues.apache.org/jira/browse/HADOOP-846
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
            Reporter: Devaraj Das
         Assigned To: Devaraj Das


Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-846?page=all ]

Devaraj Das updated HADOOP-846:
-------------------------------

    Attachment: 846.patch

Attached is the patch - this starts a single thread that does progress reporting for the entire map phase. Also made taskProgress as volatile since that is read/updated by multiple threads.

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: http://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-846?page=all ]

Devaraj Das updated HADOOP-846:
-------------------------------

    Attachment: 846-correct.patch

Oops.. had attached the wrong patch earlier. This is the correct one.

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: http://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-846?page=all ]

Devaraj Das updated HADOOP-846:
-------------------------------

    Status: Patch Available  (was: Open)

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: http://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/HADOOP-846?page=comments#action_12460568 ]
           
Hadoop QA commented on HADOOP-846:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12347767/846-correct.patch applied and successfully tested against trunk revision r489707.

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: http://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ http://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461840 ]

Owen O'Malley commented on HADOOP-846:
--------------------------------------

The status thread is not cleaned up if the merge throws an exception. I think it should probably look like:

try {
  runner.run(...);
  ... merge code ...
} finally {
  in.close();
  collector.close();
  sortProgress.interrupt();
}

so that if run() throws, the merge code won't run. But if the merge code throws then the cleanup is still done.



> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: http://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-846:
-------------------------------

    Attachment: 846.new.patch

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.new.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461922 ]

Devaraj Das commented on HADOOP-846:
------------------------------------

Thanks for the review Owen. Attached is the modified patch.

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.new.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462012 ]

Hadoop QA commented on HADOOP-846:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12348190/846.new.patch applied and successfully tested against trunk revision r492232.

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.new.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462261 ]

Owen O'Malley commented on HADOOP-846:
--------------------------------------

+1

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 846-correct.patch, 846.new.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-846) Progress report is not sent during the intermediate sorts in the map phase

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-846:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.10.0
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Devaraj!

> Progress report is not sent during the intermediate sorts in the map phase
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-846
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>             Fix For: 0.10.0
>
>         Attachments: 846-correct.patch, 846.new.patch, 846.patch
>
>
> Have seen tasks getting lost at the TaskTracker's end due to MapTask's progress not getting reported for a long time (the configured timeout). The progress report is currently not sent in the intermediate sort phases in the MapTask. But, if for some reason, the sort takes a long time, the TaskTracker might kill the task.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira