[jira] Created: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
the task tracker is getting blocked by long deletes of local files
------------------------------------------------------------------

                 Key: HADOOP-840
                 URL: http://issues.apache.org/jira/browse/HADOOP-840
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.9.2
            Reporter: Owen O'Malley
         Assigned To: Owen O'Malley
             Fix For: 0.10.0


The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Owen O'Malley updated HADOOP-840:
---------------------------------

    Attachment: lost-tt.patch

This patch needs review and testing...

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/HADOOP-840?page=comments#action_12460022 ]
           
Owen O'Malley commented on HADOOP-840:
--------------------------------------

My patch depends on HADOOP-836

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/HADOOP-840?page=comments#action_12460159 ]
           
Nigel Daley commented on HADOOP-840:
------------------------------------

I believe that because the patch for 836 was modified before committing it, this patch now needs to be updated.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Attachment: lost-tt_1.patch

updated the patch to the current trunk. Fixed a nullpointer exception in the patch and also changed synchronization so that initialization of the new job does not get stuck on cleanup of the previous one.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_1.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Attachment:     (was: lost-tt_1.patch)

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Attachment: lost-tt_2.patch

problem with the prvious patch. Updating a new one.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_2.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Status: Patch Available  (was: Open)

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_2.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Status: Open  (was: Patch Available)

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_2.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/HADOOP-840?page=comments#action_12460600 ]
           
Hadoop QA commented on HADOOP-840:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12347774/lost-tt_2.patch applied and successfully tested against trunk revision r489707.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_2.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Attachment:     (was: lost-tt_2.patch)

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-840?page=all ]

Mahadev konar updated HADOOP-840:
---------------------------------

    Attachment: lost-tt_3.patch

an updated patch that passes all the unit tests.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: http://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_3.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462260 ]

Owen O'Malley commented on HADOOP-840:
--------------------------------------

+1

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_3.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved HADOOP-840.
---------------------------------

    Resolution: Fixed
      Assignee: Devaraj Das  (was: Owen O'Malley)

I just committed this.  Thanks, Devaraj!

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Devaraj Das
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_3.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Reopened: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting reopened HADOOP-840:
---------------------------------

      Assignee: Mahadev konar  (was: Devaraj Das)

Oops!  Resolved the wrong bug.

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Mahadev konar
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_3.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (HADOOP-840) the task tracker is getting blocked by long deletes of local files

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved HADOOP-840.
---------------------------------

    Resolution: Fixed

I just committed this.  Thanks, Owen & Mahadev!

> the task tracker is getting blocked by long deletes of local files
> ------------------------------------------------------------------
>
>                 Key: HADOOP-840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-840
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.9.2
>            Reporter: Owen O'Malley
>         Assigned To: Mahadev konar
>             Fix For: 0.10.0
>
>         Attachments: lost-tt.patch, lost-tt_3.patch
>
>
> The task tracker is getting blocked in the main heartbeat loop by doing task cleanups in the main thread. We need to queue up the cleanup actions and work on them offline. This should make our task trackers not miss heartbeats and thus get lost.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira