[jira] Created: (HADOOP-1524) Task Logs userlogs don't show up for a while

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1524) Task Logs userlogs don't show up for a while

Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512576 ]

Michael Bieniosek commented on HADOOP-1524:
-------------------------------------------

My patch shouldn't affect the fs code at all.  I have no idea what is failing or why.

> Task Logs userlogs don't show up for a while
> ---------------------------------------------
>
>                 Key: HADOOP-1524
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1524
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Michael Bieniosek
>         Attachments: accelerate-task-log.patch, eliminate-split-idx.patch
>
>
> When I start a task and go to the task logs, nothing shows up for a while.  An examination of TaskLog.Writer and TaskLog.Reader reveals:
> 1. The TaskLog.Reader relies on the presence of a split.idx to identify the parts of the logs to display.
> 2. The TaskLog.Writer only updates the split.idx file when it moves on to the next log.
> As a result, updates to the log only get pushed when an entire file is done.
> Why is there a split.idx file?  It seems that since files are called part-00000, part-00001, etc., the TaskLog.Reader can just look at all files and arrange them by alphabetical order.  The split.idx file also contains file length, but this data is already stored by the filesystem.
> If nobody has objections, I'd like to write a patch to eliminate the split.idx file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (HADOOP-1524) Task Logs userlogs don't show up for a while

Hudson (Jira)
In reply to this post by Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512576 ]

Michael Bieniosek edited comment on HADOOP-1524 at 7/13/07 11:47 AM:
---------------------------------------------------------------------

My patch shouldn't affect the fs code at all.  I have no idea what is failing or why.  I have personally tried my patch on a two-node cluster without issue.


 was:
My patch shouldn't affect the fs code at all.  I have no idea what is failing or why.

> Task Logs userlogs don't show up for a while
> ---------------------------------------------
>
>                 Key: HADOOP-1524
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1524
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Michael Bieniosek
>         Attachments: accelerate-task-log.patch, eliminate-split-idx.patch
>
>
> When I start a task and go to the task logs, nothing shows up for a while.  An examination of TaskLog.Writer and TaskLog.Reader reveals:
> 1. The TaskLog.Reader relies on the presence of a split.idx to identify the parts of the logs to display.
> 2. The TaskLog.Writer only updates the split.idx file when it moves on to the next log.
> As a result, updates to the log only get pushed when an entire file is done.
> Why is there a split.idx file?  It seems that since files are called part-00000, part-00001, etc., the TaskLog.Reader can just look at all files and arrange them by alphabetical order.  The split.idx file also contains file length, but this data is already stored by the filesystem.
> If nobody has objections, I'd like to write a patch to eliminate the split.idx file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-1524) Task Logs userlogs don't show up for a while

Hudson (Jira)
In reply to this post by Hudson (Jira)

    [ https://issues.apache.org/jira/browse/HADOOP-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512984 ]

Owen O'Malley commented on HADOOP-1524:
---------------------------------------

This looks good.

> Task Logs userlogs don't show up for a while
> ---------------------------------------------
>
>                 Key: HADOOP-1524
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1524
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Michael Bieniosek
>         Attachments: accelerate-task-log.patch, eliminate-split-idx.patch
>
>
> When I start a task and go to the task logs, nothing shows up for a while.  An examination of TaskLog.Writer and TaskLog.Reader reveals:
> 1. The TaskLog.Reader relies on the presence of a split.idx to identify the parts of the logs to display.
> 2. The TaskLog.Writer only updates the split.idx file when it moves on to the next log.
> As a result, updates to the log only get pushed when an entire file is done.
> Why is there a split.idx file?  It seems that since files are called part-00000, part-00001, etc., the TaskLog.Reader can just look at all files and arrange them by alphabetical order.  The split.idx file also contains file length, but this data is already stored by the filesystem.
> If nobody has objections, I'd like to write a patch to eliminate the split.idx file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-1524) Task Logs userlogs don't show up for a while

Hudson (Jira)
In reply to this post by Hudson (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-1524:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.14.0
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Michael!

> Task Logs userlogs don't show up for a while
> ---------------------------------------------
>
>                 Key: HADOOP-1524
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1524
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Michael Bieniosek
>             Fix For: 0.14.0
>
>         Attachments: accelerate-task-log.patch, eliminate-split-idx.patch
>
>
> When I start a task and go to the task logs, nothing shows up for a while.  An examination of TaskLog.Writer and TaskLog.Reader reveals:
> 1. The TaskLog.Reader relies on the presence of a split.idx to identify the parts of the logs to display.
> 2. The TaskLog.Writer only updates the split.idx file when it moves on to the next log.
> As a result, updates to the log only get pushed when an entire file is done.
> Why is there a split.idx file?  It seems that since files are called part-00000, part-00001, etc., the TaskLog.Reader can just look at all files and arrange them by alphabetical order.  The split.idx file also contains file length, but this data is already stored by the filesystem.
> If nobody has objections, I'd like to write a patch to eliminate the split.idx file.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

12