[jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to take the same globbed input specification

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-619) Unify Map-Reduce and Streaming to take the same globbed input specification

JIRA jira@apache.org

    [ http://issues.apache.org/jira/browse/HADOOP-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461692 ]

Sanjay Dahiya commented on HADOOP-619:
--------------------------------------

>One other thing I just noticed is that it looks like globbed paths are not fully qualified with scheme and authority. This will cause problems if patterns from multiple filesystems are included in a job's input. We don't test that yet, so this won't break anything today, but we'd like that to work.

Doug:
Actually fully qualified paths should work with globbing, for example following will work with this patch -
bin/hadoop jar build/hadoop-0.9.3-dev-examples.jar wordcount  hdfs://<host>:9015/<dir>/*.txt <out>
Or were referring to something else?

> Unify Map-Reduce and Streaming to take the same globbed input specification
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-619
>                 URL: http://issues.apache.org/jira/browse/HADOOP-619
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.9.1
>            Reporter: eric baldeschwieler
>         Assigned To: Sanjay Dahiya
>             Fix For: 0.10.0
>
>         Attachments: Hadoop-619.patch, Hadoop-619.patch, Hadoop-619.patch, Hadoop-619_1.patch, Hadoop-619_1.patch, Hadoop-619_2.patch, Hadoop-619_2.patch, Hadoop-619_3.patch
>
>
> Right now streaming input is specified very differently from other map-reduce input.  It would be good if these two apps could take much more similar input specs.
> In particular -input in streaming expects a file or glob pattern while MR takes a directory.  It would be cool if both could take a glob patern of files and if both took a directory by default (with some patern excluded to allow logs, metadata and other framework output to be safely stored).
> We want to be sure that MR input is backward compatible over this change.  I propose that a single file should be accepted as an input or a single directory.  Globs should only match directories if the paterns is '/' terminated, to avoid massive inputs specified by mistake.
> Thoughts?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira