[jira] [Created] (HADOOP-17299) DistCp should fail fast if the configuration is not valid

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (HADOOP-17299) DistCp should fail fast if the configuration is not valid

Nicholas DiPiazza (Jira)
Akira Ajisaka created HADOOP-17299:
--------------------------------------

             Summary: DistCp should fail fast if the configuration is not valid
                 Key: HADOOP-17299
                 URL: https://issues.apache.org/jira/browse/HADOOP-17299
             Project: Hadoop Common
          Issue Type: Bug
          Components: tools/distcp
            Reporter: Akira Ajisaka


If the number of Map task is more than 200 and using dynamic option, DistCp always fails by DynamicInputFormat#validateNumChunksUsing. This failure is after building the src/dst paths, which takes more than 10 minutes if the directory is large (i.e. DistCp fails by configuration error after more than 10 minutes).

I think DistCp should validate the configuration before building the src/dst paths and fail if the configuration is not valid.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]