Guidance needed on HADOOP-13096 and HADOOP-13097

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Guidance needed on HADOOP-13096 and HADOOP-13097

Allen Wittenauer-2

        When the sub-projects re-merged, maven work was done, whatever, the shell scripts for MR and YARN were placed (effectively) outside of the normal maven hierarchy.  In order to add unit tests to the shell scripts for these sub-projects, it means effectively turning hadoop-yarn-project/hadoop-yarn and hadoop-mapreduce-project into “real” modules so that mvn test works as expected.   Doing so will likely have some surprising consequences, such as anyone who modifies java code and the shell code in a patch will trigger _all_ of the unit tests in yarn.

        I think we have four options:

a) Continue forward turning these into real modules with src directories, etc and we live with the consequences

b) Move the related bits into an existing module, making them similar to HDFS, common, tools

c) Move the related bits into a new module, using the layout that maven really really wants

d) Skip the unit tests; we don’t have them now

        This is clearly more work than what I really wanted to cover in this branch, but given that there was a specific request to add unit test code for this functionality, I’m sort of stuck here.

        Thoughts?
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Guidance needed on HADOOP-13096 and HADOOP-13097

Larry McCay
I would vote for C or D with a filed JIRA to clean up the maven structure as a separate effort.
Before moving to D, could you describe any reason to not go with C?

On May 4, 2016, at 9:51 PM, Allen Wittenauer <[hidden email]> wrote:

>
> When the sub-projects re-merged, maven work was done, whatever, the shell scripts for MR and YARN were placed (effectively) outside of the normal maven hierarchy.  In order to add unit tests to the shell scripts for these sub-projects, it means effectively turning hadoop-yarn-project/hadoop-yarn and hadoop-mapreduce-project into “real” modules so that mvn test works as expected.   Doing so will likely have some surprising consequences, such as anyone who modifies java code and the shell code in a patch will trigger _all_ of the unit tests in yarn.
>
> I think we have four options:
>
> a) Continue forward turning these into real modules with src directories, etc and we live with the consequences
>
> b) Move the related bits into an existing module, making them similar to HDFS, common, tools
>
> c) Move the related bits into a new module, using the layout that maven really really wants
>
> d) Skip the unit tests; we don’t have them now
>
> This is clearly more work than what I really wanted to cover in this branch, but given that there was a specific request to add unit test code for this functionality, I’m sort of stuck here.
>
> Thoughts?
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Guidance needed on HADOOP-13096 and HADOOP-13097

Allen Wittenauer-2
        After thinking about it, I think you are correct here: I’m more inclined to do D w/follow-up JIRAs to fix this up. The hadoop and hdfs script functionality is being tested, so it isn’t like HADOOP-12930 is going in with zero unit tests. Never mind that large chunks of hadoop-tools gets modified to use this “for reals” as well. The yarn and mapred tests don’t really bring _that_ much to the table.

        I think the biggest worry about doing C inside the HADOOP-12930 feature branch is that it seems like the wrong time/place to do it.  Making that big of a change to the build should probably be two separate, orthogonal JIRAs (one for YARN, one for MR) in their own right.  But I do think C is the correct, long-term path.  We should probably move hdfs and common scripts into separate dirs as well, honestly.

        Thanks for the feedback!


> On May 5, 2016, at 7:22 PM, Larry McCay <[hidden email]> wrote:
>
> I would vote for C or D with a filed JIRA to clean up the maven structure as a separate effort.
> Before moving to D, could you describe any reason to not go with C?
>
> On May 4, 2016, at 9:51 PM, Allen Wittenauer <[hidden email]> wrote:
>
>>
>> When the sub-projects re-merged, maven work was done, whatever, the shell scripts for MR and YARN were placed (effectively) outside of the normal maven hierarchy.  In order to add unit tests to the shell scripts for these sub-projects, it means effectively turning hadoop-yarn-project/hadoop-yarn and hadoop-mapreduce-project into “real” modules so that mvn test works as expected.   Doing so will likely have some surprising consequences, such as anyone who modifies java code and the shell code in a patch will trigger _all_ of the unit tests in yarn.
>>
>> I think we have four options:
>>
>> a) Continue forward turning these into real modules with src directories, etc and we live with the consequences
>>
>> b) Move the related bits into an existing module, making them similar to HDFS, common, tools
>>
>> c) Move the related bits into a new module, using the layout that maven really really wants
>>
>> d) Skip the unit tests; we don’t have them now
>>
>> This is clearly more work than what I really wanted to cover in this branch, but given that there was a specific request to add unit test code for this functionality, I’m sort of stuck here.
>>
>> Thoughts?
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Guidance needed on HADOOP-13096 and HADOOP-13097

larry mccay-2
I agree with your rationale for not doing C now.
And those clean up tasks can more easily be discussed when separated from
this effort.


On Fri, May 6, 2016 at 3:11 PM, Allen Wittenauer <[hidden email]> wrote:

>         After thinking about it, I think you are correct here: I’m more
> inclined to do D w/follow-up JIRAs to fix this up. The hadoop and hdfs
> script functionality is being tested, so it isn’t like HADOOP-12930 is
> going in with zero unit tests. Never mind that large chunks of hadoop-tools
> gets modified to use this “for reals” as well. The yarn and mapred tests
> don’t really bring _that_ much to the table.
>
>         I think the biggest worry about doing C inside the HADOOP-12930
> feature branch is that it seems like the wrong time/place to do it.  Making
> that big of a change to the build should probably be two separate,
> orthogonal JIRAs (one for YARN, one for MR) in their own right.  But I do
> think C is the correct, long-term path.  We should probably move hdfs and
> common scripts into separate dirs as well, honestly.
>
>         Thanks for the feedback!
>
>
> > On May 5, 2016, at 7:22 PM, Larry McCay <[hidden email]> wrote:
> >
> > I would vote for C or D with a filed JIRA to clean up the maven
> structure as a separate effort.
> > Before moving to D, could you describe any reason to not go with C?
> >
> > On May 4, 2016, at 9:51 PM, Allen Wittenauer <[hidden email]> wrote:
> >
> >>
> >>      When the sub-projects re-merged, maven work was done, whatever,
> the shell scripts for MR and YARN were placed (effectively) outside of the
> normal maven hierarchy.  In order to add unit tests to the shell scripts
> for these sub-projects, it means effectively turning
> hadoop-yarn-project/hadoop-yarn and hadoop-mapreduce-project into “real”
> modules so that mvn test works as expected.   Doing so will likely have
> some surprising consequences, such as anyone who modifies java code and the
> shell code in a patch will trigger _all_ of the unit tests in yarn.
> >>
> >>      I think we have four options:
> >>
> >> a) Continue forward turning these into real modules with src
> directories, etc and we live with the consequences
> >>
> >> b) Move the related bits into an existing module, making them similar
> to HDFS, common, tools
> >>
> >> c) Move the related bits into a new module, using the layout that maven
> really really wants
> >>
> >> d) Skip the unit tests; we don’t have them now
> >>
> >>      This is clearly more work than what I really wanted to cover in
> this branch, but given that there was a specific request to add unit test
> code for this functionality, I’m sort of stuck here.
> >>
> >>      Thoughts?
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>