[jira] [Commented] (SOLR-5173) Test-only Hadoop dependencies are included in release artifacts

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (SOLR-5173) Test-only Hadoop dependencies are included in release artifacts

Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745054#comment-13745054 ]

Steve Rowe commented on SOLR-5173:

From a #solr-dev IRC conversation today:

10:15 <markrmiller> i think those 3 jars are the compile time dependencies
10:16               it's the test jars that are not
10:16               the hdfs common jar requires auth and annotations for whatever reason
10:16               would be nice if they did something like we do with solrj for the client
10:17               but they dont

10:22 <sarowe>      compile succeeds with only hadoop-common, leaving out -auth,
10:22               -annotations, and -hdfs -- there's just a compilation warning about annotations
10:22               about solrj, I guess you mean a reduced-footprint set of dependencies?
10:23               runtime deps I mean

10:25 <markrmiller> perhaps its runtime then
10:25               been many months since I added them
10:26               but I seem to remember those 3 came in before any test work

10:26 <sarowe>      ok, I'm confused - are you saying that the Ant setup as it is right now,
10:26               without my patch, is as it should be?
10:26               right now, the Ant/Ivy/dist setup for solr doesn't make a distinction
10:27               between compile-time and run-time dependencies

10:27 <markrmiller> not if it was shipping jetty 6
10:27               I've got to look though
10:27               the first draft that i committed
10:27               it was setup right
10:27               all the test stuff was in the test framework
10:27               and core only had the min of what it needed

10:27 <sarowe>      jetty 6 did not ship with 4.4

10:27 <markrmiller> i thought that might be the case - its simply a dependency thing?

10:28 <sarowe>      it's a *maven* dependency thing
10:28               the direct deps for Ivy and Maven are the same

10:28 <markrmiller> not much can be done about it - unless they make a sep client jar
10:28               maven sucks in the world and thats what you get

10:28 <sarowe>      but Maven also pulls in Jetty 6 as an indirect dep

10:28 <markrmiller> yeah, thats a bummer
10:28               but nothing we can do
10:29               thats how maven and the hadoop guys have things

10:29 <sarowe>      well, we can exclude jetty 6 as an indirect dep

10:29 <markrmiller> there is prob a lot you can exclude

10:29 <sarowe>      I did exclude a bunch of stuff
10:30               but I didn't revisit the exclusions when I moved the test stuff
10:30               from test-framework to solr-core unfortunately

10:29 <markrmiller> i found the min jars i needed

10:31 <sarowe>      so to confirm about the -hdfs, -auth, and -annotations jars:
10:31               these are required at run-time, right?

10:31 <markrmiller> as far as I know yes
10:31               one sec, let me look at an older build

10:32 <sarowe>      thanks - I'll change the patch to keep them as compile-time deps then,
10:32               since as I said the Ant/Ivy setup doesn't make a distinction between
10:32               compile-time and run-time deps
10:32               assuming you find that the older build did that

10:34 <markrmiller> what I added and need for compile/runtime was:
10:35               common, hdfs, annotations, auth, commons-configuration, protobuf-java,
10:35               perhaps concurrentlinkedhashmap unless it was already there
10:35               <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-annotations" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-auth" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="commons-configuration" name="commons-configuration" rev="1.6" transitive="false"/>
10:35               <dependency org="com.google.protobuf" name="protobuf-java" rev="2.4.0a" transitive="false"/>
10:35               <dependency org="com.googlecode.concurrentlinkedhashmap" name="concurrentlinkedhashmap-lru" rev="1.2" transitive="false"/>

> Test-only Hadoop dependencies are included in release artifacts
> ---------------------------------------------------------------
>                 Key: SOLR-5173
>                 URL: https://issues.apache.org/jira/browse/SOLR-5173
>             Project: Solr
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 4.4
>            Reporter: Steve Rowe
>            Assignee: Steve Rowe
>            Priority: Blocker
>             Fix For: 4.5
>         Attachments: SOLR-5173.patch
> Chris Collins [reported on solr-user|http://markmail.org/message/evhpcougs5ppafjk] that solr-core 4.4 has dependencies on hadoop, and indirectly on jetty 6.
> As a workaround for Maven dependencies, the hadoop-hdfs, hadoop-auth, and hadoop-annotations dependencies can be excluded, which will also exclude the indirect jetty 6 dependency/ies.  hadoop-common is a compile-time dependency, though, so I'm not sure if it's safe to exclude.
> The problems, as far as I can tell, are:
> 1) The ivy configuration puts three test-only dependencies (hadoop-hdfs, hadoo-auth, and hadoop-annotations) in solr/core/lib/, rather than where they belong, in solr/core/test-lib/.  (hadoop-common is required for solr-core compilation to succeed.)
> 2) The Maven configuration makes the equivalent mistake in marking these test-only hadoop dependencies as compile-scope rather than test-scope dependencies.
> 3) The Solr .war, which packages everything under solr/core/lib/, includes these three test-only hadoop dependencies (though it does not include any jetty 6 jars).
> 4) The license files for jetty and jetty-util v6.1.26, but not the jar files corresponding to them, are included in the Solr distribution.
> I have working (tests pass) local Ant and Maven configurations that treat the three hadoop test-only dependencies properly; as result, the .war will no longer contain them - this will cover problems #1-3 above.
> I think we can just remove the jetty and jetty-util 6.1.26 license files from solr/licenses/, since we don't ship those jars.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]