[jira] [Commented] (SOLR-5173) Test-only Hadoop dependencies are included in release artifacts

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-5173) Test-only Hadoop dependencies are included in release artifacts

Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13745054#comment-13745054 ]

Steve Rowe commented on SOLR-5173:
----------------------------------

From a #solr-dev IRC conversation today:

{noformat}
10:15 <markrmiller> i think those 3 jars are the compile time dependencies
10:16               it's the test jars that are not
10:16               the hdfs common jar requires auth and annotations for whatever reason
10:16               would be nice if they did something like we do with solrj for the client
10:17               but they dont

10:22 <sarowe>      compile succeeds with only hadoop-common, leaving out -auth,
10:22               -annotations, and -hdfs -- there's just a compilation warning about annotations
10:22               about solrj, I guess you mean a reduced-footprint set of dependencies?
10:23               runtime deps I mean

10:25 <markrmiller> perhaps its runtime then
10:25               been many months since I added them
10:26               but I seem to remember those 3 came in before any test work

10:26 <sarowe>      ok, I'm confused - are you saying that the Ant setup as it is right now,
10:26               without my patch, is as it should be?
10:26               right now, the Ant/Ivy/dist setup for solr doesn't make a distinction
10:27               between compile-time and run-time dependencies

10:27 <markrmiller> not if it was shipping jetty 6
10:27               I've got to look though
10:27               the first draft that i committed
10:27               it was setup right
10:27               all the test stuff was in the test framework
10:27               and core only had the min of what it needed

10:27 <sarowe>      jetty 6 did not ship with 4.4

10:27 <markrmiller> i thought that might be the case - its simply a dependency thing?

10:28 <sarowe>      it's a *maven* dependency thing
10:28               the direct deps for Ivy and Maven are the same

10:28 <markrmiller> not much can be done about it - unless they make a sep client jar
10:28               maven sucks in the world and thats what you get

10:28 <sarowe>      but Maven also pulls in Jetty 6 as an indirect dep

10:28 <markrmiller> yeah, thats a bummer
10:28               but nothing we can do
10:29               thats how maven and the hadoop guys have things

10:29 <sarowe>      well, we can exclude jetty 6 as an indirect dep

10:29 <markrmiller> there is prob a lot you can exclude

10:29 <sarowe>      I did exclude a bunch of stuff
10:30               but I didn't revisit the exclusions when I moved the test stuff
10:30               from test-framework to solr-core unfortunately

10:29 <markrmiller> i found the min jars i needed

10:31 <sarowe>      so to confirm about the -hdfs, -auth, and -annotations jars:
10:31               these are required at run-time, right?

10:31 <markrmiller> as far as I know yes
10:31               one sec, let me look at an older build

10:32 <sarowe>      thanks - I'll change the patch to keep them as compile-time deps then,
10:32               since as I said the Ant/Ivy setup doesn't make a distinction between
10:32               compile-time and run-time deps
10:32               assuming you find that the older build did that

10:34 <markrmiller> what I added and need for compile/runtime was:
10:35               common, hdfs, annotations, auth, commons-configuration, protobuf-java,
10:35               perhaps concurrentlinkedhashmap unless it was already there
10:35               <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-annotations" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="org.apache.hadoop" name="hadoop-auth" rev="${hadoop.version}" transitive="false"/>
10:35               <dependency org="commons-configuration" name="commons-configuration" rev="1.6" transitive="false"/>
10:35               <dependency org="com.google.protobuf" name="protobuf-java" rev="2.4.0a" transitive="false"/>
10:35               <dependency org="com.googlecode.concurrentlinkedhashmap" name="concurrentlinkedhashmap-lru" rev="1.2" transitive="false"/>
{noformat}
               

> Test-only Hadoop dependencies are included in release artifacts
> ---------------------------------------------------------------
>
>                 Key: SOLR-5173
>                 URL: https://issues.apache.org/jira/browse/SOLR-5173
>             Project: Solr
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 4.4
>            Reporter: Steve Rowe
>            Assignee: Steve Rowe
>            Priority: Blocker
>             Fix For: 4.5
>
>         Attachments: SOLR-5173.patch
>
>
> Chris Collins [reported on solr-user|http://markmail.org/message/evhpcougs5ppafjk] that solr-core 4.4 has dependencies on hadoop, and indirectly on jetty 6.
> As a workaround for Maven dependencies, the hadoop-hdfs, hadoop-auth, and hadoop-annotations dependencies can be excluded, which will also exclude the indirect jetty 6 dependency/ies.  hadoop-common is a compile-time dependency, though, so I'm not sure if it's safe to exclude.
> The problems, as far as I can tell, are:
> 1) The ivy configuration puts three test-only dependencies (hadoop-hdfs, hadoo-auth, and hadoop-annotations) in solr/core/lib/, rather than where they belong, in solr/core/test-lib/.  (hadoop-common is required for solr-core compilation to succeed.)
> 2) The Maven configuration makes the equivalent mistake in marking these test-only hadoop dependencies as compile-scope rather than test-scope dependencies.
> 3) The Solr .war, which packages everything under solr/core/lib/, includes these three test-only hadoop dependencies (though it does not include any jetty 6 jars).
> 4) The license files for jetty and jetty-util v6.1.26, but not the jar files corresponding to them, are included in the Solr distribution.
> I have working (tests pass) local Ant and Maven configurations that treat the three hadoop test-only dependencies properly; as result, the .war will no longer contain them - this will cover problems #1-3 above.
> I think we can just remove the jetty and jetty-util 6.1.26 license files from solr/licenses/, since we don't ship those jars.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]