Update guava to 27.0-jre in hadoop-project

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Update guava to 27.0-jre in hadoop-project

Gabor Bota-2
Hi devs,

I'm working on the guava version from 11.0.2 to 27.0-jre in hadoop-project.
We need to do the upgrade because of CVE-2018-10237
<https://nvd.nist.gov/vuln/detail/CVE-2018-10237>.

I've created an issue (HADOOP-15960
<https://issues.apache.org/jira/browse/HADOOP-15960>) to track progress and
created subtasks for hadoop branches 3.0, 3.1, 3.2 and trunk. The first
update should be done in the trunk, and then it can be backported to lower
version branches. Backporting to 2.x is not feasible right now, because of
Guava 20 is the last Java 7 compatible version[1], and we have Java 7
compatibility on version 2 branches - but we are planning to update (
HADOOP-16219 <https://issues.apache.org/jira/browse/HADOOP-16219>).

For the new deprecations after the update, I've created another issue (
HADOOP-16222 <https://issues.apache.org/jira/browse/HADOOP-16222>). Those
can be fixed after the update is committed.

Unit and integration testing in hadoop trunk
There were modifications in the test in the following modules so
precommit tests were running on jenkins:

   - hadoop-common-project
   - hadoop-hdfs-project
   - hadoop-mapreduce-project
   - hadoop-yarn-project

There was one failure but after re-running the test locally it was
successful, so not related to the change.

Because of 5 hour test time limit for jenkins precommit build, I had to run
tests on hadoop-tools manually and the tests were successful. You can find
test results for trunk under HADOOP-16210
<https://issues.apache.org/jira/browse/HADOOP-16210>.

Integration testing with other components
I've done testing with HBase master on hadoop branch-3.0 with guava 27, and
the tests were running fine. Thanks to Peter Somogyi for help.
We are planning to do some testing with Peter Vary on Hive with branch-3.1
this week.

Thanks,
Gabor

[1]
https://groups.google.com/forum/#!msg/guava-discuss/ZRmDJnAq9T0/-HExv44eCAAJ
Reply | Threaded
Open this post in threaded view
|

Re: Update guava to 27.0-jre in hadoop-project

Steve Loughran-4
I know that the number of guava updates we could call painless is 0, but we
need to do this.

The last time we successfully updated Guava was 2012: h
ttps://issues.apache.org/jira/browse/HDFS-3187
That was the java 6 era

The last unsuccessful attempt, April 2017:
https://issues.apache.org/jira/browse/HADOOP-14386

Let's try again and this time if there are problems say: sorry, but its
time to move on.

I think we should only worry about branch-3.2+ for now, though the other
branches could be lined up for those changes needed to ensure that
everything builds if you explicitly set the version (e.g findbugs changes.
Then we can worry about 3.1.x line, which is the 3.x branch most widely
picked up to date.

I want to avoid branch-2 entirely, though as Gabor notes, I want to move us
on to java 8 builds there so that people can do a branch-2 build if they
need to.

*Is everyone happy with the proposed patch*:
https://github.com/apache/hadoop/pull/674

-Steve


On Mon, Apr 1, 2019 at 8:35 PM Gabor Bota <[hidden email]>
wrote:

> Hi devs,
>
> I'm working on the guava version from 11.0.2 to 27.0-jre in hadoop-project.
> We need to do the upgrade because of CVE-2018-10237
> <https://nvd.nist.gov/vuln/detail/CVE-2018-10237>.
>
> I've created an issue (HADOOP-15960
> <https://issues.apache.org/jira/browse/HADOOP-15960>) to track progress
> and
> created subtasks for hadoop branches 3.0, 3.1, 3.2 and trunk. The first
> update should be done in the trunk, and then it can be backported to lower
> version branches. Backporting to 2.x is not feasible right now, because of
> Guava 20 is the last Java 7 compatible version[1], and we have Java 7
> compatibility on version 2 branches - but we are planning to update (
> HADOOP-16219 <https://issues.apache.org/jira/browse/HADOOP-16219>).
>
> For the new deprecations after the update, I've created another issue (
> HADOOP-16222 <https://issues.apache.org/jira/browse/HADOOP-16222>). Those
> can be fixed after the update is committed.
>
> Unit and integration testing in hadoop trunk
> There were modifications in the test in the following modules so
> precommit tests were running on jenkins:
>
>    - hadoop-common-project
>    - hadoop-hdfs-project
>    - hadoop-mapreduce-project
>    - hadoop-yarn-project
>
> There was one failure but after re-running the test locally it was
> successful, so not related to the change.
>
> Because of 5 hour test time limit for jenkins precommit build, I had to run
> tests on hadoop-tools manually and the tests were successful. You can find
> test results for trunk under HADOOP-16210
> <https://issues.apache.org/jira/browse/HADOOP-16210>.
>
> Integration testing with other components
> I've done testing with HBase master on hadoop branch-3.0 with guava 27, and
> the tests were running fine. Thanks to Peter Somogyi for help.
> We are planning to do some testing with Peter Vary on Hive with branch-3.1
> this week.
>
> Thanks,
> Gabor
>
> [1]
>
> https://groups.google.com/forum/#!msg/guava-discuss/ZRmDJnAq9T0/-HExv44eCAAJ
>