[VOTE] Release Apache Hadoop 3.2.0 - RC0

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] Release Apache Hadoop 3.2.0 - RC0

Sunil G
Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

Ayush Saxena
Sunil, Thanks for driving this release!!!

+1 (non-binding)
-Built from source on Ubuntu 17.04 and JDK 8
-Ran basic Hdfs Commands
-Ran basic EC Commands
-Ran basic RBF commands
-Browsed HDFS and RBF WEB UI
-Ran TeraGen/TeraSort

Regards
Ayush

> On 27-Nov-2018, at 4:06 PM, Kitti Nanasi <[hidden email]> wrote:
>
> Thanks Sunil for the driving the release!
>
> +1 (non-binding)
>
> - checked out git tag release-3.2.0-RC0
> - built from source on Mac OS X 10.13.4, java version 8.0.172-zulu
> - deployed on a 5 node cluster
> - ran terasort, teragen, teravalidate with success
> - executed basic hdfs, dfsadmin and ec commands
>
> Best,
> Kitti
>
>> On Fri, Nov 23, 2018 at 1:07 PM Sunil G <[hidden email]> wrote:
>>
>> Hi folks,
>>
>>
>>
>> Thanks to all contributors who helped in this release [1]. I have created
>>
>> first release candidate (RC0) for Apache Hadoop 3.2.0.
>>
>>
>> Artifacts for this RC are available here:
>>
>> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>>
>>
>>
>> RC tag in git is release-3.2.0-RC0.
>>
>>
>>
>> The maven artifacts are available via repository.apache.org at
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>>
>>
>> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>>
>>
>>
>> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
>> additions
>>
>> are the highlights of this release.
>>
>> 1. Node Attributes Support in YARN
>>
>> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>>
>> 3. Support service upgrade via YARN Service API and CLI
>>
>> 4. HDFS Storage Policy Satisfier
>>
>> 5. Support Windows Azure Storage - Blob file system in Hadoop
>>
>> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>>
>> 7. Improvements in Router-based HDFS federation
>>
>>
>>
>> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>>
>> I have done few testing with my pseudo cluster. My +1 to start.
>>
>>
>>
>> Regards,
>>
>> Sunil
>>
>>
>>
>> [1]
>>
>>
>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
>> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
>> ORDER BY fixVersion ASC
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

Peter Bacsko
In reply to this post by Sunil G
+1 (non-binding)

- Built from source at tag 3.2.0-rc0 (Ubuntu 18.10, JDK1.8.0_191)
- Verified checksums of hadoop-3.2.0.tar.gz
- Installed on a 3-node physical cluster
- Ran teragen/terasort/teravalidate
- Ran distributed shell a couple of times
- Checked UIs (RM, NM, DN, JHS)

Peter

On Wed, Nov 28, 2018 at 5:17 PM Jason Lowe <[hidden email]> wrote:

> Thanks for driving this release, Sunil!
>
> +1 (binding)
>
> - Verified signatures and digests
> - Successfully performed a native build
> - Deployed a single-node cluster
> - Ran some sample jobs
>
> Jason
>
> On Fri, Nov 23, 2018 at 6:07 AM Sunil G <[hidden email]> wrote:
>
> > Hi folks,
> >
> >
> >
> > Thanks to all contributors who helped in this release [1]. I have created
> >
> > first release candidate (RC0) for Apache Hadoop 3.2.0.
> >
> >
> > Artifacts for this RC are available here:
> >
> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> >
> >
> >
> > RC tag in git is release-3.2.0-RC0.
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> >
> >
> >
> > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > additions
> >
> > are the highlights of this release.
> >
> > 1. Node Attributes Support in YARN
> >
> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> >
> > 3. Support service upgrade via YARN Service API and CLI
> >
> > 4. HDFS Storage Policy Satisfier
> >
> > 5. Support Windows Azure Storage - Blob file system in Hadoop
> >
> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> >
> > 7. Improvements in Router-based HDFS federation
> >
> >
> >
> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> >
> > I have done few testing with my pseudo cluster. My +1 to start.
> >
> >
> >
> > Regards,
> >
> > Sunil
> >
> >
> >
> > [1]
> >
> >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > ORDER BY fixVersion ASC
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

Wilfred Spiegelenburg
In reply to this post by Sunil G
+1 (non binding)

- build from source on MacOSX 10.14.1, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (jdk 1.8u191 / centos 7.5)

Wilfred

> On 23 Nov 2018, at 23:06, Sunil G <[hidden email]> wrote:
>
> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com <https://www.cloudera.com/>







Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

Sunil G
In reply to this post by Sunil G
Thanks @Eric Payne <[hidden email]>
Due to another issue, we have to spin an RC1. In this case, could we revert
the patch which caused this problem.

Cancelling this RC0. Thanks to everyone who voted.
I will spin RC1 as soon as possible.

- Sunil


On Fri, Nov 30, 2018 at 4:14 AM Eric Payne <[hidden email]>
wrote:

> The problem is not with preemption. The yarn-site.xml that I use for my
> pseudo-cluster includes a second xml:
> xi:include href=".../yarn-scheduler.xml"
>
> The property for yarn.resourcemanager.scheduler.monitor.enable = true is
> in this yarn-scheduler.xml.
>
> This value IS READ when then RM starts.
>
> However, when the refreshQueues command is run, this value IS NOT READ.
>
> So, it looks like xml include files are not read on refresh. This will
> affect any property. I just happened to notice it on the preemption
> properties.
>
> I would like input from all of you to determine if this is a blocker for
> release. I'm on the fence.
>
> Thanks,
> -Eric
>
>
>
>
>
>
> On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne <
> [hidden email]> wrote:
>
>
>
>
>
> Sunil,
>
> So, the basic symptoms are that if preemption is enabled on any queue, the
> preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all
> of the preemption-specific properties are set back to the default values.
>
> This was introduced in branch-3.1, so it is NOT new behavior for release
> 3.2.0. I am still tracking down the cause. I will open a JIRA once I do
> further investigation if there is not one already.
>
> This will be a problem for installations which use preemption and which
> use the refreshQueues feature.
>
> Thanks,
> -Eric
>
>
> On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <
> [hidden email]> wrote:
>
>
>
>
>
> Sunil, thanks for all of the hard work on this release.
>
> I have discovered that queue refresh doesn't work in some cases. For
> example, when I change
> yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take
> effect unless I restart the RM.
>
> I am still investigating, but I thought I should bring this up asap.
>
> Thanks,
> -Eric
>
>
>
>
> On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G <[hidden email]>
> wrote:
>
>
>
>
>
> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>