[VOTE] Release Apache Hadoop 3.1.0 (RC1)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] Release Apache Hadoop 3.1.0 (RC1)

Wangda Tan
Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Rohith Sharma K S-2
+1 (binding)

* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell  and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.

Thanks & Regards
Rohith Sharma K S

On 30 March 2018 at 09:45, Wangda Tan <[hidden email]> wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Eric Payne-4
In reply to this post by Wangda Tan
 +1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent

- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption

- Tested userlimit first intra-queue preemption

Thanks,Eric Payne


===========================================================
    On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan <[hidden email]> wrote:  
 
 Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Vinod Kumar Vavilapalli-2
In reply to this post by Wangda Tan
We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now.

Thanks
+Vinod

> On Apr 3, 2018, at 3:30 PM, Wangda Tan <[hidden email]> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <[hidden email] <mailto:[hidden email]>> wrote:
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------------------------------------------------------------------
> From: Wangda Tan <[hidden email] <mailto:[hidden email]>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <[hidden email] <mailto:[hidden email]>>
> Cc: Gera Shegalov <[hidden email] <mailto:[hidden email]>>, Sunil G <[hidden email] <mailto:[hidden email]>>, "[hidden email] <mailto:[hidden email]>" <[hidden email] <mailto:[hidden email]>>, Hdfs-dev <[hidden email] <mailto:[hidden email]>>, Hadoop Common <[hidden email] <mailto:[hidden email]>>, "[hidden email] <mailto:[hidden email]>" <[hidden email] <mailto:[hidden email]>>, Vinod Kumar Vavilapalli <[hidden email] <mailto:[hidden email]>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
>
> Since the jars inside binary tarballs are correct (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:[hidden email] <mailto:[hidden email]>> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:[hidden email] <mailto:[hidden email]>> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums <https://www.apache.org/dev/release-distribution#sigs-and-sums>
>
>  
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:[hidden email] <mailto:[hidden email]>> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:[hidden email] <mailto:[hidden email]>> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ <http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/> .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:[hidden email] <mailto:[hidden email]>> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>   - Below feature sanity is done
>      - Application priority
>      - Application timeout
>      - Intra Queue preemption with priority based
>      - DS based affinity tests to verify placement constraints.
>   - Tested basic NodeLabel scenarios.
>      - Added couple of labels to few of nodes and behavior is coming
>      correct.
>      - Verified old UI  and new YARN UI for labels.
>      - Submitted apps to labelled cluster and it works fine.
>      - Also performed few cli commands related to nodelabel.
>   - Test basic HA cases and seems correct.
>   - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:[hidden email] <mailto:[hidden email]>> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1 <http://people.apache.org/~wangda/hadoop-3.1.0-RC1>
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via http://repository.apache.org <http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/ <https://repository.apache.org/content/repositories/>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b <https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b>
> c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 <https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7>
> 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org <http://3cyarn-dev.hadoop.apache.org/>%3E
>
>
>
>
>
>
>
>
>