Planning Hadoop 2.6.1 release

classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Planning Hadoop 2.6.1 release

Vinod Kumar Vavilapalli-2
There were several requests on the user lists [1] for a 2.6.1 release. I
got many offline comments too.

Planning to do a 2.6.1 release in a few weeks time. We already have a bunch
of tickets committed to 2.7.1. I created a filter [2] to tracking pending
tickets.

We need to collectively come up with a list of critical issues. We can use
the JIRA Target Version field for the same. I see some but not a whole lot
of new work for this release, most of it is likely going to be pulling in
critical patches from 2.7.1/2.8 etc.

Thoughts?

Thanks
+Vinod

[1] Will Hadoop 2.6.1 be released soon?
http://markmail.org/thread/zlsr6prejyogdyvh
[2] 2.6.1 pending tickets
https://issues.apache.org/jira/issues/?filter=12331711
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Arpit Agarwal
HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic and improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
HDFS-7503. Namenode restart after large deletions can cause slow processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by client.
HDFS-7960. The full block report should prune zombie storages even if they're not empty.
HDFS-8072. Reserved RBW space is not released if client terminates while writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon? http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]> wrote:

>There were several requests on the user lists [1] for a 2.6.1 release. I
>got many offline comments too.
>
>Planning to do a 2.6.1 release in a few weeks time. We already have a bunch
>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>tickets.
>
>We need to collectively come up with a list of critical issues. We can use
>the JIRA Target Version field for the same. I see some but not a whole lot
>of new work for this release, most of it is likely going to be pulling in
>critical patches from 2.7.1/2.8 etc.
>
>Thoughts?
>
>Thanks
>+Vinod
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>[2] 2.6.1 pending tickets
>https://issues.apache.org/jira/issues/?filter=12331711
>


Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Chris Nauroth
Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:

>HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>requested in [1].
>
>HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>should be non static
>HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>synchronization
>
>HDFS-7009. Active NN and standby NN have different live nodes.
>HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>improve error handling
>HDFS-7425. NameNode block deletion logging uses incorrect appender.
>HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>block files are present in the same volume.
>HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>HDFS-7503. Namenode restart after large deletions can cause slow
>processReport.
>HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>HDFS-7579. Improve log reporting during block report rpc failure.
>HDFS-7587. Edit log corruption can happen if append fails with a quota
>violation.
>HDFS-7596. NameNode should prune dead storages from storageMap.
>HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>in the blocksMap on NameNode restart.
>HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>DataNode to register successfully with only one NameNode.
>HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>failure.
>HDFS-7831. Fix the starting index and end condition of the loop in
>FileDiffList.findEarlierSnapshotBlocks().
>HDFS-7885. Datanode should not trust the generation stamp provided by
>client.
>HDFS-7960. The full block report should prune zombie storages even if
>they're not empty.
>HDFS-8072. Reserved RBW space is not released if client terminates while
>writing block.
>HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>finalize upgrade.
>
>
>Arpit
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
>On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>wrote:
>
>>There were several requests on the user lists [1] for a 2.6.1 release. I
>>got many offline comments too.
>>
>>Planning to do a 2.6.1 release in a few weeks time. We already have a
>>bunch
>>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>tickets.
>>
>>We need to collectively come up with a list of critical issues. We can
>>use
>>the JIRA Target Version field for the same. I see some but not a whole
>>lot
>>of new work for this release, most of it is likely going to be pulling in
>>critical patches from 2.7.1/2.8 etc.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>>
>>[1] Will Hadoop 2.6.1 be released soon?
>>http://markmail.org/thread/zlsr6prejyogdyvh
>>[2] 2.6.1 pending tickets
>>https://issues.apache.org/jira/issues/?filter=12331711
>>
>
>

Reply | Threaded
Open this post in threaded view
|

RE: Planning Hadoop 2.6.1 release

Brahma Reddy
Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will not be usable when cluster installed in HA mode and try to get filecontext like below..


Path path = new Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration());



Thanks & Regards
Brahma Reddy Battula
________________________________________
From: Chris Nauroth [[hidden email]]
Sent: Friday, May 01, 2015 4:32 AM
To: [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:

>HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>requested in [1].
>
>HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>should be non static
>HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>synchronization
>
>HDFS-7009. Active NN and standby NN have different live nodes.
>HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>improve error handling
>HDFS-7425. NameNode block deletion logging uses incorrect appender.
>HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>block files are present in the same volume.
>HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>HDFS-7503. Namenode restart after large deletions can cause slow
>processReport.
>HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>HDFS-7579. Improve log reporting during block report rpc failure.
>HDFS-7587. Edit log corruption can happen if append fails with a quota
>violation.
>HDFS-7596. NameNode should prune dead storages from storageMap.
>HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>in the blocksMap on NameNode restart.
>HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>DataNode to register successfully with only one NameNode.
>HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>failure.
>HDFS-7831. Fix the starting index and end condition of the loop in
>FileDiffList.findEarlierSnapshotBlocks().
>HDFS-7885. Datanode should not trust the generation stamp provided by
>client.
>HDFS-7960. The full block report should prune zombie storages even if
>they're not empty.
>HDFS-8072. Reserved RBW space is not released if client terminates while
>writing block.
>HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>finalize upgrade.
>
>
>Arpit
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
>On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>wrote:
>
>>There were several requests on the user lists [1] for a 2.6.1 release. I
>>got many offline comments too.
>>
>>Planning to do a 2.6.1 release in a few weeks time. We already have a
>>bunch
>>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>tickets.
>>
>>We need to collectively come up with a list of critical issues. We can
>>use
>>the JIRA Target Version field for the same. I see some but not a whole
>>lot
>>of new work for this release, most of it is likely going to be pulling in
>>critical patches from 2.7.1/2.8 etc.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>>
>>[1] Will Hadoop 2.6.1 be released soon?
>>http://markmail.org/thread/zlsr6prejyogdyvh
>>[2] 2.6.1 pending tickets
>>https://issues.apache.org/jira/issues/?filter=12331711
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Akira AJISAKA
Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
to include the following:

* HADOOP-11343. Overflow is not properly handled in calculating final iv
for AES CTR
* YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
execute any further apps
* YARN-2992. ZKRMStateStore crashes due to session expiry
* YARN-3013. AMRMClientImpl does not update AMRM token properly
* YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die
* MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to a reducer

All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.

Regards,
Akira

On 5/4/15 11:15, Brahma Reddy Battula wrote:

> Hello Vinod,
>
> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will not be usable when cluster installed in HA mode and try to get filecontext like below..
>
>
> Path path = new Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
> FileSystem fs = path.getFileSystem(new Configuration());
> path = fs.makeQualified(path);
> FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration());
>
>
>
> Thanks & Regards
> Brahma Reddy Battula
> ________________________________________
> From: Chris Nauroth [[hidden email]]
> Sent: Friday, May 01, 2015 4:32 AM
> To: [hidden email]; [hidden email]; [hidden email]; [hidden email]
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Thank you, Arpit.  In addition, I suggest we include the following:
>
> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
> pipe is full
> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
> sockets during shutdown of DomainSocketWatcher thread.
> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
> is an I/O error during requestShortCircuitShm
>
> HADOOP-11604 and 11648 are not critical by themselves, but they are
> pre-requisites to getting a clean cherry-pick of 11802, which we believe
> finally fixes the root cause of this issue.
>
>
> --Chris Nauroth
>
>
>
>
> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
>
>> HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>> requested in [1].
>>
>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>> should be non static
>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>> synchronization
>>
>> HDFS-7009. Active NN and standby NN have different live nodes.
>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>> improve error handling
>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>> block files are present in the same volume.
>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>> HDFS-7503. Namenode restart after large deletions can cause slow
>> processReport.
>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>> HDFS-7579. Improve log reporting during block report rpc failure.
>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>> violation.
>> HDFS-7596. NameNode should prune dead storages from storageMap.
>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>> in the blocksMap on NameNode restart.
>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>> DataNode to register successfully with only one NameNode.
>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>> failure.
>> HDFS-7831. Fix the starting index and end condition of the loop in
>> FileDiffList.findEarlierSnapshotBlocks().
>> HDFS-7885. Datanode should not trust the generation stamp provided by
>> client.
>> HDFS-7960. The full block report should prune zombie storages even if
>> they're not empty.
>> HDFS-8072. Reserved RBW space is not released if client terminates while
>> writing block.
>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>> finalize upgrade.
>>
>>
>> Arpit
>>
>> [1] Will Hadoop 2.6.1 be released soon?
>> http://markmail.org/thread/zlsr6prejyogdyvh
>>
>>
>>
>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>> wrote:
>>
>>> There were several requests on the user lists [1] for a 2.6.1 release. I
>>> got many offline comments too.
>>>
>>> Planning to do a 2.6.1 release in a few weeks time. We already have a
>>> bunch
>>> of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>> tickets.
>>>
>>> We need to collectively come up with a list of critical issues. We can
>>> use
>>> the JIRA Target Version field for the same. I see some but not a whole
>>> lot
>>> of new work for this release, most of it is likely going to be pulling in
>>> critical patches from 2.7.1/2.8 etc.
>>>
>>> Thoughts?
>>>
>>> Thanks
>>> +Vinod
>>>
>>> [1] Will Hadoop 2.6.1 be released soon?
>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>> [2] 2.6.1 pending tickets
>>> https://issues.apache.org/jira/issues/?filter=12331711
>>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Zhihai Xu
Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <[hidden email]>
wrote:

> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
>> Hello Vinod,
>>
>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>> will not be usable when cluster installed in HA mode and try to get
>> filecontext like below..
>>
>>
>> Path path = new
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>> FileSystem fs = path.getFileSystem(new Configuration());
>> path = fs.makeQualified(path);
>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>> Configuration());
>>
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>> ________________________________________
>> From: Chris Nauroth [[hidden email]]
>> Sent: Friday, May 01, 2015 4:32 AM
>> To: [hidden email]; [hidden email];
>> [hidden email]; [hidden email]
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Thank you, Arpit.  In addition, I suggest we include the following:
>>
>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>> pipe is full
>> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
>> sockets during shutdown of DomainSocketWatcher thread.
>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
>> is an I/O error during requestShortCircuitShm
>>
>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>> finally fixes the root cause of this issue.
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
>>
>>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>> requested in [1].
>>>
>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>> should be non static
>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>> synchronization
>>>
>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>>> improve error handling
>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>> block files are present in the same volume.
>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>> processReport.
>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>> violation.
>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>>> in the blocksMap on NameNode restart.
>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>> DataNode to register successfully with only one NameNode.
>>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>>> failure.
>>> HDFS-7831. Fix the starting index and end condition of the loop in
>>> FileDiffList.findEarlierSnapshotBlocks().
>>> HDFS-7885. Datanode should not trust the generation stamp provided by
>>> client.
>>> HDFS-7960. The full block report should prune zombie storages even if
>>> they're not empty.
>>> HDFS-8072. Reserved RBW space is not released if client terminates while
>>> writing block.
>>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>>> finalize upgrade.
>>>
>>>
>>> Arpit
>>>
>>> [1] Will Hadoop 2.6.1 be released soon?
>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>
>>>
>>>
>>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>>> wrote:
>>>
>>>  There were several requests on the user lists [1] for a 2.6.1 release. I
>>>> got many offline comments too.
>>>>
>>>> Planning to do a 2.6.1 release in a few weeks time. We already have a
>>>> bunch
>>>> of tickets committed to 2.7.1. I created a filter [2] to tracking
>>>> pending
>>>> tickets.
>>>>
>>>> We need to collectively come up with a list of critical issues. We can
>>>> use
>>>> the JIRA Target Version field for the same. I see some but not a whole
>>>> lot
>>>> of new work for this release, most of it is likely going to be pulling
>>>> in
>>>> critical patches from 2.7.1/2.8 etc.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> [1] Will Hadoop 2.6.1 be released soon?
>>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>> [2] 2.6.1 pending tickets
>>>> https://issues.apache.org/jira/issues/?filter=12331711
>>>>
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

RE: Planning Hadoop 2.6.1 release

Brahma Reddy
HI vinod

any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1 as stable give..?


Thanks & Regards
 Brahma Reddy Battula

________________________________________
From: Zhihai Xu [[hidden email]]
Sent: Wednesday, May 13, 2015 12:04 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: Re: Planning Hadoop 2.6.1 release

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <[hidden email]>
wrote:

> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
>> Hello Vinod,
>>
>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>> will not be usable when cluster installed in HA mode and try to get
>> filecontext like below..
>>
>>
>> Path path = new
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>> FileSystem fs = path.getFileSystem(new Configuration());
>> path = fs.makeQualified(path);
>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>> Configuration());
>>
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>> ________________________________________
>> From: Chris Nauroth [[hidden email]]
>> Sent: Friday, May 01, 2015 4:32 AM
>> To: [hidden email]; [hidden email];
>> [hidden email]; [hidden email]
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Thank you, Arpit.  In addition, I suggest we include the following:
>>
>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>> pipe is full
>> HADOOP-11604. Prevent ConcurrentModificationException while closing domain
>> sockets during shutdown of DomainSocketWatcher thread.
>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
>> is an I/O error during requestShortCircuitShm
>>
>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>> finally fixes the root cause of this issue.
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
>>
>>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>> requested in [1].
>>>
>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>> should be non static
>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>> synchronization
>>>
>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>>> improve error handling
>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>> block files are present in the same volume.
>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>> processReport.
>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>> violation.
>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>>> in the blocksMap on NameNode restart.
>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>> DataNode to register successfully with only one NameNode.
>>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>>> failure.
>>> HDFS-7831. Fix the starting index and end condition of the loop in
>>> FileDiffList.findEarlierSnapshotBlocks().
>>> HDFS-7885. Datanode should not trust the generation stamp provided by
>>> client.
>>> HDFS-7960. The full block report should prune zombie storages even if
>>> they're not empty.
>>> HDFS-8072. Reserved RBW space is not released if client terminates while
>>> writing block.
>>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>>> finalize upgrade.
>>>
>>>
>>> Arpit
>>>
>>> [1] Will Hadoop 2.6.1 be released soon?
>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>
>>>
>>>
>>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>>> wrote:
>>>
>>>  There were several requests on the user lists [1] for a 2.6.1 release. I
>>>> got many offline comments too.
>>>>
>>>> Planning to do a 2.6.1 release in a few weeks time. We already have a
>>>> bunch
>>>> of tickets committed to 2.7.1. I created a filter [2] to tracking
>>>> pending
>>>> tickets.
>>>>
>>>> We need to collectively come up with a list of critical issues. We can
>>>> use
>>>> the JIRA Target Version field for the same. I see some but not a whole
>>>> lot
>>>> of new work for this release, most of it is likely going to be pulling
>>>> in
>>>> critical patches from 2.7.1/2.8 etc.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> [1] Will Hadoop 2.6.1 be released soon?
>>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>> [2] 2.6.1 pending tickets
>>>> https://issues.apache.org/jira/issues/?filter=12331711
>>>>
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Sean Busbey
Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
[hidden email]> wrote:

> HI vinod
>
> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
> as stable give..?
>
>
> Thanks & Regards
>  Brahma Reddy Battula
>
> ________________________________________
> From: Zhihai Xu [[hidden email]]
> Sent: Wednesday, May 13, 2015 12:04 PM
> To: [hidden email]
> Cc: [hidden email]; [hidden email];
> [hidden email]
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Hi Akira,
>
> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
> bug.
> It will work better with YARN-2992.
>
> thanks
> zhihai
>
>
> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
> [hidden email]>
> wrote:
>
> > Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> > to include the following:
> >
> > * HADOOP-11343. Overflow is not properly handled in calculating final iv
> > for AES CTR
> > * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> > execute any further apps
> > * YARN-2992. ZKRMStateStore crashes due to session expiry
> > * YARN-3013. AMRMClientImpl does not update AMRM token properly
> > * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> > die
> > * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
> to
> > a reducer
> >
> > All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
> >
> > Regards,
> > Akira
> >
> >
> > On 5/4/15 11:15, Brahma Reddy Battula wrote:
> >
> >> Hello Vinod,
> >>
> >> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
> >> will not be usable when cluster installed in HA mode and try to get
> >> filecontext like below..
> >>
> >>
> >> Path path = new
> >>
> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
> >> FileSystem fs = path.getFileSystem(new Configuration());
> >> path = fs.makeQualified(path);
> >> FileContext fc = FileContext.getFileContext(path.toUri(),new
> >> Configuration());
> >>
> >>
> >>
> >> Thanks & Regards
> >> Brahma Reddy Battula
> >> ________________________________________
> >> From: Chris Nauroth [[hidden email]]
> >> Sent: Friday, May 01, 2015 4:32 AM
> >> To: [hidden email]; [hidden email];
> >> [hidden email]; [hidden email]
> >> Subject: Re: Planning Hadoop 2.6.1 release
> >>
> >> Thank you, Arpit.  In addition, I suggest we include the following:
> >>
> >> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
> >> pipe is full
> >> HADOOP-11604. Prevent ConcurrentModificationException while closing
> domain
> >> sockets during shutdown of DomainSocketWatcher thread.
> >> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
> >> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
> there
> >> is an I/O error during requestShortCircuitShm
> >>
> >> HADOOP-11604 and 11648 are not critical by themselves, but they are
> >> pre-requisites to getting a clean cherry-pick of 11802, which we believe
> >> finally fixes the root cause of this issue.
> >>
> >>
> >> --Chris Nauroth
> >>
> >>
> >>
> >>
> >> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
> >>
> >>  HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
> >>> requested in [1].
> >>>
> >>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
> >>> should be non static
> >>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
> >>> synchronization
> >>>
> >>> HDFS-7009. Active NN and standby NN have different live nodes.
> >>> HDFS-7035. Make adding a new data directory to the DataNode an atomic
> and
> >>> improve error handling
> >>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
> >>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
> >>> block files are present in the same volume.
> >>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
> datanodes
> >>> HDFS-7503. Namenode restart after large deletions can cause slow
> >>> processReport.
> >>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
> >>> HDFS-7579. Improve log reporting during block report rpc failure.
> >>> HDFS-7587. Edit log corruption can happen if append fails with a quota
> >>> violation.
> >>> HDFS-7596. NameNode should prune dead storages from storageMap.
> >>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
> blocks
> >>> in the blocksMap on NameNode restart.
> >>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
> >>> DataNode to register successfully with only one NameNode.
> >>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
> >>> failure.
> >>> HDFS-7831. Fix the starting index and end condition of the loop in
> >>> FileDiffList.findEarlierSnapshotBlocks().
> >>> HDFS-7885. Datanode should not trust the generation stamp provided by
> >>> client.
> >>> HDFS-7960. The full block report should prune zombie storages even if
> >>> they're not empty.
> >>> HDFS-8072. Reserved RBW space is not released if client terminates
> while
> >>> writing block.
> >>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
> >>> finalize upgrade.
> >>>
> >>>
> >>> Arpit
> >>>
> >>> [1] Will Hadoop 2.6.1 be released soon?
> >>> http://markmail.org/thread/zlsr6prejyogdyvh
> >>>
> >>>
> >>>
> >>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
> >>> wrote:
> >>>
> >>>  There were several requests on the user lists [1] for a 2.6.1
> release. I
> >>>> got many offline comments too.
> >>>>
> >>>> Planning to do a 2.6.1 release in a few weeks time. We already have a
> >>>> bunch
> >>>> of tickets committed to 2.7.1. I created a filter [2] to tracking
> >>>> pending
> >>>> tickets.
> >>>>
> >>>> We need to collectively come up with a list of critical issues. We can
> >>>> use
> >>>> the JIRA Target Version field for the same. I see some but not a whole
> >>>> lot
> >>>> of new work for this release, most of it is likely going to be pulling
> >>>> in
> >>>> critical patches from 2.7.1/2.8 etc.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Thanks
> >>>> +Vinod
> >>>>
> >>>> [1] Will Hadoop 2.6.1 be released soon?
> >>>> http://markmail.org/thread/zlsr6prejyogdyvh
> >>>> [2] 2.6.1 pending tickets
> >>>> https://issues.apache.org/jira/issues/?filter=12331711
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
>



--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Vinod Vavilapalli
Got pinged on a recent thread on this one.

As I mentioned there, I had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts.

Thanks
+Vinod


> On Jul 1, 2015, at 12:41 PM, Sean Busbey <[hidden email]> wrote:
>
> Any update on a release plan for 2.6.1?
>
> On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
> [hidden email]> wrote:
>
>> HI vinod
>>
>> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
>> as stable give..?
>>
>>
>> Thanks & Regards
>> Brahma Reddy Battula
>>
>> ________________________________________
>> From: Zhihai Xu [[hidden email]]
>> Sent: Wednesday, May 13, 2015 12:04 PM
>> To: [hidden email]
>> Cc: [hidden email]; [hidden email];
>> [hidden email]
>> Subject: Re: Planning Hadoop 2.6.1 release
>>
>> Hi Akira,
>>
>> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
>> bug.
>> It will work better with YARN-2992.
>>
>> thanks
>> zhihai
>>
>>
>> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
>> [hidden email]>
>> wrote:
>>
>>> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
>>> to include the following:
>>>
>>> * HADOOP-11343. Overflow is not properly handled in calculating final iv
>>> for AES CTR
>>> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
>>> execute any further apps
>>> * YARN-2992. ZKRMStateStore crashes due to session expiry
>>> * YARN-3013. AMRMClientImpl does not update AMRM token properly
>>> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
>>> die
>>> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
>> to
>>> a reducer
>>>
>>> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>>>
>>>> Hello Vinod,
>>>>
>>>> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
>>>> will not be usable when cluster installed in HA mode and try to get
>>>> filecontext like below..
>>>>
>>>>
>>>> Path path = new
>>>>
>> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
>>>> FileSystem fs = path.getFileSystem(new Configuration());
>>>> path = fs.makeQualified(path);
>>>> FileContext fc = FileContext.getFileContext(path.toUri(),new
>>>> Configuration());
>>>>
>>>>
>>>>
>>>> Thanks & Regards
>>>> Brahma Reddy Battula
>>>> ________________________________________
>>>> From: Chris Nauroth [[hidden email]]
>>>> Sent: Friday, May 01, 2015 4:32 AM
>>>> To: [hidden email]; [hidden email];
>>>> [hidden email]; [hidden email]
>>>> Subject: Re: Planning Hadoop 2.6.1 release
>>>>
>>>> Thank you, Arpit.  In addition, I suggest we include the following:
>>>>
>>>> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
>>>> pipe is full
>>>> HADOOP-11604. Prevent ConcurrentModificationException while closing
>> domain
>>>> sockets during shutdown of DomainSocketWatcher thread.
>>>> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
>>>> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
>> there
>>>> is an I/O error during requestShortCircuitShm
>>>>
>>>> HADOOP-11604 and 11648 are not critical by themselves, but they are
>>>> pre-requisites to getting a clean cherry-pick of 11802, which we believe
>>>> finally fixes the root cause of this issue.
>>>>
>>>>
>>>> --Chris Nauroth
>>>>
>>>>
>>>>
>>>>
>>>> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
>>>>
>>>> HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>>>>> requested in [1].
>>>>>
>>>>> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>>>>> should be non static
>>>>> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>>>>> synchronization
>>>>>
>>>>> HDFS-7009. Active NN and standby NN have different live nodes.
>>>>> HDFS-7035. Make adding a new data directory to the DataNode an atomic
>> and
>>>>> improve error handling
>>>>> HDFS-7425. NameNode block deletion logging uses incorrect appender.
>>>>> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>>>>> block files are present in the same volume.
>>>>> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
>> datanodes
>>>>> HDFS-7503. Namenode restart after large deletions can cause slow
>>>>> processReport.
>>>>> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>>>>> HDFS-7579. Improve log reporting during block report rpc failure.
>>>>> HDFS-7587. Edit log corruption can happen if append fails with a quota
>>>>> violation.
>>>>> HDFS-7596. NameNode should prune dead storages from storageMap.
>>>>> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
>> blocks
>>>>> in the blocksMap on NameNode restart.
>>>>> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>>>>> DataNode to register successfully with only one NameNode.
>>>>> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>>>>> failure.
>>>>> HDFS-7831. Fix the starting index and end condition of the loop in
>>>>> FileDiffList.findEarlierSnapshotBlocks().
>>>>> HDFS-7885. Datanode should not trust the generation stamp provided by
>>>>> client.
>>>>> HDFS-7960. The full block report should prune zombie storages even if
>>>>> they're not empty.
>>>>> HDFS-8072. Reserved RBW space is not released if client terminates
>> while
>>>>> writing block.
>>>>> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>>>>> finalize upgrade.
>>>>>
>>>>>
>>>>> Arpit
>>>>>
>>>>> [1] Will Hadoop 2.6.1 be released soon?
>>>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>>>
>>>>>
>>>>>
>>>>> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> There were several requests on the user lists [1] for a 2.6.1
>> release. I
>>>>>> got many offline comments too.
>>>>>>
>>>>>> Planning to do a 2.6.1 release in a few weeks time. We already have a
>>>>>> bunch
>>>>>> of tickets committed to 2.7.1. I created a filter [2] to tracking
>>>>>> pending
>>>>>> tickets.
>>>>>>
>>>>>> We need to collectively come up with a list of critical issues. We can
>>>>>> use
>>>>>> the JIRA Target Version field for the same. I see some but not a whole
>>>>>> lot
>>>>>> of new work for this release, most of it is likely going to be pulling
>>>>>> in
>>>>>> critical patches from 2.7.1/2.8 etc.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Thanks
>>>>>> +Vinod
>>>>>>
>>>>>> [1] Will Hadoop 2.6.1 be released soon?
>>>>>> http://markmail.org/thread/zlsr6prejyogdyvh
>>>>>> [2] 2.6.1 pending tickets
>>>>>> https://issues.apache.org/jira/issues/?filter=12331711
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
> --
> Sean

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Vinod Vavilapalli
Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>

Next steps, I’ll do the following
 - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 2.6.1 and add as candidates
 - I haven’t reviewed the current list yet, the seed list is from this email thread. Will review them.
 - I also have a bunch of patches that I’d like to include, will update them right away.

Others, please look at the current list and let me know what else you’d like to include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and then start the release process. @Akira, let’s sync up offline on how to take this forward in terms of the release process.

Thanks
+Vinod


On Jul 15, 2015, at 1:12 PM, Vinod Kumar Vavilapalli <[hidden email]<mailto:[hidden email]>> wrote:

Got pinged on a recent thread on this one.

As I mentioned there, I had many offline discussions re 2.6.1.

The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts.

Thanks
+Vinod


On Jul 1, 2015, at 12:41 PM, Sean Busbey <[hidden email]<mailto:[hidden email]>> wrote:

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
[hidden email]<mailto:[hidden email]>> wrote:

HI vinod

any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
as stable give..?


Thanks & Regards
Brahma Reddy Battula

________________________________________
From: Zhihai Xu [[hidden email]<mailto:[hidden email]>]
Sent: Wednesday, May 13, 2015 12:04 PM
To: [hidden email]<mailto:[hidden email]>
Cc: [hidden email]<mailto:[hidden email]>; [hidden email]<mailto:[hidden email]>;
[hidden email]<mailto:[hidden email]>
Subject: Re: Planning Hadoop 2.6.1 release

Hi Akira,

Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
bug.
It will work better with YARN-2992.

thanks
zhihai


On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
[hidden email]>
wrote:

Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
to include the following:

* HADOOP-11343. Overflow is not properly handled in calculating final iv
for AES CTR
* YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
execute any further apps
* YARN-2992. ZKRMStateStore crashes due to session expiry
* YARN-3013. AMRMClientImpl does not update AMRM token properly
* YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
die
* MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to
a reducer

All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.

Regards,
Akira


On 5/4/15 11:15, Brahma Reddy Battula wrote:

Hello Vinod,

I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
will not be usable when cluster installed in HA mode and try to get
filecontext like below..


Path path = new

Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
FileSystem fs = path.getFileSystem(new Configuration());
path = fs.makeQualified(path);
FileContext fc = FileContext.getFileContext(path.toUri(),new
Configuration());



Thanks & Regards
Brahma Reddy Battula
________________________________________
From: Chris Nauroth [[hidden email]]
Sent: Friday, May 01, 2015 4:32 AM
To: [hidden email]; [hidden email];
[hidden email]; [hidden email]
Subject: Re: Planning Hadoop 2.6.1 release

Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing
domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:

HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
requested in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
should be non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic
and
improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
block files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
datanodes
HDFS-7503. Namenode restart after large deletions can cause slow
processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota
violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
blocks
in the blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
DataNode to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
failure.
HDFS-7831. Fix the starting index and end condition of the loop in
FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by
client.
HDFS-7960. The full block report should prune zombie storages even if
they're not empty.
HDFS-8072. Reserved RBW space is not released if client terminates
while
writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
finalize upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon?
http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
wrote:

There were several requests on the user lists [1] for a 2.6.1
release. I
got many offline comments too.

Planning to do a 2.6.1 release in a few weeks time. We already have a
bunch
of tickets committed to 2.7.1. I created a filter [2] to tracking
pending
tickets.

We need to collectively come up with a list of critical issues. We can
use
the JIRA Target Version field for the same. I see some but not a whole
lot
of new work for this release, most of it is likely going to be pulling
in
critical patches from 2.7.1/2.8 etc.

Thoughts?

Thanks
+Vinod

[1] Will Hadoop 2.6.1 be released soon?
http://markmail.org/thread/zlsr6prejyogdyvh
[2] 2.6.1 pending tickets
https://issues.apache.org/jira/issues/?filter=12331711










--
Sean


Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Akira AJISAKA
Thanks Vinod for creating the list!

 > @Akira, let’s sync up offline on how to take this forward in terms of
the release process.

Okay.

Thanks,
Akira

On 7/16/15 10:24, Vinod Kumar Vavilapalli wrote:

> Alright, I’d like to make progress while the issue is hot.
>
> I created a label to discuss on the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>
> Next steps, I’ll do the following
>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 2.6.1 and add as candidates
>   - I haven’t reviewed the current list yet, the seed list is from this email thread. Will review them.
>   - I also have a bunch of patches that I’d like to include, will update them right away.
>
> Others, please look at the current list and let me know what else you’d like to include.
>
> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and then start the release process. @Akira, let’s sync up offline on how to take this forward in terms of the release process.
>
> Thanks
> +Vinod
>
>
> On Jul 15, 2015, at 1:12 PM, Vinod Kumar Vavilapalli <[hidden email]<mailto:[hidden email]>> wrote:
>
> Got pinged on a recent thread on this one.
>
> As I mentioned there, I had many offline discussions re 2.6.1.
>
> The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts.
>
> Thanks
> +Vinod
>
>
> On Jul 1, 2015, at 12:41 PM, Sean Busbey <[hidden email]<mailto:[hidden email]>> wrote:
>
> Any update on a release plan for 2.6.1?
>
> On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula <
> [hidden email]<mailto:[hidden email]>> wrote:
>
> HI vinod
>
> any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
> as stable give..?
>
>
> Thanks & Regards
> Brahma Reddy Battula
>
> ________________________________________
> From: Zhihai Xu [[hidden email]<mailto:[hidden email]>]
> Sent: Wednesday, May 13, 2015 12:04 PM
> To: [hidden email]<mailto:[hidden email]>
> Cc: [hidden email]<mailto:[hidden email]>; [hidden email]<mailto:[hidden email]>;
> [hidden email]<mailto:[hidden email]>
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Hi Akira,
>
> Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
> bug.
> It will work better with YARN-2992.
>
> thanks
> zhihai
>
>
> On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA <
> [hidden email]>
> wrote:
>
> Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
> to include the following:
>
> * HADOOP-11343. Overflow is not properly handled in calculating final iv
> for AES CTR
> * YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to
> execute any further apps
> * YARN-2992. ZKRMStateStore crashes due to session expiry
> * YARN-3013. AMRMClientImpl does not update AMRM token properly
> * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
> die
> * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
> to
> a reducer
>
> All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
>
> Regards,
> Akira
>
>
> On 5/4/15 11:15, Brahma Reddy Battula wrote:
>
> Hello Vinod,
>
> I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
> will not be usable when cluster installed in HA mode and try to get
> filecontext like below..
>
>
> Path path = new
>
> Path("har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har");
> FileSystem fs = path.getFileSystem(new Configuration());
> path = fs.makeQualified(path);
> FileContext fc = FileContext.getFileContext(path.toUri(),new
> Configuration());
>
>
>
> Thanks & Regards
> Brahma Reddy Battula
> ________________________________________
> From: Chris Nauroth [[hidden email]]
> Sent: Friday, May 01, 2015 4:32 AM
> To: [hidden email]; [hidden email];
> [hidden email]; [hidden email]
> Subject: Re: Planning Hadoop 2.6.1 release
>
> Thank you, Arpit.  In addition, I suggest we include the following:
>
> HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
> pipe is full
> HADOOP-11604. Prevent ConcurrentModificationException while closing
> domain
> sockets during shutdown of DomainSocketWatcher thread.
> HADOOP-11648. Set DomainSocketWatcher thread name explicitly
> HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
> there
> is an I/O error during requestShortCircuitShm
>
> HADOOP-11604 and 11648 are not critical by themselves, but they are
> pre-requisites to getting a clean cherry-pick of 11802, which we believe
> finally fixes the root cause of this issue.
>
>
> --Chris Nauroth
>
>
>
>
> On 4/30/15, 3:55 PM, "Arpit Agarwal" <[hidden email]> wrote:
>
> HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
> requested in [1].
>
> HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
> should be non static
> HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
> synchronization
>
> HDFS-7009. Active NN and standby NN have different live nodes.
> HDFS-7035. Make adding a new data directory to the DataNode an atomic
> and
> improve error handling
> HDFS-7425. NameNode block deletion logging uses incorrect appender.
> HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
> block files are present in the same volume.
> HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
> datanodes
> HDFS-7503. Namenode restart after large deletions can cause slow
> processReport.
> HDFS-7575. Upgrade should generate a unique storage ID for each volume.
> HDFS-7579. Improve log reporting during block report rpc failure.
> HDFS-7587. Edit log corruption can happen if append fails with a quota
> violation.
> HDFS-7596. NameNode should prune dead storages from storageMap.
> HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
> blocks
> in the blocksMap on NameNode restart.
> HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
> DataNode to register successfully with only one NameNode.
> HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
> failure.
> HDFS-7831. Fix the starting index and end condition of the loop in
> FileDiffList.findEarlierSnapshotBlocks().
> HDFS-7885. Datanode should not trust the generation stamp provided by
> client.
> HDFS-7960. The full block report should prune zombie storages even if
> they're not empty.
> HDFS-8072. Reserved RBW space is not released if client terminates
> while
> writing block.
> HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
> finalize upgrade.
>
>
> Arpit
>
> [1] Will Hadoop 2.6.1 be released soon?
> http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
> On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" <[hidden email]>
> wrote:
>
> There were several requests on the user lists [1] for a 2.6.1
> release. I
> got many offline comments too.
>
> Planning to do a 2.6.1 release in a few weeks time. We already have a
> bunch
> of tickets committed to 2.7.1. I created a filter [2] to tracking
> pending
> tickets.
>
> We need to collectively come up with a list of critical issues. We can
> use
> the JIRA Target Version field for the same. I see some but not a whole
> lot
> of new work for this release, most of it is likely going to be pulling
> in
> critical patches from 2.7.1/2.8 etc.
>
> Thoughts?
>
> Thanks
> +Vinod
>
> [1] Will Hadoop 2.6.1 be released soon?
> http://markmail.org/thread/zlsr6prejyogdyvh
> [2] 2.6.1 pending tickets
> https://issues.apache.org/jira/issues/?filter=12331711
>
>
>
>
>
>
>
>
>
>
> --
> Sean
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Vinod Vavilapalli
In reply to this post by Vinod Vavilapalli
 - I also have a bunch of patches that I’d like to include, will update them right away.

I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.

Others, please look at the list and post anything else you’d like to get included for 2.6.1.

Thanks
+Vinod


On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <[hidden email]<mailto:[hidden email]>> wrote:

Alright, I’d like to make progress while the issue is hot.

I created a label to discuss on the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>

Next steps, I’ll do the following
 - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 2.6.1 and add as candidates
 - I haven’t reviewed the current list yet, the seed list is from this email thread. Will review them.
 - I also have a bunch of patches that I’d like to include, will update them right away.

Others, please look at the current list and let me know what else you’d like to include.

I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and then start the release process. @Akira, let’s sync up offline on how to take this forward in terms of the release process.

Thanks
+Vinod


Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Akira AJISAKA
Thanks Vinod for updating the candidate list.
I'd like to include the followings 12 JIRAs:

* YARN-3641
* YARN-3585
* YARN-2910
* HDFS-8431
* HDFS-7830
* HDFS-7763
* HDFS-7742
* HDFS-7235
* HDFS-7225
* MAPREDUCE-6324
* HADOOP-11934
* HADOOP-11491

Thanks,
Akira

On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:

>   - I also have a bunch of patches that I’d like to include, will update them right away.
>
> I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.
>
> Others, please look at the list and post anything else you’d like to get included for 2.6.1.
>
> Thanks
> +Vinod
>
>
> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <[hidden email]<mailto:[hidden email]>> wrote:
>
> Alright, I’d like to make progress while the issue is hot.
>
> I created a label to discuss on the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>
> Next steps, I’ll do the following
>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 2.6.1 and add as candidates
>   - I haven’t reviewed the current list yet, the seed list is from this email thread. Will review them.
>   - I also have a bunch of patches that I’d like to include, will update them right away.
>
> Others, please look at the current list and let me know what else you’d like to include.
>
> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and then start the release process. @Akira, let’s sync up offline on how to take this forward in terms of the release process.
>
> Thanks
> +Vinod
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Joep Rottinghuis
Hi Vinod,

We've gone through the various lists of upstream jiras and wanted to
provide our take on what we'd like to see in 2.6.1 release.
In addition, we have ~58 jiras (some a group of work such as DN maintenance
state) that we already have in production
in a pre-2.6 release. I've gone through and selected those that seem
relevant for 2.6.1.

Jiras already marked with 2.6.1-candidate (we agree should be part of
2.6.1):
HADOOP-11802
HDFS-7489
HDFS-7533
HDFS-7587
HDFS-7596
HDFS-7707
HDFS-7742
HDFS-8046
HDFS-8072
HDFS-8127
MAPREDUCE-6303
MAPREDUCE-6361
YARN-2414
YARN-2905
YARN-3369
YARN-3485
YARN-3585
YARN-3641
HDFS-7443
HDFS-7575


Jiras not yet marked with 2.6.1-candidate that we'd like to see in 2.6.1:
HDFS-7213
HDFS-7446
HDFS-7704
HDFS-7788
HDFS-7884
HDFS-7894
HDFS-7916
HDFS-7929
HDFS-7930
HDFS-7980
HDFS-8245
HDFS-8270
HDFS-8404
HDFS-8480
HDFS-8486
MAPREDUCE-6238
YARN-2856
MAPREDUCE-6300
YARN-2952
YARN-2997
YARN-3094
YARN-3222
YARN-3238
YARN-3464
YARN-3526
YARN-3850

Jiras that we're already running in production pre-2.6 that we'd like in
2.6.1-candidate:

HADOOP-11812 (committed in 2.8)
MAPREDUCE-5649 (committed in 2.8)
YARN-3231 (committed in 2.7)
MAPREDUCE-6166 (committed in 2.7)
HADOOP-11295 (committed in 2.7)
HDFS-7314 (committed in 2.8)
HDFS-7182 (committed in 2.7)
MAPREDUCE-5465 (committed in 2.8)

Lower priority jira that we're already running in production in pre-2.6
that are more a nice to have:
HDFS-7281 (committed in 3.0)
YARN-3176 (not yet committed in OSS)

Note that these lists are a culmination of lots of work by many people in
our team, so credit goes to all of them.
Any possible typo or mistake in copying jira ids are entirely to be blamed
on me.

Thanks,

Joep


On Tue, Jul 21, 2015 at 2:15 AM, Akira AJISAKA <[hidden email]>
wrote:

> Thanks Vinod for updating the candidate list.
> I'd like to include the followings 12 JIRAs:
>
> * YARN-3641
> * YARN-3585
> * YARN-2910
> * HDFS-8431
> * HDFS-7830
> * HDFS-7763
> * HDFS-7742
> * HDFS-7235
> * HDFS-7225
> * MAPREDUCE-6324
> * HADOOP-11934
> * HADOOP-11491
>
> Thanks,
> Akira
>
> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>
>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>
>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>> JIRAs.
>>
>> Others, please look at the list and post anything else you’d like to get
>> included for 2.6.1.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>> [hidden email]<mailto:[hidden email]>> wrote:
>>
>> Alright, I’d like to make progress while the issue is hot.
>>
>> I created a label to discuss on the candidate list of patches:
>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>> <https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>>
>> Next steps, I’ll do the following
>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
>> for 2.6.1 and add as candidates
>>   - I haven’t reviewed the current list yet, the seed list is from this
>> email thread. Will review them.
>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>
>> Others, please look at the current list and let me know what else you’d
>> like to include.
>>
>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
>> and then start the release process. @Akira, let’s sync up offline on how to
>> take this forward in terms of the release process.
>>
>> Thanks
>> +Vinod
>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Vinod Vavilapalli
In reply to this post by Akira AJISAKA
I’ve added them all to the 2.6.1-candidate list. I included everything even though some of them are major tickets. The list is getting large, we will have to cut these down once we get down to the next phase of figuring out what to include and what not to.

Thanks
+Vinod

> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA <[hidden email]> wrote:
>
> Thanks Vinod for updating the candidate list.
> I'd like to include the followings 12 JIRAs:
>
> * YARN-3641
> * YARN-3585
> * YARN-2910
> * HDFS-8431
> * HDFS-7830
> * HDFS-7763
> * HDFS-7742
> * HDFS-7235
> * HDFS-7225
> * MAPREDUCE-6324
> * HADOOP-11934
> * HADOOP-11491
>
> Thanks,
> Akira
>
> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>>  - I also have a bunch of patches that I’d like to include, will update them right away.
>>
>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64 JIRAs.
>>
>> Others, please look at the list and post anything else you’d like to get included for 2.6.1.
>>
>> Thanks
>> +Vinod
>>
>>
>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <[hidden email]<mailto:[hidden email]>> wrote:
>>
>> Alright, I’d like to make progress while the issue is hot.
>>
>> I created a label to discuss on the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>>
>> Next steps, I’ll do the following
>>  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense for 2.6.1 and add as candidates
>>  - I haven’t reviewed the current list yet, the seed list is from this email thread. Will review them.
>>  - I also have a bunch of patches that I’d like to include, will update them right away.
>>
>> Others, please look at the current list and let me know what else you’d like to include.
>>
>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week and then start the release process. @Akira, let’s sync up offline on how to take this forward in terms of the release process.
>>
>> Thanks
>> +Vinod
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Sangjin Lee
Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
applies to 2.6.

Thanks,
Sangjin

On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
[hidden email]> wrote:

> I’ve added them all to the 2.6.1-candidate list. I included everything
> even though some of them are major tickets. The list is getting large, we
> will have to cut these down once we get down to the next phase of figuring
> out what to include and what not to.
>
> Thanks
> +Vinod
>
> > On Jul 21, 2015, at 2:15 AM, Akira AJISAKA <[hidden email]>
> wrote:
> >
> > Thanks Vinod for updating the candidate list.
> > I'd like to include the followings 12 JIRAs:
> >
> > * YARN-3641
> > * YARN-3585
> > * YARN-2910
> > * HDFS-8431
> > * HDFS-7830
> > * HDFS-7763
> > * HDFS-7742
> > * HDFS-7235
> > * HDFS-7225
> > * MAPREDUCE-6324
> > * HADOOP-11934
> > * HADOOP-11491
> >
> > Thanks,
> > Akira
> >
> > On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
> >>  - I also have a bunch of patches that I’d like to include, will update
> them right away.
> >>
> >> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
> JIRAs.
> >>
> >> Others, please look at the list and post anything else you’d like to
> get included for 2.6.1.
> >>
> >> Thanks
> >> +Vinod
> >>
> >>
> >> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
> [hidden email]<mailto:[hidden email]>> wrote:
> >>
> >> Alright, I’d like to make progress while the issue is hot.
> >>
> >> I created a label to discuss on the candidate list of patches:
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
> >>
> >> Next steps, I’ll do the following
> >>  - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
> for 2.6.1 and add as candidates
> >>  - I haven’t reviewed the current list yet, the seed list is from this
> email thread. Will review them.
> >>  - I also have a bunch of patches that I’d like to include, will update
> them right away.
> >>
> >> Others, please look at the current list and let me know what else you’d
> like to include.
> >>
> >> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
> and then start the release process. @Akira, let’s sync up offline on how to
> take this forward in terms of the release process.
> >>
> >> Thanks
> >> +Vinod
> >>
> >>
> >
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Akira AJISAKA
Thanks Joep and your team members for creating the list. I really
appreciate your work. I looked your 'not yet marked with
2.6.1-candidate' list and categorized them.

1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.

* HDFS-7213
* HDFS-7788
* HDFS-7884
* HDFS-7930
* YARN-2856
* YARN-3222
* YARN-3238
* YARN-3464
* YARN-3526
* YARN-3850

Thanks Sangjin for creating patches for branch-2.6.

2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1

* HDFS-7182
* HDFS-7314
* HDFS-7704
* HDFS-7894
* HDFS-7929 and HDFS-8480 (they are related)
* HDFS-7980
* HDFS-8245
* HDFS-8270
* HDFS-8404
* HDFS-8486
* MAPREDUCE-5465
* MAPREDUCE-5649
* MAPREDUCE-6166
* MAPREDUCE-6238
* MAPREDUCE-6300
* YARN-2952
* YARN-2997
* YARN-3094
* YARN-3176 (to be fixed soon, I think)
* YARN-3231
* HADOOP-11295
* HADOOP-11812

3) Not yet marked as 2.6.1-candidate. I'd like to drop

* HDFS-7281 (incompatible change)
* HDFS-7446 (this looks to be an improvement)
* HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)

Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?

I'd like to freeze the candidate list in about 7 days and start
backporting them. Do you have any thoughts?

Regards,
Akira

On 7/25/15 10:32, Sangjin Lee wrote:

> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
> applies to 2.6.
>
> Thanks,
> Sangjin
>
> On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
> [hidden email]> wrote:
>
>> I’ve added them all to the 2.6.1-candidate list. I included everything
>> even though some of them are major tickets. The list is getting large, we
>> will have to cut these down once we get down to the next phase of figuring
>> out what to include and what not to.
>>
>> Thanks
>> +Vinod
>>
>>> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA <[hidden email]>
>> wrote:
>>>
>>> Thanks Vinod for updating the candidate list.
>>> I'd like to include the followings 12 JIRAs:
>>>
>>> * YARN-3641
>>> * YARN-3585
>>> * YARN-2910
>>> * HDFS-8431
>>> * HDFS-7830
>>> * HDFS-7763
>>> * HDFS-7742
>>> * HDFS-7235
>>> * HDFS-7225
>>> * MAPREDUCE-6324
>>> * HADOOP-11934
>>> * HADOOP-11491
>>>
>>> Thanks,
>>> Akira
>>>
>>> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>>>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>>>
>>>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>> JIRAs.
>>>>
>>>> Others, please look at the list and post anything else you’d like to
>> get included for 2.6.1.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>>
>>>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>> [hidden email]<mailto:[hidden email]>> wrote:
>>>>
>>>> Alright, I’d like to make progress while the issue is hot.
>>>>
>>>> I created a label to discuss on the candidate list of patches:
>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate<
>> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>>>>
>>>> Next steps, I’ll do the following
>>>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes sense
>> for 2.6.1 and add as candidates
>>>>   - I haven’t reviewed the current list yet, the seed list is from this
>> email thread. Will review them.
>>>>   - I also have a bunch of patches that I’d like to include, will update
>> them right away.
>>>>
>>>> Others, please look at the current list and let me know what else you’d
>> like to include.
>>>>
>>>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
>> and then start the release process. @Akira, let’s sync up offline on how to
>> take this forward in terms of the release process.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>>
>>>
>>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Sangjin Lee
Thanks Akira.

I'd like to make one small correction. If we're getting HDFS-7704, then we
should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
not included in the list. But if is (and I think it should), then we should
also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
Hope it makes it clear.

Sangjin

On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA <[hidden email]>
wrote:

> Thanks Joep and your team members for creating the list. I really
> appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
> list and categorized them.
>
> 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.
>
> * HDFS-7213
> * HDFS-7788
> * HDFS-7884
> * HDFS-7930
> * YARN-2856
> * YARN-3222
> * YARN-3238
> * YARN-3464
> * YARN-3526
> * YARN-3850
>
> Thanks Sangjin for creating patches for branch-2.6.
>
> 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1
>
> * HDFS-7182
> * HDFS-7314
> * HDFS-7704
> * HDFS-7894
> * HDFS-7929 and HDFS-8480 (they are related)
> * HDFS-7980
> * HDFS-8245
> * HDFS-8270
> * HDFS-8404
> * HDFS-8486
> * MAPREDUCE-5465
> * MAPREDUCE-5649
> * MAPREDUCE-6166
> * MAPREDUCE-6238
> * MAPREDUCE-6300
> * YARN-2952
> * YARN-2997
> * YARN-3094
> * YARN-3176 (to be fixed soon, I think)
> * YARN-3231
> * HADOOP-11295
> * HADOOP-11812
>
> 3) Not yet marked as 2.6.1-candidate. I'd like to drop
>
> * HDFS-7281 (incompatible change)
> * HDFS-7446 (this looks to be an improvement)
> * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)
>
> Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?
>
> I'd like to freeze the candidate list in about 7 days and start
> backporting them. Do you have any thoughts?
>
> Regards,
> Akira
>
>
> On 7/25/15 10:32, Sangjin Lee wrote:
>
>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
>> applies to 2.6.
>>
>> Thanks,
>> Sangjin
>>
>> On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
>> [hidden email]> wrote:
>>
>> I’ve added them all to the 2.6.1-candidate list. I included everything
>>> even though some of them are major tickets. The list is getting large, we
>>> will have to cut these down once we get down to the next phase of
>>> figuring
>>> out what to include and what not to.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA <[hidden email]>
>>>>
>>> wrote:
>>>
>>>>
>>>> Thanks Vinod for updating the candidate list.
>>>> I'd like to include the followings 12 JIRAs:
>>>>
>>>> * YARN-3641
>>>> * YARN-3585
>>>> * YARN-2910
>>>> * HDFS-8431
>>>> * HDFS-7830
>>>> * HDFS-7763
>>>> * HDFS-7742
>>>> * HDFS-7235
>>>> * HDFS-7225
>>>> * MAPREDUCE-6324
>>>> * HADOOP-11934
>>>> * HADOOP-11491
>>>>
>>>> Thanks,
>>>> Akira
>>>>
>>>> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>>>>
>>>>>   - I also have a bunch of patches that I’d like to include, will
>>>>> update
>>>>>
>>>> them right away.
>>>
>>>>
>>>>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>>>>>
>>>> JIRAs.
>>>
>>>>
>>>>> Others, please look at the list and post anything else you’d like to
>>>>>
>>>> get included for 2.6.1.
>>>
>>>>
>>>>> Thanks
>>>>> +Vinod
>>>>>
>>>>>
>>>>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>>>>>
>>>> [hidden email]<mailto:[hidden email]>> wrote:
>>>
>>>>
>>>>> Alright, I’d like to make progress while the issue is hot.
>>>>>
>>>>> I created a label to discuss on the candidate list of patches:
>>>>>
>>>>
>>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>>> <
>>> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate>
>>>
>>>>
>>>>> Next steps, I’ll do the following
>>>>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
>>>>> sense
>>>>>
>>>> for 2.6.1 and add as candidates
>>>
>>>>   - I haven’t reviewed the current list yet, the seed list is from this
>>>>>
>>>> email thread. Will review them.
>>>
>>>>   - I also have a bunch of patches that I’d like to include, will update
>>>>>
>>>> them right away.
>>>
>>>>
>>>>> Others, please look at the current list and let me know what else you’d
>>>>>
>>>> like to include.
>>>
>>>>
>>>>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a week
>>>>>
>>>> and then start the release process. @Akira, let’s sync up offline on
>>> how to
>>> take this forward in terms of the release process.
>>>
>>>>
>>>>> Thanks
>>>>> +Vinod
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Sangjin Lee
Just for the completeness, the following JIRAs have already been committed
to branch-2.6 and thus will be part of 2.6.1:

HADOOP-11307
YARN-2375
HDFS-7425
HDFS-4882
HDFS-7489
HDFS-7503
HDFS-7443
HDFS-3443
HADOOP-11466
HDFS-7733
MAPREDUCE-6237
YARN-3251
HDFS-6153 reverted


On Fri, Jul 31, 2015 at 10:07 AM, Sangjin Lee <[hidden email]> wrote:

> Thanks Akira.
>
> I'd like to make one small correction. If we're getting HDFS-7704, then we
> should also get HDFS-7916. My earlier comment was assuming HDFS-7704 was
> not included in the list. But if is (and I think it should), then we should
> also get HDFS-7916 as it addresses an important issue related to HDFS-7704.
> Hope it makes it clear.
>
> Sangjin
>
> On Fri, Jul 31, 2015 at 10:01 AM, Akira AJISAKA <
> [hidden email]> wrote:
>
>> Thanks Joep and your team members for creating the list. I really
>> appreciate your work. I looked your 'not yet marked with 2.6.1-candidate'
>> list and categorized them.
>>
>> 1) Now marked as 2.6.1-candidate and I agree with you to keep it marked.
>>
>> * HDFS-7213
>> * HDFS-7788
>> * HDFS-7884
>> * HDFS-7930
>> * YARN-2856
>> * YARN-3222
>> * YARN-3238
>> * YARN-3464
>> * YARN-3526
>> * YARN-3850
>>
>> Thanks Sangjin for creating patches for branch-2.6.
>>
>> 2) Not yet marked as 2.6.1-candidate that I'd like to see in 2.6.1
>>
>> * HDFS-7182
>> * HDFS-7314
>> * HDFS-7704
>> * HDFS-7894
>> * HDFS-7929 and HDFS-8480 (they are related)
>> * HDFS-7980
>> * HDFS-8245
>> * HDFS-8270
>> * HDFS-8404
>> * HDFS-8486
>> * MAPREDUCE-5465
>> * MAPREDUCE-5649
>> * MAPREDUCE-6166
>> * MAPREDUCE-6238
>> * MAPREDUCE-6300
>> * YARN-2952
>> * YARN-2997
>> * YARN-3094
>> * YARN-3176 (to be fixed soon, I think)
>> * YARN-3231
>> * HADOOP-11295
>> * HADOOP-11812
>>
>> 3) Not yet marked as 2.6.1-candidate. I'd like to drop
>>
>> * HDFS-7281 (incompatible change)
>> * HDFS-7446 (this looks to be an improvement)
>> * HDFS-7916 (cannot apply to branch-2.6 as Sangjin mentioned)
>>
>> Hi Vinod, could you mark the issues in 2) as 2.6.1-candidate?
>>
>> I'd like to freeze the candidate list in about 7 days and start
>> backporting them. Do you have any thoughts?
>>
>> Regards,
>> Akira
>>
>>
>> On 7/25/15 10:32, Sangjin Lee wrote:
>>
>>> Out of the JIRAs we proposed, please remove HDFS-7916. I don't think it
>>> applies to 2.6.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Wed, Jul 22, 2015 at 4:02 PM, Vinod Kumar Vavilapalli <
>>> [hidden email]> wrote:
>>>
>>> I’ve added them all to the 2.6.1-candidate list. I included everything
>>>> even though some of them are major tickets. The list is getting large,
>>>> we
>>>> will have to cut these down once we get down to the next phase of
>>>> figuring
>>>> out what to include and what not to.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> On Jul 21, 2015, at 2:15 AM, Akira AJISAKA <[hidden email]>
>>>>>
>>>> wrote:
>>>>
>>>>>
>>>>> Thanks Vinod for updating the candidate list.
>>>>> I'd like to include the followings 12 JIRAs:
>>>>>
>>>>> * YARN-3641
>>>>> * YARN-3585
>>>>> * YARN-2910
>>>>> * HDFS-8431
>>>>> * HDFS-7830
>>>>> * HDFS-7763
>>>>> * HDFS-7742
>>>>> * HDFS-7235
>>>>> * HDFS-7225
>>>>> * MAPREDUCE-6324
>>>>> * HADOOP-11934
>>>>> * HADOOP-11491
>>>>>
>>>>> Thanks,
>>>>> Akira
>>>>>
>>>>> On 7/18/15 11:13, Vinod Kumar Vavilapalli wrote:
>>>>>
>>>>>>   - I also have a bunch of patches that I’d like to include, will
>>>>>> update
>>>>>>
>>>>> them right away.
>>>>
>>>>>
>>>>>> I’ve just finished this. The latest 2.6.1-candidate list is up at 64
>>>>>>
>>>>> JIRAs.
>>>>
>>>>>
>>>>>> Others, please look at the list and post anything else you’d like to
>>>>>>
>>>>> get included for 2.6.1.
>>>>
>>>>>
>>>>>> Thanks
>>>>>> +Vinod
>>>>>>
>>>>>>
>>>>>> On Jul 15, 2015, at 6:24 PM, Vinod Kumar Vavilapalli <
>>>>>>
>>>>> [hidden email]<mailto:[hidden email]>> wrote:
>>>>
>>>>>
>>>>>> Alright, I’d like to make progress while the issue is hot.
>>>>>>
>>>>>> I created a label to discuss on the candidate list of patches:
>>>>>>
>>>>>
>>>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidate
>>>> <
>>>> https://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate
>>>> >
>>>>
>>>>>
>>>>>> Next steps, I’ll do the following
>>>>>>   - Review 2.7 and 2.8 blocker/critical tickets and see what makes
>>>>>> sense
>>>>>>
>>>>> for 2.6.1 and add as candidates
>>>>
>>>>>   - I haven’t reviewed the current list yet, the seed list is from this
>>>>>>
>>>>> email thread. Will review them.
>>>>
>>>>>   - I also have a bunch of patches that I’d like to include, will
>>>>>> update
>>>>>>
>>>>> them right away.
>>>>
>>>>>
>>>>>> Others, please look at the current list and let me know what else
>>>>>> you’d
>>>>>>
>>>>> like to include.
>>>>
>>>>>
>>>>>> I’d like to keep this ‘candidate-collection’ cycle’ for a max of a
>>>>>> week
>>>>>>
>>>>> and then start the release process. @Akira, let’s sync up offline on
>>>> how to
>>>> take this forward in terms of the release process.
>>>>
>>>>>
>>>>>> Thanks
>>>>>> +Vinod
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning Hadoop 2.6.1 release

Vinod Vavilapalli
In reply to this post by Joep Rottinghuis
Tx for the list, Joep!

> Jiras not yet marked with 2.6.1-candidate that we'd like to see in 2.6.1:

A whole bunch of them already are marked so, presumably from my list. I’ve added the remaining for discussion.

> HDFS-7281 (committed in 3.0)

This is an incompatible change, so not adding it.


> YARN-3176 (not yet committed in OSS)

We should let it naturally go into 2.6.1 and 2.7.2 depending on when it goes in. We shouldn't block the release for this till it gets reviewed/committed in case the later can’t come in time.

Thanks
+Vinod
123