Wire compatibility between Hadoop 3.x client and 2.x server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Wire compatibility between Hadoop 3.x client and 2.x server

Chao Sun
Hi community,

Does Hadoop 3.x provide wire compatibility between 3.x clients and 2.x
server? There is a blog post from Cloudera [1] mentioning wire
compatibility between 2.x clients and 3.x server, but not the other
direction. Curious if someone knows this. Also it'd be good to know if
someone is running this setup in prod. Thanks!

Chao

[1]:
https://blog.cloudera.com/upgrading-clusters-workloads-hadoop-2-hadoop-3
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Zhenyu Zheng
Hi,

I'm not a Hadoop expert but I've had some experiences in the cloud
infrastructure that I think also makes sense in your case.
For 2.x client + 3.x Server case, the Server side can obviously provide
compatibility, so it should work fine. But in the opposite case,
it might have problem, because 2.x server may not understand new client
calls added in 3.x, for old calls, it should work fine, but
if some internal implementation has changed, it might also be a problem, so
you might have to check case by case.

BR,

On Tue, Oct 13, 2020 at 6:41 AM Chao Sun <[hidden email]> wrote:

> Hi community,
>
> Does Hadoop 3.x provide wire compatibility between 3.x clients and 2.x
> server? There is a blog post from Cloudera [1] mentioning wire
> compatibility between 2.x clients and 3.x server, but not the other
> direction. Curious if someone knows this. Also it'd be good to know if
> someone is running this setup in prod. Thanks!
>
> Chao
>
> [1]:
> https://blog.cloudera.com/upgrading-clusters-workloads-hadoop-2-hadoop-3
>
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Wu,Jianliang(vip.com)
In reply to this post by Chao Sun
In our case, when nn has upgraded to 3.1.3 and dn’s version was still 2.6,  we found hive to call getContentSummary method , the client and server was not compatible  because of hadoop3 added new PROVIDED storage type.

2020年10月13日 06:41,Chao Sun <[hidden email]<mailto:[hidden email]>> 写道:



本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作! This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Wei-Chiu Chuang-3
Thanks Jialiang for reporting the issue.
That sounds bad and should've not happened. Could you file a HDFS jira and
fill in more details?

On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
[hidden email]> wrote:

> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> 2.6,  we found hive to call getContentSummary method , the client and
> server was not compatible  because of hadoop3 added new PROVIDED storage
> type.
>
> 2020年10月13日 06:41,Chao Sun <[hidden email]<mailto:[hidden email]>>
> 写道:
>
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Steven Rand-2
Another backcompat issue was HDFS-15191, in 3.2.1.

But aside from that, we've been using 3x client libraries against both 2x and 3x clusters without issue.

On 10/13/20, 2:43 PM, "Wei-Chiu Chuang" <[hidden email]> wrote:

    CAUTION: This email originates from an external party (outside of Palantir). If you believe this message is suspicious in nature, please use the "Report Phishing" button built into Outlook.


    Thanks Jialiang for reporting the issue.
    That sounds bad and should've not happened. Could you file a HDFS jira and
    fill in more details?

    On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
    [hidden email]> wrote:

    > In our case, when nn has upgraded to 3.1.3 and dn’s version was still
    > 2.6,  we found hive to call getContentSummary method , the client and
    > server was not compatible  because of hadoop3 added new PROVIDED storage
    > type.
    >
    > 2020年10月13日 06:41,Chao Sun <[hidden email]<mailto:[hidden email]>>
    > 写道:
    >
    >
    >
    > 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
    > This communication is intended only for the addressee(s) and may contain
    > information that is privileged and confidential. You are hereby notified
    > that, if you are not an intended recipient listed above, or an authorized
    > employee or agent of an addressee of this communication responsible for
    > delivering e-mail messages to an intended recipient, any dissemination,
    > distribution or reproduction of this communication (including any
    > attachments hereto) is strictly prohibited. If you have received this
    > communication in error, please notify us immediately by a reply e-mail
    > addressed to the sender and permanently delete the original e-mail
    > communication and any attachments from all storage devices without making
    > or otherwise retaining a copy.
    >


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Wu,Jianliang(vip.com)
In reply to this post by Wei-Chiu Chuang-3
Ok, I will file a HDFS jira to report this issue.

> 2020年10月13日 20:43,Wei-Chiu Chuang <[hidden email]> 写道:
>
> Thanks Jialiang for reporting the issue.
> That sounds bad and should've not happened. Could you file a HDFS jira and
> fill in more details?
>
> On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> [hidden email]> wrote:
>
>> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
>> 2.6,  we found hive to call getContentSummary method , the client and
>> server was not compatible  because of hadoop3 added new PROVIDED storage
>> type.
>>
>> 2020年10月13日 06:41,Chao Sun <[hidden email]<mailto:[hidden email]>>
>> 写道:
>>
>>
>>
>> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
>> This communication is intended only for the addressee(s) and may contain
>> information that is privileged and confidential. You are hereby notified
>> that, if you are not an intended recipient listed above, or an authorized
>> employee or agent of an addressee of this communication responsible for
>> delivering e-mail messages to an intended recipient, any dissemination,
>> distribution or reproduction of this communication (including any
>> attachments hereto) is strictly prohibited. If you have received this
>> communication in error, please notify us immediately by a reply e-mail
>> addressed to the sender and permanently delete the original e-mail
>> communication and any attachments from all storage devices without making
>> or otherwise retaining a copy.
>>

本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作! This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Chao Sun
Thanks for the replies all!

> But in the opposite case, it might have problem, because 2.x server may
not understand new client calls added in 3.x

Yes not expecting this to work. I'm thinking about the case where one
upgrades existing 2.x clients to 3.x and expects it to still work against
2.x server, which should not involve those new APIs.

> Another backcompat issue was HDFS-15191, in 3.2.1.

Thanks for pointing this out! In fact this looks like a serious bug in
3.2.1. Glad to see it is fixed in 3.2.2.

> But aside from that, we've been using 3x client libraries against both 2x
and 3x clusters without issue.

Great! thanks.

IMO it will be great if the community maintains official compatibility doc
w.r.t 2.x/3.x, which can help the migration easier.

Chao

On Tue, Oct 13, 2020 at 11:57 PM Wu,Jianliang(vip.com) <
[hidden email]> wrote:

> Ok, I will file a HDFS jira to report this issue.
>
> > 2020年10月13日 20:43,Wei-Chiu Chuang <[hidden email]> 写道:
> >
> > Thanks Jialiang for reporting the issue.
> > That sounds bad and should've not happened. Could you file a HDFS jira
> and
> > fill in more details?
> >
> > On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> > [hidden email]> wrote:
> >
> >> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> >> 2.6,  we found hive to call getContentSummary method , the client and
> >> server was not compatible  because of hadoop3 added new PROVIDED storage
> >> type.
> >>
> >> 2020年10月13日 06:41,Chao Sun <[hidden email]<mailto:
> [hidden email]>>
> >> 写道:
> >>
> >>
> >>
> >>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
> >> This communication is intended only for the addressee(s) and may contain
> >> information that is privileged and confidential. You are hereby notified
> >> that, if you are not an intended recipient listed above, or an
> authorized
> >> employee or agent of an addressee of this communication responsible for
> >> delivering e-mail messages to an intended recipient, any dissemination,
> >> distribution or reproduction of this communication (including any
> >> attachments hereto) is strictly prohibited. If you have received this
> >> communication in error, please notify us immediately by a reply e-mail
> >> addressed to the sender and permanently delete the original e-mail
> >> communication and any attachments from all storage devices without
> making
> >> or otherwise retaining a copy.
> >>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>