Windows and Linux hadoop cluster

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Windows and Linux hadoop cluster

Prachi Sharma

Hi All,

 

Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).

 

If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.

 

Thanks !

Prachi 

Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Alexander Alten-Lorenz-2

Hi,

 

That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html

 

cheers,

 --alex

 

--

b: mapredit.blogspot.com

 

From: [hidden email]
Sent: Wednesday, July 20, 2016 9:31 AM
To: [hidden email]
Subject: Windows and Linux hadoop cluster

 

Hi All,

 

Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).

 

If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.

 

Thanks !

Prachi 

 

Reply | Threaded
Open this post in threaded view
|

Re: Windows and Linux hadoop cluster

Prachi Sharma
Hi Alex,

what potential misbehavior and performance impact do you foresee . can you please give a brief . Any other security threats that could be present in such cluster or any other constraints in such environment . Please let me know I need to check the mixed cluster feasibility for production environment.

Thanks 
Prachi

On Wed, Jul 20, 2016 at 4:42 PM, Alexander Alten-Lorenz <[hidden email]> wrote:

Hi,

 

That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html

 

cheers,

 --alex

 

--

b: mapredit.blogspot.com

 

From: [hidden email]
Sent: Wednesday, July 20, 2016 9:31 AM
To: [hidden email]
Subject: Windows and Linux hadoop cluster

 

Hi All,

 

Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).

 

If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.

 

Thanks !

Prachi 

 


Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Alexander Alten-Lorenz-2

Security:

Windows doesn’t have a working OpenSSL implementation

Malware, Virus and other typical windows based threads

Disk level permission or encryption

 

Performance:

Different thread handling per OS

Yarn implementation differs, can cause negative performance

Windows CPU / Core scaling isn’t the same as Linux

 

I would not go with a mixed environment in production, and I see no sense behind. Stable solutions are often use CentOS, since the TCO is much smaller than in Windows environments. If you’re a Windows shop, go with Azure.

 

Cheers,

--alex

 

 

From: [hidden email]
Sent: Wednesday, July 20, 2016 1:32 PM
To: [hidden email]
Cc: [hidden email]
Subject: Re: Windows and Linux hadoop cluster

 

Hi Alex,

 

what potential misbehavior and performance impact do you foresee . can you please give a brief . Any other security threats that could be present in such cluster or any other constraints in such environment . Please let me know I need to check the mixed cluster feasibility for production environment.

 

Thanks 

Prachi

 

On Wed, Jul 20, 2016 at 4:42 PM, Alexander Alten-Lorenz <[hidden email]> wrote:

Hi,

 

That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html

 

cheers,

 --alex

 

--

b: mapredit.blogspot.com

 

From: [hidden email]
Sent: Wednesday, July 20, 2016 9:31 AM
To: [hidden email]
Subject: Windows and Linux hadoop cluster

 

Hi All,

 

Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).

 

If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.

 

Thanks !

Prachi 

 

 

 

Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Ashish Kumar9
In reply to this post by Alexander Alten-Lorenz-2
I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        Alexander Alten-Lorenz <[hidden email]>
To:        Prachi Sharma <[hidden email]>, "[hidden email]" <[hidden email]>
Date:        07/20/2016 04:42 PM
Subject:        RE: Windows and Linux hadoop cluster




Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b: mapredit.blogspot.com
 
From: Prachi Sharma
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
user@...
Subject:
Windows and Linux hadoop cluster

 
Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi
 

Reply | Threaded
Open this post in threaded view
|

Re: Windows and Linux hadoop cluster

Santosh Pingale
For "trial" it should just work but for production maintaining multiple environment will be painful. This holds true not only for Hadoop but for languages such as python(in case you use it as a programming language). Not all Additional Hadoop components can run natively on windows.

On 20-Jul-2016, at 8:13 PM, Ashish Kumar9 <[hidden email]> wrote:

I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        Alexander Alten-Lorenz <[hidden email]>
To:        Prachi Sharma <[hidden email]>, "[hidden email]" <[hidden email]>
Date:        07/20/2016 04:42 PM
Subject:        RE: Windows and Linux hadoop cluster




Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b: mapredit.blogspot.com
 
From: [hidden email]
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
[hidden email]
Subject:
Windows and Linux hadoop cluster

 
Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi
 

Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

sreebalineni .
In reply to this post by Ashish Kumar9

Was it in production and good amount of workload? That's interesting. Which distribution was used


On Jul 20, 2016 8:13 PM, "Ashish Kumar9" <[hidden email]> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        Alexander Alten-Lorenz <[hidden email]>
To:        Prachi Sharma <[hidden email]>, "[hidden email]" <[hidden email]>
Date:        07/20/2016 04:42 PM
Subject:        RE: Windows and Linux hadoop cluster




Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b: mapredit.blogspot.com
 
From: [hidden email]
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
[hidden email]
Subject:
Windows and Linux hadoop cluster

 
Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi
 

Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Ashish Kumar9
I had used community edition . It was PoC and not production . I had hive use-case .



From:        "sreebalineni ." <[hidden email]>
To:        Ashish Kumar9/India/IBM@IBMIN
Cc:        Prachi Sharma <[hidden email]>, [hidden email], Alexander Alten-Lorenz <[hidden email]>
Date:        07/20/2016 10:24 PM
Subject:        RE: Windows and Linux hadoop cluster




Was it in production and good amount of workload? That's interesting. Which distribution was used


On Jul 20, 2016 8:13 PM, "Ashish Kumar9" <ashishk4@...> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        
Alexander Alten-Lorenz <wget.null@...>
To:        
Prachi Sharma <prachi.sharma0202.ps@...>, "user@..." <user@...>
Date:        
07/20/2016 04:42 PM
Subject:        
RE: Windows and Linux hadoop cluster





Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b:
mapredit.blogspot.com
 

From:
Prachi Sharma
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
user@...
Subject:
Windows and Linux hadoop cluster

 

Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(
hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi
 



Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Prachi Sharma
In reply to this post by Ashish Kumar9

Hi Ashish,

Did you also try the heterogeneous cluster with windows and Linux nodes ?

Thanks!
Prachi

On 20-Jul-2016 8:13 PM, "Ashish Kumar9" <[hidden email]> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        Alexander Alten-Lorenz <[hidden email]>
To:        Prachi Sharma <[hidden email]>, "[hidden email]" <[hidden email]>
Date:        07/20/2016 04:42 PM
Subject:        RE: Windows and Linux hadoop cluster




Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b: mapredit.blogspot.com
 
From: [hidden email]
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
[hidden email]
Subject:
Windows and Linux hadoop cluster

 
Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi
 

Reply | Threaded
Open this post in threaded view
|

RE: Windows and Linux hadoop cluster

Ashish Kumar9
Only linux distros.



From:        Prachi Sharma <[hidden email]>
To:        Ashish Kumar9/India/IBM@IBMIN
Cc:        [hidden email], Alexander Alten-Lorenz <[hidden email]>
Date:        07/21/2016 08:45 AM
Subject:        RE: Windows and Linux hadoop cluster




Hi Ashish,

Did you also try the heterogeneous cluster with windows and Linux nodes ?

Thanks!
Prachi

On 20-Jul-2016 8:13 PM, "Ashish Kumar9" <ashishk4@...> wrote:
I have tried heterogeneous hadoop 2.6 cluster across multiple linux distros and h/w architecture ( x86_64,ppc64le,aarc64) and it worked . I did not see any technical challenge .



From:        
Alexander Alten-Lorenz <wget.null@...>
To:        
Prachi Sharma <prachi.sharma0202.ps@...>, "user@..." <user@...>
Date:        
07/20/2016 04:42 PM
Subject:        
RE: Windows and Linux hadoop cluster





Hi,
 
That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html
 
cheers,
 --alex
 
--
b:
mapredit.blogspot.com
 

From:
Prachi Sharma
Sent:
Wednesday, July 20, 2016 9:31 AM
To:
user@...
Subject:
Windows and Linux hadoop cluster

 

Hi All,
 
Please let me know if it’s feasible to have hadoop cluster with data nodes running on multiple Operating systems. For instance few data nodes running on windows server and others on linux based OS (RHEL,centOS).
 
If above scenario is feasible then please provide configuration settings required in various xml files(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml) and environment files(
hadoop-env.sh/hadoop-cmd.sh) for windows and linux data nodes and namenode.
 
Thanks !
Prachi