Configuring different mapred.tasktracker.tasks.maximum per node

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuring different mapred.tasktracker.tasks.maximum per node

Venkat Seeth
Hello there,

Howdy. I've a quick question on configuring
mapred.tasktracker.tasks.maximum for a node in a
hadoop cluster.

I have N nodes with 4-Dual-proc CPUs, 64GB Ram and M
nodes with 2-Dual-proc CPUs, 8 GB Ram.

I'd like to configure mapred.tasktracker.tasks.maximum
= 10 for N nodes and mapred.tasktracker.tasks.maximum
= 4 for M set of nodes.

Can I do that and how. If I change hadoop-site.xml on
each of the nodes, will it work?

Your input is greatly appreciated.

Thanks,
Venkat



 
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/
Reply | Threaded
Open this post in threaded view
|

Re: Configuring different mapred.tasktracker.tasks.maximum per node

Andrzej Białecki-2
Venkat Seeth wrote:

> Hello there,
>
> Howdy. I've a quick question on configuring
> mapred.tasktracker.tasks.maximum for a node in a
> hadoop cluster.
>
> I have N nodes with 4-Dual-proc CPUs, 64GB Ram and M
> nodes with 2-Dual-proc CPUs, 8 GB Ram.
>
> I'd like to configure mapred.tasktracker.tasks.maximum
> = 10 for N nodes and mapred.tasktracker.tasks.maximum
> = 4 for M set of nodes.
>
> Can I do that and how. If I change hadoop-site.xml on
> each of the nodes, will it work?
>  

Yes. Each tasktracker gets this number from its local config in
hadoop-site.xml, and then reports to the jobtracker, who in turn
calculates the total capacity of the cluster.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: Configuring different mapred.tasktracker.tasks.maximum per node

Venkat Seeth
Thanks Andrzej for your quick response.

How do we manage hadoop-site.xml in CVS if we have
different versions of the file for different nodes.

Can we pass this value at runtime using
-Dmapred.tasktracker.tasks.maximum=N?

Or will it work is set as an ENV variable on the host?

Any ideas are greatly appreciated.

Venkat

--- Andrzej Bialecki <[hidden email]> wrote:

> Venkat Seeth wrote:
> > Hello there,
> >
> > Howdy. I've a quick question on configuring
> > mapred.tasktracker.tasks.maximum for a node in a
> > hadoop cluster.
> >
> > I have N nodes with 4-Dual-proc CPUs, 64GB Ram and
> M
> > nodes with 2-Dual-proc CPUs, 8 GB Ram.
> >
> > I'd like to configure
> mapred.tasktracker.tasks.maximum
> > = 10 for N nodes and
> mapred.tasktracker.tasks.maximum
> > = 4 for M set of nodes.
> >
> > Can I do that and how. If I change hadoop-site.xml
> on
> > each of the nodes, will it work?
> >  
>
> Yes. Each tasktracker gets this number from its
> local config in
> hadoop-site.xml, and then reports to the jobtracker,
> who in turn
> calculates the total capacity of the cluster.
>
> --
> Best regards,
> Andrzej Bialecki     <><
>  ___. ___ ___ ___ _ _  
> __________________________________
> [__ || __|__/|__||\/|  Information Retrieval,
> Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System
> Integration
> http://www.sigram.com  Contact: info at sigram dot
> com
>
>
>



 
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/