Not able to run more than one map task

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Not able to run more than one map task

Amit Kumar
Hi All,

We recently started working on Hadoop. We have setup the hadoop in pseduo distribution mode along with oozie. 

Every developer has set it up on his laptop. The problem is that we are not able to run more than one map task concurrently on our laptops. Resource manager is not allowing more than one task on our machine.  

My task gets completed if I submit it without Oozie. Oozie requires one map task for its own functioning. Actual task that oozie submit does not start.

Here is my configuration

-- Hadoop setup in Pseudo distribution mode 
-- Hadoop Version - 2.6
-- Oozie Version - 4.0.1

Regards,
Amit
Reply | Threaded
Open this post in threaded view
|

Re: Not able to run more than one map task

Harsh J-2
You are likely memory/vcore starved in the NM's configs. Increase your
 yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
MR job memory request values to gain more parallelism.

On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <[hidden email]> wrote:

> Hi All,
>
> We recently started working on Hadoop. We have setup the hadoop in pseduo
> distribution mode along with oozie.
>
> Every developer has set it up on his laptop. The problem is that we are not
> able to run more than one map task concurrently on our laptops. Resource
> manager is not allowing more than one task on our machine.
>
> My task gets completed if I submit it without Oozie. Oozie requires one map
> task for its own functioning. Actual task that oozie submit does not start.
>
> Here is my configuration
>
> -- Hadoop setup in Pseudo distribution mode
> -- Hadoop Version - 2.6
> -- Oozie Version - 4.0.1
>
> Regards,
> Amit



--
Harsh J
Reply | Threaded
Open this post in threaded view
|

RE: Not able to run more than one map task

Amit Kumar
Thanks a lot Harsha for replying

This problem has waster at least last one week.

We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?

System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1

MR2:

mapreduce.map.memory.mb : 512
mapreduce.tasktracker.map.tasks.maximum : 4

Yarn:

yarn.app.mapreduce.am.resource.mb : 512
yarn.nodemanager.resource.cpu-vcores : 4
yarn.scheduler.minimum-allocation-mb : 512
yarn.nodemanager.resource.memory-mb : 5080


Regards,
Amit



> From: [hidden email]

> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: [hidden email]
>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <[hidden email]> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: Not able to run more than one map task

Niels Basjes

Just curious: what is the input for your job ? If it is a single gzipped file then that is the cause of getting exactly 1 mapper.

Niels


On Fri, Apr 10, 2015, 09:21 Amit Kumar <[hidden email]> wrote:
Thanks a lot Harsha for replying

This problem has waster at least last one week.

We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?

System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1

MR2:

mapreduce.map.memory.mb : 512
mapreduce.tasktracker.map.tasks.maximum : 4

Yarn:

yarn.app.mapreduce.am.resource.mb : 512
yarn.nodemanager.resource.cpu-vcores : 4
yarn.scheduler.minimum-allocation-mb : 512
yarn.nodemanager.resource.memory-mb : 5080


Regards,
Amit



> From: [hidden email]

> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: [hidden email]

>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <[hidden email]> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
Reply | Threaded
Open this post in threaded view
|

Re: Not able to run more than one map task

Drake민영근
Hi, Amit.

Test these: 
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for testing. And decrease "mapreduce.map.memory.mb" to 256 and add "yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.

Thanks.

Drake 민영근 Ph.D
kt NexR

On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <[hidden email]> wrote:

Just curious: what is the input for your job ? If it is a single gzipped file then that is the cause of getting exactly 1 mapper.

Niels


On Fri, Apr 10, 2015, 09:21 Amit Kumar <[hidden email]> wrote:
Thanks a lot Harsha for replying

This problem has waster at least last one week.

We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?

System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1

MR2:

mapreduce.map.memory.mb : 512
mapreduce.tasktracker.map.tasks.maximum : 4

Yarn:

yarn.app.mapreduce.am.resource.mb : 512
yarn.nodemanager.resource.cpu-vcores : 4
yarn.scheduler.minimum-allocation-mb : 512
yarn.nodemanager.resource.memory-mb : 5080


Regards,
Amit



> From: [hidden email]

> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: [hidden email]

>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <[hidden email]> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J