AM Container exits with code 2

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

AM Container exits with code 2

Rahul Chhiber

Hi all,

 

I have launched an application on yarn cluster which has following config.

Master (Resource Manager) - 16GB RAM + 8 vCPU

Slave 1 (Node manager 1) - 8GB RAM + 4 vCPU

 

Intermittently AM(2GB, 1 core) is exiting with code - 2 with the following trace. I am not able to find anything about exit code 2.

 

Last log is

org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 22504 for container-id container_1469709900068_0002_01_000001: 203.8 MB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used

 

Does this have anything to do with my application logic or Is it possible that it is killed because of exceeding the memory limits?

 

2016-07-28 17:08:50,672 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1469709900068_0002_01_000001 and exit code: 2

ExitCodeException exitCode=2:

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

        at org.apache.hadoop.util.Shell.run(Shell.java:455)

        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)

        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1469709900068_0002_01_000001

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 2

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=2:

 

Thanks,

Rahul Chhiber

 

Reply | Threaded
Open this post in threaded view
|

Re: AM Container exits with code 2

Sunil Govind
Hi Rahul,
From the given log, I do not think YARN is killing containers due to memory issue. Usage is under the limits. However full log is not shared, so you can verify that when the AM launch is failed whether memory was under limit or not.
Which application are you trying to run? 
Also its better if we have "application master container" log.  sysout or syserr of that launch will have some more information. 

Thanks 
Sunil

On Fri, Jul 29, 2016 at 12:49 PM Rahul Chhiber <[hidden email]> wrote:

Hi all,

 

I have launched an application on yarn cluster which has following config.

Master (Resource Manager) - 16GB RAM + 8 vCPU

Slave 1 (Node manager 1) - 8GB RAM + 4 vCPU

 

Intermittently AM(2GB, 1 core) is exiting with code - 2 with the following trace. I am not able to find anything about exit code 2.

 

Last log is

org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 22504 for container-id container_1469709900068_0002_01_000001: 203.8 MB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used

 

Does this have anything to do with my application logic or Is it possible that it is killed because of exceeding the memory limits?

 

2016-07-28 17:08:50,672 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1469709900068_0002_01_000001 and exit code: 2

ExitCodeException exitCode=2:

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

        at org.apache.hadoop.util.Shell.run(Shell.java:455)

        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)

        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1469709900068_0002_01_000001

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 2

2016-07-28 17:08:50,674 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=2:

 

Thanks,

Rahul Chhiber