Redefine yarn.application.classpath value

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Redefine yarn.application.classpath value

José Luis Larroque
Hi guys!

i have a Ubuntu 14.04 LTS, with a hadoop 2.4.0 single node cluster. When i execute yarn classpath on console, gives me this:
/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*

As you can see, the first value is repeated three times. I looked for a way to redefine or reset this. The only way, as far as i know, it's redefining a property in yarn-site.xml, like this:

 <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
        <value>
        $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADO$
     </value>
  </property>

But yarn.application.classpath remains the same, without changes, even if i stop every process of hadoop and start them all again. What other option do i have? i'm doing something wrong here?

Reply | Threaded
Open this post in threaded view
|

Re: Redefine yarn.application.classpath value

José Luis Larroque
In SO someone recommend me this:

"check the place where you are setting $HADOOP_CONF_DIR, you might be setting that multiple places.. The $HADOOP_CONF_DIR is added by the bin/hadoop script to the front of the path. "

i'm setting it on hadoop-env.sh:
 export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/local/hadoop/etc/hadoop"}


But it isn't working, the classpath remains the same. I tried removing it, but was the same anyway.

Please!!! any help would be greatly appreciated.

Bye!
Jose

2016-01-30 13:42 GMT-03:00 José Luis Larroque <[hidden email]>:
Hi guys!

i have a Ubuntu 14.04 LTS, with a hadoop 2.4.0 single node cluster. When i execute yarn classpath on console, gives me this:
/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*

As you can see, the first value is repeated three times. I looked for a way to redefine or reset this. The only way, as far as i know, it's redefining a property in yarn-site.xml, like this:

 <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
        <value>
        $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADO$
     </value>
  </property>

But yarn.application.classpath remains the same, without changes, even if i stop every process of hadoop and start them all again. What other option do i have? i'm doing something wrong here?


Reply | Threaded
Open this post in threaded view
|

Re: Redefine yarn.application.classpath value

Dejan Menges
Hello Jose,

For Yarn's classpath, depending how you installed everything on Ubuntu, take a look also into yarn-env.sh, and inside /etc/default/${whateveryarnorhadoopfile}.

However, I would personally expect it to be in yarn-env.sh. 

Cheers

On Sun, Feb 7, 2016 at 2:02 AM José Luis Larroque <[hidden email]> wrote:
In SO someone recommend me this:

"check the place where you are setting $HADOOP_CONF_DIR, you might be setting that multiple places.. The $HADOOP_CONF_DIR is added by the bin/hadoop script to the front of the path. "

i'm setting it on hadoop-env.sh:
 export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/local/hadoop/etc/hadoop"}


But it isn't working, the classpath remains the same. I tried removing it, but was the same anyway.

Please!!! any help would be greatly appreciated.

Bye!
Jose

2016-01-30 13:42 GMT-03:00 José Luis Larroque <[hidden email]>:
Hi guys!

i have a Ubuntu 14.04 LTS, with a hadoop 2.4.0 single node cluster. When i execute yarn classpath on console, gives me this:
/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*

As you can see, the first value is repeated three times. I looked for a way to redefine or reset this. The only way, as far as i know, it's redefining a property in yarn-site.xml, like this:

 <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
        <value>
        $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADO$
     </value>
  </property>

But yarn.application.classpath remains the same, without changes, even if i stop every process of hadoop and start them all again. What other option do i have? i'm doing something wrong here?


Reply | Threaded
Open this post in threaded view
|

Re: Redefine yarn.application.classpath value

José Luis Larroque
Hi Dejan, thanks for your answer and sorry for taking too long to response.

I have this line in yarn-env.sh:
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"

I changed to
export YARN_CONF_DIR=${HADOOP_CONF_DIR}

But it didn't change anything.

I also take a look at $HADOOP_HOME/etc/hadoop searching for HADOOP_CONF_DIR, as you already suggested Dejan, and i founded this on hadoop-env.sh:
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/local/hadoop/etc/hadoop"}

Those are the only ocurrences that i could find.

I'm starting to believe that i'm making some huge mistake, because i keep modifyng configuration files and the command yarn classpath is always showing me the same output.

Please, any help would be really appreciated !!

Bye!
Jose

2016-02-07 6:21 GMT-03:00 Dejan Menges <[hidden email]>:
Hello Jose,

For Yarn's classpath, depending how you installed everything on Ubuntu, take a look also into yarn-env.sh, and inside /etc/default/${whateveryarnorhadoopfile}.

However, I would personally expect it to be in yarn-env.sh. 

Cheers

On Sun, Feb 7, 2016 at 2:02 AM José Luis Larroque <[hidden email]> wrote:
In SO someone recommend me this:

"check the place where you are setting $HADOOP_CONF_DIR, you might be setting that multiple places.. The $HADOOP_CONF_DIR is added by the bin/hadoop script to the front of the path. "

i'm setting it on hadoop-env.sh:
 export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/local/hadoop/etc/hadoop"}


But it isn't working, the classpath remains the same. I tried removing it, but was the same anyway.

Please!!! any help would be greatly appreciated.

Bye!
Jose

2016-01-30 13:42 GMT-03:00 José Luis Larroque <[hidden email]>:
Hi guys!

i have a Ubuntu 14.04 LTS, with a hadoop 2.4.0 single node cluster. When i execute yarn classpath on console, gives me this:
/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*

As you can see, the first value is repeated three times. I looked for a way to redefine or reset this. The only way, as far as i know, it's redefining a property in yarn-site.xml, like this:

 <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
        <value>
        $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADO$
     </value>
  </property>

But yarn.application.classpath remains the same, without changes, even if i stop every process of hadoop and start them all again. What other option do i have? i'm doing something wrong here?