Classpath setting problem for Map tasks

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Classpath setting problem for Map tasks

alakshman
How do I specify the classpath via the JobConf ? In my jobtracker web UI I
see a whole slew of ClassNotFoundException's for the WordCount$MapClass. The
stack trace is pasted below. But how do I fix this - just pass the -cp
<classpath> to the mapred.child.java.opts property of the JobConf ?
exporting CLASSPATH doesn't seem to work.hh

Thanks
Avinash

java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
        at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
Caused by: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
        ... 6 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:242)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
        ... 7 more

java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
        at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
Caused by: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
        ... 6 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.WordCount$MapClass
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:242)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
        ... 7 more
Reply | Threaded
Open this post in threaded view
|

RE: Classpath setting problem for Map tasks

Mahadev Konar
Hi Avinash,
 Can you explain how you are running the wordcount example? Also, which
version of Hadoop are you using? I would really recommend using 0.12.3.

Regards
Mahadev

> -----Original Message-----
> From: Phantom [mailto:[hidden email]]
> Sent: Tuesday, May 29, 2007 1:10 PM
> To: [hidden email]
> Subject: Classpath setting problem for Map tasks
>
> How do I specify the classpath via the JobConf ? In my jobtracker web UI I
> see a whole slew of ClassNotFoundException's for the WordCount$MapClass.
> The
> stack trace is pasted below. But how do I fix this - just pass the -cp
> <classpath> to the mapred.child.java.opts property of the JobConf ?
> exporting CLASSPATH doesn't seem to work.hh
>
> Thanks
> Avinash
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> Caused by: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:242)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
> )
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
> ... 7 more
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> Caused by: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.WordCount$MapClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:242)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
> )
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
> ... 7 more

Reply | Threaded
Open this post in threaded view
|

Re: Classpath setting problem for Map tasks

Avinash Lakshman-2
Hi Mahadev

This is how I am running it. I created a test data file and put it in HDFS
using ./hadoop fs ­put command. Then I modified my WordCount sample to add
two lines to the main which look like this :

conf.set("fs.default.name", "hdfs://dev030.sctm.facebook.com:9000");
conf.set("mapred.job.tracker", "dev030.sctm.facebook.com:50029");

Then I run the sample by executing the command :

java org.apache.hadoop.examples.WordCount -m 10 -r 4
/user/alakshman/test.dat ~/output-dir

where the ~/output-dir is the local file system while the
/user/alakshman/test.dat is the path in HDFS.

This is when I get a whole slew of classnotfoundexceptions. Is the process
incorrect ?

Thanks
Avinash



On 5/29/07 1:17 PM, "Mahadev Konar" <[hidden email]> wrote:

> Hi Avinash,
>  Can you explain how you are running the wordcount example? Also, which
> version of Hadoop are you using? I would really recommend using 0.12.3.
>
> Regards
> Mahadev
>
>> > -----Original Message-----
>> > From: Phantom [mailto:[hidden email]]
>> > Sent: Tuesday, May 29, 2007 1:10 PM
>> > To: [hidden email]
>> > Subject: Classpath setting problem for Map tasks
>> >
>> > How do I specify the classpath via the JobConf ? In my jobtracker web UI I
>> > see a whole slew of ClassNotFoundException's for the WordCount$MapClass.
>> > The
>> > stack trace is pasted below. But how do I fix this - just pass the -cp
>> > <classpath> to the mapred.child.java.opts property of the JobConf ?
>> > exporting CLASSPATH doesn't seem to work.hh
>> >
>> > Thanks
>> > Avinash
>> >
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>> >       at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>> >       at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> > )
>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>> >       at
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>> > Caused by: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>> >       ... 6 more
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >       at java.lang.Class.forName0(Native Method)
>> >       at java.lang.Class.forName(Class.java:242)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>> > )
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>> >       ... 7 more
>> >
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>> >       at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>> >       at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> > )
>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>> >       at
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>> > Caused by: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>> >       ... 6 more
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >       at java.lang.Class.forName0(Native Method)
>> >       at java.lang.Class.forName(Class.java:242)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>> > )
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>> >       ... 7 more
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Classpath setting problem for Map tasks

Avinash Lakshman-2
In reply to this post by Mahadev Konar
Yes I am using hadoop-0.12.3.

Thanks
A


On 5/29/07 1:17 PM, "Mahadev Konar" <[hidden email]> wrote:

> Hi Avinash,
>  Can you explain how you are running the wordcount example? Also, which
> version of Hadoop are you using? I would really recommend using 0.12.3.
>
> Regards
> Mahadev
>
>> > -----Original Message-----
>> > From: Phantom [mailto:[hidden email]]
>> > Sent: Tuesday, May 29, 2007 1:10 PM
>> > To: [hidden email]
>> > Subject: Classpath setting problem for Map tasks
>> >
>> > How do I specify the classpath via the JobConf ? In my jobtracker web UI I
>> > see a whole slew of ClassNotFoundException's for the WordCount$MapClass.
>> > The
>> > stack trace is pasted below. But how do I fix this - just pass the -cp
>> > <classpath> to the mapred.child.java.opts property of the JobConf ?
>> > exporting CLASSPATH doesn't seem to work.hh
>> >
>> > Thanks
>> > Avinash
>> >
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>> >       at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>> >       at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> > )
>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>> >       at
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>> > Caused by: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>> >       ... 6 more
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >       at java.lang.Class.forName0(Native Method)
>> >       at java.lang.Class.forName(Class.java:242)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>> > )
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>> >       ... 7 more
>> >
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>> >       at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>> >       at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> >       at
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> > )
>> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>> >       at
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>> > Caused by: java.lang.RuntimeException:
>> > java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>> >       ... 6 more
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.examples.WordCount$MapClass
>> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> >       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> >       at java.lang.Class.forName0(Native Method)
>> >       at java.lang.Class.forName(Class.java:242)
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>> > )
>> >       at
>> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>> >       ... 7 more
>
>


Reply | Threaded
Open this post in threaded view
|

RE: Classpath setting problem for Map tasks

Mahadev Konar
In reply to this post by Avinash Lakshman-2
Hi Avinash,
  Java org.apache.... will not work. This does not copy the job.jar and
that's why you run into these Classnotfoundexcpetions.
The way to run the wordocunt example is:
Bin/hadoop jar hadoop-examples.jar wordcount .......

This way the jobclient is called by the hadoop command to transfer the
job.jar and get the status of the job.

Regards
Mahadev

> -----Original Message-----
> From: Avinash Lakshman [mailto:[hidden email]]
> Sent: Tuesday, May 29, 2007 1:30 PM
> To: [hidden email]
> Subject: Re: Classpath setting problem for Map tasks
>
> Hi Mahadev
>
> This is how I am running it. I created a test data file and put it in HDFS
> using ./hadoop fs -put command. Then I modified my WordCount sample to add
> two lines to the main which look like this :
>
> conf.set("fs.default.name", "hdfs://dev030.sctm.facebook.com:9000");
> conf.set("mapred.job.tracker", "dev030.sctm.facebook.com:50029");
>
> Then I run the sample by executing the command :
>
> java org.apache.hadoop.examples.WordCount -m 10 -r 4
> /user/alakshman/test.dat ~/output-dir
>
> where the ~/output-dir is the local file system while the
> /user/alakshman/test.dat is the path in HDFS.
>
> This is when I get a whole slew of classnotfoundexceptions. Is the process
> incorrect ?
>
> Thanks
> Avinash
>
>
>
> On 5/29/07 1:17 PM, "Mahadev Konar" <[hidden email]> wrote:
>
> > Hi Avinash,
> >  Can you explain how you are running the wordcount example? Also, which
> > version of Hadoop are you using? I would really recommend using 0.12.3.
> >
> > Regards
> > Mahadev
> >
> >> > -----Original Message-----
> >> > From: Phantom [mailto:[hidden email]]
> >> > Sent: Tuesday, May 29, 2007 1:10 PM
> >> > To: [hidden email]
> >> > Subject: Classpath setting problem for Map tasks
> >> >
> >> > How do I specify the classpath via the JobConf ? In my jobtracker web
> UI I
> >> > see a whole slew of ClassNotFoundException's for the
> WordCount$MapClass.
> >> > The
> >> > stack trace is pasted below. But how do I fix this - just pass the -
> cp
> >> > <classpath> to the mapred.child.java.opts property of the JobConf ?
> >> > exporting CLASSPATH doesn't seem to work.hh
> >> >
> >> > Thanks
> >> > Avinash
> >> >
> >> > java.lang.RuntimeException: java.lang.RuntimeException:
> >> > java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> >> >       at
> org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
> >> >       at
> org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> >> >       at
> >> >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> >> >       at
> >> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> >> > )
> >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
> >> >       at
> >> >
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> >> > Caused by: java.lang.RuntimeException:
> >> > java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
> >> >       ... 6 more
> >> > Caused by: java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >> >       at java.security.AccessController.doPrivileged(Native Method)
> >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >> >       at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >> >       at
> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >> >       at java.lang.Class.forName0(Native Method)
> >> >       at java.lang.Class.forName(Class.java:242)
> >> >       at
> >> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
> >> > )
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
> >> >       ... 7 more
> >> >
> >> > java.lang.RuntimeException: java.lang.RuntimeException:
> >> > java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
> >> >       at
> org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
> >> >       at
> org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> >> >       at
> >> >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> >> >       at
> >> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> >> > )
> >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
> >> >       at
> >> >
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> >> > Caused by: java.lang.RuntimeException:
> >> > java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
> >> >       ... 6 more
> >> > Caused by: java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.examples.WordCount$MapClass
> >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> >> >       at java.security.AccessController.doPrivileged(Native Method)
> >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >> >       at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> >> >       at
> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> >> >       at java.lang.Class.forName0(Native Method)
> >> >       at java.lang.Class.forName(Class.java:242)
> >> >       at
> >> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
> >> > )
> >> >       at
> >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
> >> >       ... 7 more
> >
> >
>


Reply | Threaded
Open this post in threaded view
|

Re: Classpath setting problem for Map tasks

Avinash Lakshman-2
Thanks a million. Everything works now.

Thanks
Avinash


On 5/29/07 1:51 PM, "Mahadev Konar" <[hidden email]> wrote:

> Hi Avinash,
>   Java org.apache.... will not work. This does not copy the job.jar and
> that's why you run into these Classnotfoundexcpetions.
> The way to run the wordocunt example is:
> Bin/hadoop jar hadoop-examples.jar wordcount .......
>
> This way the jobclient is called by the hadoop command to transfer the
> job.jar and get the status of the job.
>
> Regards
> Mahadev
>
>> > -----Original Message-----
>> > From: Avinash Lakshman [mailto:[hidden email]]
>> > Sent: Tuesday, May 29, 2007 1:30 PM
>> > To: [hidden email]
>> > Subject: Re: Classpath setting problem for Map tasks
>> >
>> > Hi Mahadev
>> >
>> > This is how I am running it. I created a test data file and put it in HDFS
>> > using ./hadoop fs -put command. Then I modified my WordCount sample to add
>> > two lines to the main which look like this :
>> >
>> > conf.set("fs.default.name", "hdfs://dev030.sctm.facebook.com:9000");
>> > conf.set("mapred.job.tracker", "dev030.sctm.facebook.com:50029");
>> >
>> > Then I run the sample by executing the command :
>> >
>> > java org.apache.hadoop.examples.WordCount -m 10 -r 4
>> > /user/alakshman/test.dat ~/output-dir
>> >
>> > where the ~/output-dir is the local file system while the
>> > /user/alakshman/test.dat is the path in HDFS.
>> >
>> > This is when I get a whole slew of classnotfoundexceptions. Is the process
>> > incorrect ?
>> >
>> > Thanks
>> > Avinash
>> >
>> >
>> >
>> > On 5/29/07 1:17 PM, "Mahadev Konar" <[hidden email]> wrote:
>> >
>>> > > Hi Avinash,
>>> > >  Can you explain how you are running the wordcount example? Also, which
>>> > > version of Hadoop are you using? I would really recommend using 0.12.3.
>>> > >
>>> > > Regards
>>> > > Mahadev
>>> > >
>>>>> > >> > -----Original Message-----
>>>>> > >> > From: Phantom [mailto:[hidden email]]
>>>>> > >> > Sent: Tuesday, May 29, 2007 1:10 PM
>>>>> > >> > To: [hidden email]
>>>>> > >> > Subject: Classpath setting problem for Map tasks
>>>>> > >> >
>>>>> > >> > How do I specify the classpath via the JobConf ? In my jobtracker
web
>> > UI I
>>>>> > >> > see a whole slew of ClassNotFoundException's for the
>> > WordCount$MapClass.
>>>>> > >> > The
>>>>> > >> > stack trace is pasted below. But how do I fix this - just pass the
-

>> > cp
>>>>> > >> > <classpath> to the mapred.child.java.opts property of the JobConf ?
>>>>> > >> > exporting CLASSPATH doesn't seem to work.hh
>>>>> > >> >
>>>>> > >> > Thanks
>>>>> > >> > Avinash
>>>>> > >> >
>>>>> > >> > java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>>>>> > >> > )
>>>>> > >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>>>>> > >> > Caused by: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>>>>> > >> >       ... 6 more
>>>>> > >> > Caused by: java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> > >> >       at java.security.AccessController.doPrivileged(Native Method)
>>>>> > >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> > >> >       at
>> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> > >> >       at
>> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>>> > >> >       at java.lang.Class.forName0(Native Method)
>>>>> > >> >       at java.lang.Class.forName(Class.java:242)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>>>>> > >> > )
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>>>>> > >> >       ... 7 more
>>>>> > >> >
>>>>> > >> > java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:459)
>>>>> > >> >       at
>> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>>>>> > >> > )
>>>>> > >> >       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:170)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>>>>> > >> > Caused by: java.lang.RuntimeException:
>>>>> > >> > java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:328)
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:339)
>>>>> > >> >       ... 6 more
>>>>> > >> > Caused by: java.lang.ClassNotFoundException:
>>>>> > >> > org.apache.hadoop.examples.WordCount$MapClass
>>>>> > >> >       at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>>> > >> >       at java.security.AccessController.doPrivileged(Native Method)
>>>>> > >> >       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> > >> >       at
>> > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
>>>>> > >> >       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>>>>> > >> >       at
>> > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>>> > >> >       at java.lang.Class.forName0(Native Method)
>>>>> > >> >       at java.lang.Class.forName(Class.java:242)
>>>>> > >> >       at
>>>>> > >> >
>> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:315
>>>>> > >> > )
>>>>> > >> >       at
>>>>> > >> >
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:326)
>>>>> > >> >       ... 7 more
>>> > >
>>> > >
>> >
>
>
>