Quantcast

Mechanism of hadoop -jar

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Mechanism of hadoop -jar

jay vyas
Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
works i.e. how it copies/runs the jar file across the cluster, in order to
debug a jar issue.

1) Where can I get a good explanation of how the hadoop commands (i.e.
-jar) are implemented ?

2) Specifically, Im trying to access a bundled text file from a jar :

class.getResource("myfile.txt")

from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
ability to aquire local resources change  in the mapper/reducer JVMs ?



--
Jay Vyas
MMSB/UCHC
jay vyas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mechanism of hadoop -jar

Bertrand Dechoux
1) Source code for sure. I don't know if you could find any other technical
document about it.

2) Where is your file? If it is inside your jar, hadoop should not infer
with the 'normal way'. It is a 'classical jvm'. If you want to distribute
your files (across your nodes), you should look at DistributedCache.
http://hadoop.apache.org/common/docs/r1.0.0/api/org/apache/hadoop/filecache/DistributedCache.html

Regards

Bertrand

On Sun, Aug 12, 2012 at 12:09 AM, Jay Vyas <[hidden email]> wrote:

> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?
>
> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>



--
Bertrand Dechoux
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mechanism of hadoop -jar

in.abdul

Hi jay vas,
   If you are trying for accessing bungle of text file . Instead of going for class.getResource() why dont you go for distributed.cache in hadoop . As i had previously noticed about that in the mailing list

I hope it will help you better . If there is any problem let us know
Regards
Syed abdul kather

On Aug 12, 2012 4:17 AM, "Bertrand Dechoux [via Lucene]" <[hidden email]> wrote:
1) Source code for sure. I don't know if you could find any other technical
document about it.

2) Where is your file? If it is inside your jar, hadoop should not infer
with the 'normal way'. It is a 'classical jvm'. If you want to distribute
your files (across your nodes), you should look at DistributedCache.
http://hadoop.apache.org/common/docs/r1.0.0/api/org/apache/hadoop/filecache/DistributedCache.html

Regards

Bertrand

On Sun, Aug 12, 2012 at 12:09 AM, Jay Vyas <[hidden email]> wrote:

> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?
>
> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>


--
Bertrand Dechoux



If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Mechanism-of-hadoop-jar-tp4000622p4000625.html
To unsubscribe from Lucene, click here.
NAML
THANKS AND REGARDS, SYED ABDUL KATHER
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mechanism of hadoop -jar

Harsh J-2
In reply to this post by jay vyas
Hey,

On Sun, Aug 12, 2012 at 3:39 AM, Jay Vyas <[hidden email]> wrote:
> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?

The "jar" sub-command executes the org.apache.hadoop.util.RunJar class.

> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?

I believe this should work.

--
Harsh J
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mechanism of hadoop -jar

jay vyas
Sorry for the confusion... To be clear It is TOTALLY okay to jar up a text
file and access it in hadoop via the class.getResource(...) api !

1) Hadoop doesn't do anything funny with the class loader, it just uses the
Simple sun JVM class loader.

2) My problem was simply that I wasn't jarring up my text file properly.
 This was causing (obviously) all my mappers/reducers to not see my file.

Thanks for all the responses they were helpful !
jay vyas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mechanism of hadoop -jar

Arun Murthy
As someone already pointed out, you are better off using the DistributedCache (http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache) for what you are trying to achieve...

Arun

On Aug 11, 2012, at 7:09 PM, Jay Vyas wrote:

> Sorry for the confusion... To be clear It is TOTALLY okay to jar up a text
> file and access it in hadoop via the class.getResource(...) api !
>
> 1) Hadoop doesn't do anything funny with the class loader, it just uses the
> Simple sun JVM class loader.
>
> 2) My problem was simply that I wasn't jarring up my text file properly.
> This was causing (obviously) all my mappers/reducers to not see my file.
>
> Thanks for all the responses they were helpful !

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


Loading...