Mechanism of hadoop -jar

[jay vyas]

Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
works i.e. how it copies/runs the jar file across the cluster, in order to
debug a jar issue.

1) Where can I get a good explanation of how the hadoop commands (i.e.
-jar) are implemented ?

2) Specifically, Im trying to access a bundled text file from a jar :

class.getResource("myfile.txt")

from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
ability to aquire local resources change  in the mapper/reducer JVMs ?



--
Jay Vyas
MMSB/UCHC

[Bertrand Dechoux]

1) Source code for sure. I don't know if you could find any other technical
document about it.

2) Where is your file? If it is inside your jar, hadoop should not infer
with the 'normal way'. It is a 'classical jvm'. If you want to distribute
your files (across your nodes), you should look at DistributedCache.
http://hadoop.apache.org/common/docs/r1.0.0/api/org/apache/hadoop/filecache/DistributedCache.html

Regards

Bertrand

On Sun, Aug 12, 2012 at 12:09 AM, Jay Vyas <[hidden email]> wrote:

> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?
>
> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>

--
Bertrand Dechoux

[in.abdul]

Hi jay vas,
   If you are trying for accessing bungle of text file . Instead of going for class.getResource() why dont you go for distributed.cache in hadoop . As i had previously noticed about that in the mailing list

I hope it will help you better . If there is any problem let us know
Regards
Syed abdul kather

On Aug 12, 2012 4:17 AM, "Bertrand Dechoux [via Lucene]" <[hidden email]> wrote:

1) Source code for sure. I don't know if you could find any other technical
document about it.

2) Where is your file? If it is inside your jar, hadoop should not infer
with the 'normal way'. It is a 'classical jvm'. If you want to distribute
your files (across your nodes), you should look at DistributedCache.
http://hadoop.apache.org/common/docs/r1.0.0/api/org/apache/hadoop/filecache/DistributedCache.html

Regards

Bertrand

On Sun, Aug 12, 2012 at 12:09 AM, Jay Vyas <[hidden email]> wrote:

> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?
>
> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>

--
Bertrand Dechoux

---

If you reply to this email, your message will be added to the discussion below:

http://lucene.472066.n3.nabble.com/Mechanism-of-hadoop-jar-tp4000622p4000625.html

To unsubscribe from Lucene, click here.
NAML

[Harsh J-2]

Hey,

On Sun, Aug 12, 2012 at 3:39 AM, Jay Vyas <[hidden email]> wrote:
> Hi guys:  I'm trying to find documentation on how "hadoop jar" actually
> works i.e. how it copies/runs the jar file across the cluster, in order to
> debug a jar issue.
>
> 1) Where can I get a good explanation of how the hadoop commands (i.e.
> -jar) are implemented ?

The "jar" sub-command executes the org.apache.hadoop.util.RunJar class.

> 2) Specifically, Im trying to access a bundled text file from a jar :
>
> class.getResource("myfile.txt")
>
> from inside a mapreduce job.... Is it okay to do this ?  Or does a classes
> ability to aquire local resources change  in the mapper/reducer JVMs ?

I believe this should work.

--
Harsh J

[jay vyas]

Sorry for the confusion... To be clear It is TOTALLY okay to jar up a text
file and access it in hadoop via the class.getResource(...) api !

1) Hadoop doesn't do anything funny with the class loader, it just uses the
Simple sun JVM class loader.

2) My problem was simply that I wasn't jarring up my text file properly.
 This was causing (obviously) all my mappers/reducers to not see my file.

Thanks for all the responses they were helpful !

[Arun C Murthy-4]

As someone already pointed out, you are better off using the DistributedCache (http://hadoop.apache.org/common/docs/r1.0.3/mapred_tutorial.html#DistributedCache) for what you are trying to achieve...

Arun

On Aug 11, 2012, at 7:09 PM, Jay Vyas wrote:

> Sorry for the confusion... To be clear It is TOTALLY okay to jar up a text
> file and access it in hadoop via the class.getResource(...) api !
>
> 1) Hadoop doesn't do anything funny with the class loader, it just uses the
> Simple sun JVM class loader.
>
> 2) My problem was simply that I wasn't jarring up my text file properly.
> This was causing (obviously) all my mappers/reducers to not see my file.
>
> Thanks for all the responses they were helpful !

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/