Starting a job on a hadoop cluster remotly

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Starting a job on a hadoop cluster remotly

Sebastian Ruff (open4business GmbH)
Hey,

 

is it possible to start a job on a hadoop cluster from remote? For
example we have a web application

which runs on an apache tomcat server. And would like to start a
mapreduce job on our cluster, from

within the webapp.

 

Is this possible? And if yes, what are the steps to get there? Do I just
have to put my namenode and datanode

in a core-site.xml in the webapp and call the api?

 

Thanks a lot,

 

Sebastian

 

Reply | Threaded
Open this post in threaded view
|

Re: Starting a job on a hadoop cluster remotly

Michael Sutter-3

  Hey,

yes it is possible. I'm doing it exactly this way for my implementation
from a remote client.

Implement it like this:
Configuration conf = new Configuration();
conf.set("hadoop.job.ugi", "user, group");
conf.set("namenode.host", somehost.somedomain);
conf.set("jobtracker.host", somehost.somedomain);
conf.set("mapred.job.tracker", somehost.somedomain:someport);
conf.set("fs.default.name", hdfs://somehost.somedomain:someport);
Job job = new Job(conf, jobname);
job.setJarByClass(...);
...

Cheers
Michael

On 07/28/2010 03:36 PM, Sebastian Ruff (open4business GmbH) wrote:

> Hey,
>
>
>
> is it possible to start a job on a hadoop cluster from remote? For
> example we have a web application
>
> which runs on an apache tomcat server. And would like to start a
> mapreduce job on our cluster, from
>
> within the webapp.
>
>
>
> Is this possible? And if yes, what are the steps to get there? Do I just
> have to put my namenode and datanode
>
> in a core-site.xml in the webapp and call the api?
>
>
>
> Thanks a lot,
>
>
>
> Sebastian
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Starting a job on a hadoop cluster remotly

James Seigel
Not sure exactly your goals, but look into SOCKS proxy stuff as well.  You can have the hadoop command binary running locally and talking over a socks proxy to the actual cluster, without having to have the machines exposed all over the place.

Cheers
James.


On 2010-07-28, at 10:42 AM, Michael Sutter wrote:

>
>  Hey,
>
> yes it is possible. I'm doing it exactly this way for my implementation
> from a remote client.
>
> Implement it like this:
> Configuration conf = new Configuration();
> conf.set("hadoop.job.ugi", "user, group");
> conf.set("namenode.host", somehost.somedomain);
> conf.set("jobtracker.host", somehost.somedomain);
> conf.set("mapred.job.tracker", somehost.somedomain:someport);
> conf.set("fs.default.name", hdfs://somehost.somedomain:someport);
> Job job = new Job(conf, jobname);
> job.setJarByClass(...);
> ...
>
> Cheers
> Michael
>
> On 07/28/2010 03:36 PM, Sebastian Ruff (open4business GmbH) wrote:
>> Hey,
>>
>>
>>
>> is it possible to start a job on a hadoop cluster from remote? For
>> example we have a web application
>>
>> which runs on an apache tomcat server. And would like to start a
>> mapreduce job on our cluster, from
>>
>> within the webapp.
>>
>>
>>
>> Is this possible? And if yes, what are the steps to get there? Do I just
>> have to put my namenode and datanode
>>
>> in a core-site.xml in the webapp and call the api?
>>
>>
>>
>> Thanks a lot,
>>
>>
>>
>> Sebastian
>>
>>
>>
>>