Specifying Input conditon to split file or specifying map tasks to work on assigned files individually

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Specifying Input conditon to split file or specifying map tasks to work on assigned files individually

Pallavi Palleti
Hi,
 I am exploring hadoop and using it for one of my machine learning application.
 I have a problem in which I need to route a particular input to each map task separately. For example, I have list of <key, value >pairs sorted on some condition in an input file. I want to split the input file on some condition (for example, all key,value pairs which have the same key should be given as input to a particular map task). I want to do this, so that all the necessary extra information related to that input can be loaded into memory once in that map task so that my map procedure will be faster.

So, Can some one please tell me is there a way to specify a condition in such a way that a particular input will be given to a particular map task? Or, I can split the files before hand and is there a way I can specify each map task to work on each file separately with out any splits. Please clarify.

Thanks in advance,
Reply | Threaded
Open this post in threaded view
|

Re: Specifying Input conditon to split file or specifying map tasks to work on assigned files individually

Ted Dunning-3


If you over-ride some implementation of InputFormat such as TextInputFormat,
you can cause Hadoop to not split your files by over-riding isSplitable.
This over-ride can be configured using JobConf.setInputFormat.

Now go to the wiki where you would hvave expected to find this answer and
either put the answer there or tell me where you would have looked.

Remember ASF = Apache Software Foundation.


On 7/23/07 9:42 PM, "novice user" <[hidden email]> wrote:

>
> Hi,
>  I am exploring hadoop and using it for one of my machine learning
> application.
>  I have a problem in which I need to route a particular input to each map
> task separately. For example, I have list of <key, value >pairs sorted on
> some condition in an input file. I want to split the input file on some
> condition (for example, all key,value pairs which have the same key should
> be given as input to a particular map task). I want to do this, so that all
> the necessary extra information related to that input can be loaded into
> memory once in that map task so that my map procedure will be faster.
>
> So, Can some one please tell me is there a way to specify a condition in
> such a way that a particular input will be given to a particular map task?
> Or, I can split the files before hand and is there a way I can specify each
> map task to work on each file separately with out any splits. Please
> clarify.
>
> Thanks in advance,

Reply | Threaded
Open this post in threaded view
|

Re: Specifying Input conditon to split file or specifying map tasks to work on assigned files individually

David Biesack
In reply to this post by Pallavi Palleti
> Date: Mon, 23 Jul 2007 21:42:34 -0700 (PDT)
> From: novice user <[hidden email]>
>
>
> Hi,
>  I am exploring hadoop and using it for one of my machine learning
> application.
>  I have a problem in which I need to route a particular input to each map
> task separately. For example, I have list of <key, value >pairs sorted on
> some condition in an input file. I want to split the input file on some
> condition (for example, all key,value pairs which have the same key should
> be given as input to a particular map task). I want to do this, so that all
> the necessary extra information related to that input can be loaded into
> memory once in that map task so that my map procedure will be faster.

This sounds like you can put your map processing into your Reduce operation,
since Hadoop will already pass all values with the same key to your reducer.
Thus, an Identity map may suffice. (I've not tried running a job without
specifying a Map; maybe Hadoop works without one, in which case you do
not even need the Map.) In the case where you want to partition on
other condition, can you not simply do that by mapping your keys into
the different enumerated values (perhaps pushing the old key into the
output value).  If I'm of the mark, sorry if I misinterpreted your problem statement.

--
David J. Biesack     SAS Institute Inc.
(919) 531-7771       SAS Campus Drive
http://www.sas.com   Cary, NC 27513