Hadoop job question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop job question

HUYLEBROECK Jeremy RD-ILAB-SSF-2

I currently have a MR task that reads a SequenceFile via the map method
to output some data.

My goal is to output some data to MySQL but I'd like to read several
records before doing the INSERT.

But I can't figure out how to get several records...
They all have different keys so the reduce task only gets one at a time.


Thanks for any help!

Reply | Threaded
Open this post in threaded view
|

Re: Hadoop job question

Dennis Kubes
Although it is kinda hacking the system you may be able to do it in the
map method by writing a custom MapRunner and having an object that lives
in the MapRunner but that you set into each mapper instance.

Dennis

HUYLEBROECK Jeremy RD-ILAB-SSF wrote:

> I currently have a MR task that reads a SequenceFile via the map method
> to output some data.
>
> My goal is to output some data to MySQL but I'd like to read several
> records before doing the INSERT.
>
> But I can't figure out how to get several records...
> They all have different keys so the reduce task only gets one at a time.
>
>
> Thanks for any help!
>
>  
Reply | Threaded
Open this post in threaded view
|

RE: Hadoop job question

HUYLEBROECK Jeremy RD-ILAB-SSF-2
In reply to this post by HUYLEBROECK Jeremy RD-ILAB-SSF-2
Thanks for the pointer.
It does perfectly the job!


-----Original Message-----
From: Dennis Kubes [mailto:[hidden email]]
Sent: Tuesday, August 29, 2006 7:59 AM
To: [hidden email]
Subject: Re: Hadoop job question

Although it is kinda hacking the system you may be able to do it in the
map method by writing a custom MapRunner and having an object that lives
in the MapRunner but that you set into each mapper instance.

Dennis

HUYLEBROECK Jeremy RD-ILAB-SSF wrote:
> I currently have a MR task that reads a SequenceFile via the map
> method to output some data.
>
> My goal is to output some data to MySQL but I'd like to read several
> records before doing the INSERT.
>
> But I can't figure out how to get several records...
> They all have different keys so the reduce task only gets one at a
time.
>
>
> Thanks for any help!
>
>