hadoop on unstable nodes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

hadoop on unstable nodes

real great..
Hi,
Is there any research currently going on where map reduce is applied to
nodes in normal internet scenarios?.In environments where network bandwidth
is at premium what are the tweaks applied to hadoop?
I would be very thankful if you can post me links in this direction.

--
Regards,
R.V.
Reply | Threaded
Open this post in threaded view
|

Re: hadoop on unstable nodes

Alex Loddengaard-3
I don't know of any research, but such a scenario is likely not going to
turn out so well.  Hadoop is very network hungry and is designed to be run
in a datacenter.  Sorry I don't have more information for you.

Alex

On Mon, Aug 2, 2010 at 9:14 PM, Rahul.V. <[hidden email]>wrote:

> Hi,
> Is there any research currently going on where map reduce is applied to
> nodes in normal internet scenarios?.In environments where network bandwidth
> is at premium what are the tweaks applied to hadoop?
> I would be very thankful if you can post me links in this direction.
>
> --
> Regards,
> R.V.
>
Reply | Threaded
Open this post in threaded view
|

Re: hadoop on unstable nodes

Chen He
Condor has a hadoop subproject in UW-Madison, and there are also some
scientists from VT. They worked on security Hadoop MapReduce on Internet.

In my opinion, Alex is correct, Hadoop MR is communication intensive
especially in the map and shuffle stage. In the map stage, every mapper
needs input data from File System. If your data distributed among Internet,
you may encounter heavy delay. Also in the shuffle stage, reducer collect
mapper's intermediate results through Internet. This is another bottleneck
we can not overlook.

Hope this will help.

Chen

On Tue, Aug 3, 2010 at 11:37 AM, Alex Loddengaard <[hidden email]> wrote:

> I don't know of any research, but such a scenario is likely not going to
> turn out so well.  Hadoop is very network hungry and is designed to be run
> in a datacenter.  Sorry I don't have more information for you.
>
> Alex
>
> On Mon, Aug 2, 2010 at 9:14 PM, Rahul.V. <[hidden email]
> >wrote:
>
> > Hi,
> > Is there any research currently going on where map reduce is applied to
> > nodes in normal internet scenarios?.In environments where network
> bandwidth
> > is at premium what are the tweaks applied to hadoop?
> > I would be very thankful if you can post me links in this direction.
> >
> > --
> > Regards,
> > R.V.
> >
>