Running multiple copies of each task

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Running multiple copies of each task

Sultan Alamro
Hi there,

I have been looking at the hadoop source code 2.6.0 trying to understand the low level details and how the framework is actually working.

I have a simple idea and I am trying to figure out where and how the idea can be implemented. The idea can be described in one sentence: "Running multiple copies of each task". However, implementing the idea is not as simple as I think.

What I am aware of is that I only need to modify a few classes. But, which classes?

I just need someone to guide me to the right direction.


Best,
Sultan
Reply | Threaded
Open this post in threaded view
|

Re: Running multiple copies of each task

Namikaze Minato
I think you are looking for mapreduce.reduce.speculative
Be careful, for some reason, this fell into my spam folder.

Regards,
LLoyd

On 3 December 2015 at 01:05, Sultan Alamro <[hidden email]> wrote:

> Hi there,
>
> I have been looking at the hadoop source code 2.6.0 trying to understand the
> low level details and how the framework is actually working.
>
> I have a simple idea and I am trying to figure out where and how the idea
> can be implemented. The idea can be described in one sentence: "Running
> multiple copies of each task". However, implementing the idea is not as
> simple as I think.
>
> What I am aware of is that I only need to modify a few classes. But, which
> classes?
>
> I just need someone to guide me to the right direction.
>
>
> Best,
> Sultan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Running multiple copies of each task

Sultan Alamro
Thanks Namikaze!

Another question:

New tasks in hadoop always have higher priority than speculative tasks.
Does anyone know how and where I can change this priority?


Thanks,
Sultan


On Thu, Dec 3, 2015 at 7:46 AM, Namikaze Minato <[hidden email]> wrote:
I think you are looking for mapreduce.reduce.speculative
Be careful, for some reason, this fell into my spam folder.

Regards,
LLoyd

On 3 December 2015 at 01:05, Sultan Alamro <[hidden email]> wrote:
> Hi there,
>
> I have been looking at the hadoop source code 2.6.0 trying to understand the
> low level details and how the framework is actually working.
>
> I have a simple idea and I am trying to figure out where and how the idea
> can be implemented. The idea can be described in one sentence: "Running
> multiple copies of each task". However, implementing the idea is not as
> simple as I think.
>
> What I am aware of is that I only need to modify a few classes. But, which
> classes?
>
> I just need someone to guide me to the right direction.
>
>
> Best,
> Sultan

Reply | Threaded
Open this post in threaded view
|

Re: Running multiple copies of each task

Namikaze Minato
Talking about priority without telling us which scheduler and which scheduling method you are using is like forgetting to say in which unit you are measuring somehting :)

Regards,
LLoyd

On 17 December 2015 at 20:09, Sultan Alamro <[hidden email]> wrote:
Thanks Namikaze!

Another question:

New tasks in hadoop always have higher priority than speculative tasks.
Does anyone know how and where I can change this priority?


Thanks,
Sultan


On Thu, Dec 3, 2015 at 7:46 AM, Namikaze Minato <[hidden email]> wrote:
I think you are looking for mapreduce.reduce.speculative
Be careful, for some reason, this fell into my spam folder.

Regards,
LLoyd

On 3 December 2015 at 01:05, Sultan Alamro <[hidden email]> wrote:
> Hi there,
>
> I have been looking at the hadoop source code 2.6.0 trying to understand the
> low level details and how the framework is actually working.
>
> I have a simple idea and I am trying to figure out where and how the idea
> can be implemented. The idea can be described in one sentence: "Running
> multiple copies of each task". However, implementing the idea is not as
> simple as I think.
>
> What I am aware of is that I only need to modify a few classes. But, which
> classes?
>
> I just need someone to guide me to the right direction.
>
>
> Best,
> Sultan