Quantcast

rack awareness unexpected behaviour

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

rack awareness unexpected behaviour

Marc Sturlese
This post was updated on .
Hey there,
I've set up rack awareness on my hadoop cluster with replication 3. I have 2 racks and each contains 50% of the nodes.
I can see that the blocks are spread on the 2 racks, the problem is that all nodes from a rack are storing 2 replicas and the nodes of the other rack just one. If I launch the hadoop balancer script, it will properly spread the replicas across the 2 racks, leaving all nodes with exactly the same available disk space but, after jobs are running for hours, the data will be unbalanced again (rack1 having all nodes with less empty disk space than all nodes from rack2)

Any clue whats going on?
Thanks in advance
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Nicolas Liochon
Do the jobs run on the whole cluster or a single rack?
If you write from a single rack, you will get something similar to what you
described, because the default policy is to put one block locally and 2
blocks on the same remote rack. It does check that there is enough place
available, but does not try to balance.


On Thu, Aug 22, 2013 at 9:41 AM, Marc Sturlese <[hidden email]>wrote:

> Hey there,
> I've set up rack awareness on my hadoop cluster with replication 3. I have
> 2
> racks and each contains 50% of the nodes.
> I can see that the blocks are spread on the 2 racks, the problem is that
> all
> nodes from a rack are storing 2 replicas and the nodes of the other rack
> just one. If I launch the hadoop balancer script, it will properly spread
> the replicas across the 2 racks, leaving all nodes with exactly the same
> available disk space but, after jobs are running for hours, the data will
> be
> unbalanced again (rack1 having all nodes with less empty disk space than
> all
> nodes from rack2)
>
> Any clue whats going on?
> Thanks in advance
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/rack-awarness-unexpected-behaviour-tp4086029.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Marc Sturlese
Jobs run on the whole cluster. After rebalancing everything is properly allocated. Then I start running jobs using all the slots of the 2 racks and the problem starts to happen.
Maybe I'm missing something. When using the rack awareness, do you have to specify to the jobs to run in slots form both racks and not just one? (I guess not)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Nicolas Liochon
When you rebalance, the block is fully written, so the writer locality does
not have to be taken into account (there is no writer anymore), hence it
can rebalance across the racks. That's why jobs asymmetry was the easy
guess. What's your hadoop version by the way? I remember a bug around rack
awareness, but it has been fixed a year ago (and I'm not sure it would have
had this effect).


On Thu, Aug 22, 2013 at 10:29 AM, Marc Sturlese <[hidden email]>wrote:

> Jobs run on the whole cluster. After rebalancing everything is properly
> allocated. Then I start running jobs using all the slots of the 2 racks and
> the problem starts to happen.
> Maybe I'm missing something. When using the rack awareness, do you have to
> specify to the jobs to run in slots form both racks and not just one? (I
> guess not)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4086038.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Marc Sturlese
I'm on cdh3u4 (0.20.2), gonna try to read a bit on this bug
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Harsh J-2
I'm not aware of a bug in 0.20.2 that would not honor the Rack
Awareness, but have you done the two below checks as well?

1. Ensuring JT has the same rack awareness scripts and configuration
so it can use it for scheduling, and,
2. Checking if the map and reduce tasks are being evenly spread across
both racks.

On Thu, Aug 22, 2013 at 2:50 PM, Marc Sturlese <[hidden email]> wrote:
> I'm on cdh3u4 (0.20.2), gonna try to read a bit on this bug
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4086049.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.



--
Harsh J
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Michael Segel
Rack aware is an artificial concept.
Meaning you can define where a node is regardless of is real position in the rack.

Going from memory, and its probably been changed in later versions of the code...

Isn't the replication... Copy on node 1, copy on same rack, third copy on different rack?

Or has this been improved upon?

Sent from a remote device. Please excuse any typos...

Mike Segel

On Aug 22, 2013, at 5:14 AM, Harsh J <[hidden email]> wrote:

> I'm not aware of a bug in 0.20.2 that would not honor the Rack
> Awareness, but have you done the two below checks as well?
>
> 1. Ensuring JT has the same rack awareness scripts and configuration
> so it can use it for scheduling, and,
> 2. Checking if the map and reduce tasks are being evenly spread across
> both racks.
>
> On Thu, Aug 22, 2013 at 2:50 PM, Marc Sturlese <[hidden email]> wrote:
>> I'm on cdh3u4 (0.20.2), gonna try to read a bit on this bug
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4086049.html
>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
>
> --
> Harsh J
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Jun Ping Du
For 3 replicas, the replication sequence is: 1st on local node of Writer, 2nd on remote rack node of 1st replica, 3rd on same rack of 2nd replica.
There could be some special cases like: disk is full on 1st node, or no node available for 2nd replica rack, and Hadoop already take care it well. Agree with Harsh, you should check if tasks are evenly distributed across two racks first.

Thanks,

Junping

----- Original Message -----
From: "Michel Segel" <[hidden email]>
To: [hidden email]
Cc: [hidden email], [hidden email]
Sent: Thursday, August 22, 2013 6:57:15 PM
Subject: Re: rack awarness unexpected behaviour

Rack aware is an artificial concept.
Meaning you can define where a node is regardless of is real position in the rack.

Going from memory, and its probably been changed in later versions of the code...

Isn't the replication... Copy on node 1, copy on same rack, third copy on different rack?

Or has this been improved upon?

Sent from a remote device. Please excuse any typos...

Mike Segel

On Aug 22, 2013, at 5:14 AM, Harsh J <[hidden email]> wrote:

> I'm not aware of a bug in 0.20.2 that would not honor the Rack
> Awareness, but have you done the two below checks as well?
>
> 1. Ensuring JT has the same rack awareness scripts and configuration
> so it can use it for scheduling, and,
> 2. Checking if the map and reduce tasks are being evenly spread across
> both racks.
>
> On Thu, Aug 22, 2013 at 2:50 PM, Marc Sturlese <[hidden email]> wrote:
>> I'm on cdh3u4 (0.20.2), gonna try to read a bit on this bug
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4086049.html
>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
>
>
> --
> Harsh J
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Marc Sturlese
I've check it out and it works like that. The problem is, if the two racks have not the same capacity, one will have the disk space filled up much faster than the other (that's what I'm seeing).
If one rack (rack A) has 2 servers of 8 cores with 4 reduce slots each and the other rack (rack B) has 2 servers of 16 cores with 8 reduce slots each, rack A will get filled up faster as rack B is writing more (because has more reduce slots).

Could a solution be to modify the bash script used to decide to which replica write a block? Would use probability and give to rack B double chance to receive de write.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Michael Segel
Marc,

The rack aware script is an artificial concept. Meaning you can tell which machine is in which rack and that may or may not reflect where the machine is actually located.
The idea is to balance the number of nodes in the racks, at least on paper.  So you can have 14 machines in rack 1, and 16 machines in rack 2 even though they may physically be 20 machines in rack 1 and 10 machines in rack 2.

HTH

-Mike

On Oct 3, 2013, at 2:52 AM, Marc Sturlese <[hidden email]> wrote:

> I've check it out and it works like that. The problem is, if the two racks
> have not the same capacity, one will have the disk space filled up much
> faster than the other (that's what I'm seeing).
> If one rack (rack A) has 2 servers of 8 cores with 4 reduce slots each and
> the other rack (rack B) has 2 servers of 16 cores with 8 reduce slots each,
> rack A will get filled up faster as rack B is writing more (because has more
> reduce slots).
>
> Could a solution be to modify the bash script used to decide to which
> replica write a block? Would use probability and give to rack B double
> chance to receive de write.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4093270.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>

The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Marc Sturlese
Doing that will balance the block writing but I think here you loose the concept of physical rack awareness.
Let's say you have 2 physical racks, one with 2 servers and one with 4. If you artificially tell hadoop that one rack has 3 servers and the other 3 you are loosing the concept of rack awareness. You're not guaranteeing that each physical rack contains at least a replica of each block.

So if you have 2 racks with different number of servers, it's not possible to do proper rack awareness without filling the disks of the rack with less servers first. Am I right or am I missing something?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Michael Segel
And that's the rub.
Rack awareness is an artificial construct.

You want to fix it and match the real world, you need to balance the racks physically.
Otherwise you need to rewrite load balancing to take in to consideration the number and power of the nodes in the rack.

The short answer, it's easier to fudge the values in the script.

Sent from a remote device. Please excuse any typos...

Mike Segel

> On Oct 3, 2013, at 8:58 AM, Marc Sturlese <[hidden email]> wrote:
>
> Doing that will balance the block writing but I think here you loose the
> concept of physical rack awareness.
> Let's say you have 2 physical racks, one with 2 servers and one with 4. If
> you artificially tell hadoop that one rack has 3 servers and the other 3 you
> are loosing the concept of rack awareness. You're not guaranteeing that each
> physical rack contains at least a replica of each block.
>
> So if you have 2 racks with different number of servers, it's not possible
> to do proper rack awareness without filling the disks of the rack with less
> servers first. Am I right or am I missing something?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4093337.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: rack awarness unexpected behaviour

Jun Ping Du
In reply to this post by Michael Segel
The current HDFS's default replica placement policy don't fit two biased racks case very well: assume local rack has more nodes, which means more reducer slots and more disk capacity, then more reducer tasks will be executed within local rack. According to replica placement policy, it will put 1 replica on local rack and 2 replicas on remote rack which means data load are doubled in remote rack although less capacity there.
The workaround of cheating rack-aware script (like described below) may help to resolve unbalanced data problem but will take following two issues:
1. data reliability - all 3 replicas of some blocks may fall into the same "real" rack.
2. rack level data locality - no matter task scheduling or replica choosing in HDFS read will get mis-understand on real rack topology.
See if this is tradeoff you want to get in your case.
Another workaround, although not design for this case, may be helpful: to enable "NodeGroup" level of locality that between node and rack which is supported after 1.2.0. Nodes under the same "NodeGroup" can only have one replica placed which is designed for getting rid of replica duplicated for VMs on the same host. Specifically in your case, assume you have 20 machines in rack A and 10 machines in rack B, you can put rack A nodes to two NodeGroups (so each NodeGroup has 10 nodes) and rack B nodes to one NodeGroups. In this case, the replica will be distributed in ratio of 2:1, no matter where the writer is. Hope it helps.


Thanks,

Junping

----- Original Message -----
From: "Michael Segel" <[hidden email]>
To: [hidden email]
Cc: [hidden email]
Sent: Thursday, October 3, 2013 8:23:58 PM
Subject: Re: rack awarness unexpected behaviour

Marc,

The rack aware script is an artificial concept. Meaning you can tell which machine is in which rack and that may or may not reflect where the machine is actually located.
The idea is to balance the number of nodes in the racks, at least on paper.  So you can have 14 machines in rack 1, and 16 machines in rack 2 even though they may physically be 20 machines in rack 1 and 10 machines in rack 2.

HTH

-Mike

On Oct 3, 2013, at 2:52 AM, Marc Sturlese <[hidden email]> wrote:

> I've check it out and it works like that. The problem is, if the two racks
> have not the same capacity, one will have the disk space filled up much
> faster than the other (that's what I'm seeing).
> If one rack (rack A) has 2 servers of 8 cores with 4 reduce slots each and
> the other rack (rack B) has 2 servers of 16 cores with 8 reduce slots each,
> rack A will get filled up faster as rack B is writing more (because has more
> reduce slots).
>
> Could a solution be to modify the bash script used to decide to which
> replica write a block? Would use probability and give to rack B double
> chance to receive de write.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/rack-awareness-unexpected-behaviour-tp4086029p4093270.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>

The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Loading...