[jira] [Resolved] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Resolved] (HADOOP-17408) Optimize NetworkTopology while sorting of block locations

Steve Loughran (Jira)

     [ https://issues.apache.org/jira/browse/HADOOP-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Brennan resolved HADOOP-17408.
    Fix Version/s: 3.4.0
       Resolution: Fixed

Thanks for the contribution [~ahussein] and [~daryn]!  I have committed this to trunk and branch-3.3.

The patch does not apply cleanly to branch-3.2 or earlier.  Please provide a patch for 3.2 if desired.

> Optimize NetworkTopology while sorting of block locations
> ---------------------------------------------------------
>                 Key: HADOOP-17408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17408
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: common, net
>            Reporter: Ahmed Hussein
>            Assignee: Ahmed Hussein
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.3.1, 3.4.0
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
> In {{NetworkTopology}}, I noticed that there are some hanging fruits to improve the performance.
> Inside {{sortByDistance}}, collections.shuffle is performed on the list before calling {{secondarySort}}.
> {code:java}
> Collections.shuffle(list, r);
> if (secondarySort != null) {
>   secondarySort.accept(list);
> }
> {code}
> However, in different call sites, {{collections.shuffle}} is passed as the secondarySort to {{sortByDistance}}. This means that the shuffle is executed twice on each list.
> Also, logic wise, it is useless to shuffle before applying a tie breaker which might make the shuffle work obsolete.
> In addition, [~daryn] reported that:
> * topology is unnecessarily locking/unlocking to calculate the distance for every node
> * shuffling uses a seeded Random, instead of ThreadLocalRandom, which is heavily synchronized

This message was sent by Atlassian Jira

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]