[jira] Created: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
change the open method in ClientProtocol to take an additional argument: clientMachine
--------------------------------------------------------------------------------------

                 Key: HADOOP-842
                 URL: http://issues.apache.org/jira/browse/HADOOP-842
             Project: Hadoop
          Issue Type: Improvement
          Components: dfs
    Affects Versions: 0.9.2
            Reporter: Hairong Kuang
         Assigned To: Hairong Kuang
             Fix For: 0.10.0


The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org

     [ http://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-842:
---------------------------------

    Attachment: open.patch

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: http://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.0
>
>         Attachments: open.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ http://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-842:
---------------------------------

    Status: Patch Available  (was: Open)

The patch changes the open method of ClientProtocol to take an additional argument clientMachine in the form of machineName:datanodeport#. The purpose is to allow the namenode to sort the returning datanodes by its distance to the client machine. The patch includes all the interface changes to dfs. The change to implementation of the open method will be included in the rack-aware HDFS patch.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: http://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.0
>
>         Attachments: open.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ http://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461807 ]

Hadoop QA commented on HADOOP-842:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12348170/open.patch applied and successfully tested against trunk revision r489707.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: http://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.0
>
>         Attachments: open.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-842:
--------------------------------

    Status: Open  (was: Patch Available)

Overall this sounds like a good change.  A few minor issues:

The following is repeated many times:

+          org.apache.hadoop.net.DNS.getDefaultHost("default")+":"
+                          +jspHelper.conf.get("dfs.datanode.port")

It seems this code should be rather be shared.

Also, you commented out a line of code that I think you meant to delete.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.0
>
>         Attachments: open.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462377 ]

Hairong Kuang commented on HADOOP-842:
--------------------------------------

Thanks Doug for your suggestion.

I realized that passing the datanode port # as a part of clientMachine is not a good idea. When a datanode starts up, it tries to use the port from conf.get("dfs.datanode.port"). But if there is a failure, it tries a different port#. So it is not right that a dfs client simply assumes that the local datanode has the port# of conf.get("dfs.datanode.port").

I'd like to change the open method so that parameter clientMachine contains only hostName but no port#. At the name node side, the datanode list will be searched based on hostname not hostname:port#.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.0
>
>         Attachments: open.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-842:
---------------------------------

    Attachment: open1.patch

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>         Attachments: open.patch, open1.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-842:
---------------------------------

    Fix Version/s: 0.10.1
           Status: Patch Available  (was: Open)

The new patch allows the open request send the client host name along with the name of the file to be opened.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.10.1
>
>         Attachments: open.patch, open1.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-842:
--------------------------------

    Fix Version/s:     (was: 0.10.1)
                   0.11.0

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.11.0
>
>         Attachments: open.patch, open1.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462657 ]

Hadoop QA commented on HADOOP-842:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12348421/open1.patch applied and successfully tested against trunk revision r493236.

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.11.0
>
>         Attachments: open.patch, open1.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-842) change the open method in ClientProtocol to take an additional argument: clientMachine

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-842:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Hairong!

> change the open method in ClientProtocol to take an additional argument: clientMachine
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-842
>                 URL: https://issues.apache.org/jira/browse/HADOOP-842
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.9.2
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.11.0
>
>         Attachments: open.patch, open1.patch
>
>
> The open method returns a list of blocks that belong to a file and, for each block, a list of datanodes that contain the block.To improve reading efficiency, I'd like to return the list of datanodes in the order of its distance to the client machine. So I suggest that we make the open method to take an additional argument clientMachine which is in the format of hostname:port#.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira