Lucene › Hadoop Lucene › Hadoop lucene-users

tools.DistCp: Invalid arguments

‹ Previous Topic Next Topic ›
 
classic Classic list List threaded Threaded
♦
♦
 12 messages Options
xeonmailinglist-gmail
Reply | Threaded
Open this post in threaded view
♦
♦
|

tools.DistCp: Invalid arguments

xeonmailinglist-gmail

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​
Alexander Alten-Lorenz-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Alexander Alten-Lorenz-2
Have a closer look:

<a href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​
xeonmailinglist-gmail
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

xeonmailinglist-gmail

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

<a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a moz-do-not-send="true" href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
xeonmailinglist-gmail
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

xeonmailinglist-gmail
In reply to this post by Alexander Alten-Lorenz-2

Maybe this has to do with this error… I can’t do ls to my own machine using the command below. Can this be related to the other problem? Shouldn't I list the files with this command?

vagrant@hadoop-coc-1:~$ hdfs dfs -ls hdfs://192.168.56.100/
ls: Call From hadoop-coc-1/192.168.56.100 to hadoop-coc-1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

<a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a moz-do-not-send="true" href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
xeonmailinglist-gmail
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

xeonmailinglist-gmail
In reply to this post by xeonmailinglist-gmail
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

<a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a moz-do-not-send="true" href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
Alexander Alten-Lorenz-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Alexander Alten-Lorenz-2
In reply to this post by xeonmailinglist-gmail
Hi,

Can you please try webhdfs instead hdfs?

- Alexander 

On 03 Feb 2015, at 12:05, xeonmailinglist <[hidden email]> wrote:

Maybe this has to do with this error… I can’t do ls to my own machine using the command below. Can this be related to the other problem? Shouldn't I list the files with this command?

vagrant@hadoop-coc-1:~$ hdfs dfs -ls <a href="hdfs://192.168.56.100/" class="">hdfs://192.168.56.100/
ls: Call From hadoop-coc-1/192.168.56.100 to hadoop-coc-1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:



Have a closer look:

<a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a moz-do-not-send="true" href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​



​
Alexander Alten-Lorenz-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Alexander Alten-Lorenz-2
In reply to this post by xeonmailinglist-gmail
Ah, good. Cross-posting :)

BR,
 Alex

On 03 Feb 2015, at 12:41, xeonmailinglist <[hidden email]> wrote:

I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp <a href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a href="hdfs://hadoop-coc-2:50070/input1" class="">hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

<a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp <a moz-do-not-send="true" href="hdfs://hadoop-coc-1:50070/input1" class="">hdfs://hadoop-coc-1:50070/input1 <a moz-do-not-send="true" href="hdfs://hadoop-coc-2:50070/" class="">hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
Artem Ervits-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Artem Ervits-2
In reply to this post by xeonmailinglist-gmail

Another good option is hftp.

Artem Ervits

On Feb 3, 2015 6:42 AM, "xeonmailinglist" <[hidden email]> wrote:
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
Giri P
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Giri P
will distcp do checksum after it copied the data to target ?

On Tue, Feb 3, 2015 at 4:15 AM, Artem Ervits <[hidden email]> wrote:

Another good option is hftp.

Artem Ervits

On Feb 3, 2015 6:42 AM, "xeonmailinglist" <[hidden email]> wrote:
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​
Harsh J-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Harsh J-2
Yes, if the length matches and if you haven't specifically asked it to ignore checksums.

On Thursday, July 9, 2015, Giri P <[hidden email]> wrote:
will distcp do checksum after it copied the data to target ?

On Tue, Feb 3, 2015 at 4:15 AM, Artem Ervits <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;artemervits@gmail.com&#39;);" target="_blank">artemervits@...> wrote:

Another good option is hftp.

Artem Ervits

On Feb 3, 2015 6:42 AM, "xeonmailinglist" <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;xeonmailinglist@gmail.com&#39;);" target="_blank">xeonmailinglist@...> wrote:
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;xeonmailinglist@gmail.com&#39;);" target="_blank">xeonmailinglist@...> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​




--
Harsh J
Giri P
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Giri P
so ,there is no need to explicitly do checksum on source and target after we migrate.

I was thinking of using comparechecksum in distcpUtils to compare after migration

On Fri, Jul 10, 2015 at 4:20 PM, Harsh J <[hidden email]> wrote:
Yes, if the length matches and if you haven't specifically asked it to ignore checksums.


On Thursday, July 9, 2015, Giri P <[hidden email]> wrote:
will distcp do checksum after it copied the data to target ?

On Tue, Feb 3, 2015 at 4:15 AM, Artem Ervits <[hidden email]> wrote:

Another good option is hftp.

Artem Ervits

On Feb 3, 2015 6:42 AM, "xeonmailinglist" <[hidden email]> wrote:
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​




--
Harsh J
Harsh J-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: tools.DistCp: Invalid arguments

Harsh J-2
No, you don't need to recheck.

On Saturday, July 11, 2015, Giri P <[hidden email]> wrote:
so ,there is no need to explicitly do checksum on source and target after we migrate.

I was thinking of using comparechecksum in distcpUtils to compare after migration

On Fri, Jul 10, 2015 at 4:20 PM, Harsh J <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;harsh@cloudera.com&#39;);" target="_blank">harsh@...> wrote:
Yes, if the length matches and if you haven't specifically asked it to ignore checksums.


On Thursday, July 9, 2015, Giri P <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;gpatcham@gmail.com&#39;);" target="_blank">gpatcham@...> wrote:
will distcp do checksum after it copied the data to target ?

On Tue, Feb 3, 2015 at 4:15 AM, Artem Ervits <[hidden email]> wrote:

Another good option is hftp.

Artem Ervits

On Feb 3, 2015 6:42 AM, "xeonmailinglist" <[hidden email]> wrote:
I have found the problem. I started to use `webhdfs` and everything is ok.


On 03-02-2015 10:40, xeonmailinglist wrote:

What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here?

$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1

Thanks,

On 02-02-2015 19:59, Alexander Alten-Lorenz wrote:

Have a closer look:

No Path is given.


On 02 Feb 2015, at 20:52, xeonmailinglist <[hidden email]> wrote:

Hi,

I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?


vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/
15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:111)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:401)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
    at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
    at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop-coc-1/127.0.1.1"; destination host is: "hadoop-coc-2":50070; 
usage: distcp OPTIONS [source_path...] <target_path>

Thanks,

​

​




--
Harsh J



--
Harsh J
Free forum by Nabble Edit this page