Installing just the HDFS client

classic Classic list List threaded Threaded
3 messages Options
F21
Reply | Threaded
Open this post in threaded view
|

Installing just the HDFS client

F21
Hi all,

I am currently building a HBase docker image. As part of the bootstrap
process, I need to run some `hdfs dfs` commands to create directories on
HDFS.

The whole hadoop distribution is pretty heavy contains things to run
namenodes, etc. I just need a copy of the dfs client for my docker
image. I have done some poking around and see that I need to include the
files in bin/, libexec/, lib/ and share/hadoop/common and share/hadoop/hdfs.

However, including the above still takes up quite a bit of space. Is
there a single JAR I can add to my image to perform operations against HDFS?


Cheers,

Francis


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Installing just the HDFS client

권병창

Hi

 

refer to below pom.xml

you should modify hadoop version that you want.

 

build is simple:  mvn clean package 

It will make a jar  of 34mb size. 

 

usage is simple:

java -jar ${build_jar}.jar -mkdir /user/home

java -jar ${build_jar}.jar -ls /user/home

 

 

pom.xml

<?xml version="1.0" encoding="UTF-8"?> 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.naver.c3</groupId>
  <artifactId>hdfs-connector</artifactId>
  <version>1.0-SNAPSHOT</version>

  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.7.1</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <minimizeJar>false</minimizeJar>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>org.apache.hadoop.fs.FsShell</mainClass>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>
           </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

 

 

-----Original Message-----
From: "F21"<[hidden email]>
To: <[hidden email]>;
Cc:
Sent: 2016-08-29 (월) 14:25:09
Subject: Installing just the HDFS client
 

Hi all,

I am currently building a HBase docker image. As part of the bootstrap
process, I need to run some `hdfs dfs` commands to create directories on
HDFS.

The whole hadoop distribution is pretty heavy contains things to run
namenodes, etc. I just need a copy of the dfs client for my docker
image. I have done some poking around and see that I need to include the
files in bin/, libexec/, lib/ and share/hadoop/common and share/hadoop/hdfs.

However, including the above still takes up quite a bit of space. Is
there a single JAR I can add to my image to perform operations against HDFS?


Cheers,

Francis


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: Installing just the HDFS client

권병창

let's  default core-site.xml, hdfs-site.xml log4j.properties (is not your customized *.xml) locate like below tree.

and re-build jar using 'mvn clean package'.  

finally resources(*.xml *.properties) will locate into a jar.

 .

├── pom.xml 

└── src

    └── main

        └── resources

            ├── core-site.xml

            ├── hdfs-site.xml

            └── log4j.properties

 

and you can use a jar:

 

java -jar ${build_jar} -conf /opt/hadoop/etc/hdfs-site.xml -ls hdfs://${nameservices}/user/home


 

-----Original Message-----
From: "F21"<[hidden email]>
To: "권병창"<[hidden email]>;
Cc:
Sent: 2016-08-30 (화) 12:47:16
Subject: Re: Installing just the HDFS client
 

I am still getting the same error despite setting the variable. In my case, hdfs-site.xml and core-site.xml is customized at run time, so cannot be compiled into the jar.

My core-site.xml and hdfs-site.xml is located in /opt/hbase/conf

Here's what I did:

bash-4.3# export HADOOP_CONF_DIR="/opt/hbase/conf"
bash-4.3# echo $HADOOP_CONF_DIR
/opt/hbase/conf
bash-4.3# cd /opt/hadoop/
bash-4.3# java -jar hdfs-fs.jar
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.RuntimeException: core-site.xml not found
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2566)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1143)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1451)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

On 30/08/2016 11:20 AM, 권병창 wrote:
set below environmental variable.
 
export HADOOP_CONF_DIR=/opt/hadoop/etc

or
 
hdfs-site.xml core-site.xml can locate in the jar.


-----Original Message-----
보낸 사람: F21 [hidden email]
받는 사람: 권병창 [hidden email]
참조:
날짜: 2016. 8. 30 오전 9:12:58
제목: Re: Installing just the HDFS client

Hi,

Thanks for the pom.xml. I was able to build it successfully. How do I point it to the config files? My core-site.xml and hdfs-site.xml are located in /opt/hadoop/etc.

I tried the following:
java -jar hdfs-fs.jar -ls /
java -jar hdfs-fs.jar --config /opt/hbase/etc -ls /
java -jar hdfs-fs.jar -conf /opt/hbase/etc -ls /

This is the error I am getting:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.RuntimeException: core-site.xml not found
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2566)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1143)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1451)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

How do I point the jar to use the hdfs-site.xml and core-site.xml located in /opt/hadoop/etc?

Cheers,
Francis

On 29/08/2016 5:18 PM, 권병창 wrote:

Hi

 

refer to below pom.xml

you should modify hadoop version that you want.

 

build is simple:  mvn clean package 

It will make a jar  of 34mb size. 

 

usage is simple:

java -jar ${build_jar}.jar -mkdir /user/home

java -jar ${build_jar}.jar -ls /user/home

 

 

pom.xml

<?xml version="1.0" encoding="UTF-8"?> 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.naver.c3</groupId>
  <artifactId>hdfs-connector</artifactId>
  <version>1.0-SNAPSHOT</version>

  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.7.1</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <minimizeJar>false</minimizeJar>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>org.apache.hadoop.fs.FsShell</mainClass>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>
           </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

 

 

-----Original Message-----
From: "F21"[hidden email]
To: [hidden email];
Cc:
Sent: 2016-08-29 (월) 14:25:09
Subject: Installing just the HDFS client
 

Hi all,

I am currently building a HBase docker image. As part of the bootstrap
process, I need to run some `hdfs dfs` commands to create directories on
HDFS.

The whole hadoop distribution is pretty heavy contains things to run
namenodes, etc. I just need a copy of the dfs client for my docker
image. I have done some poking around and see that I need to include the
files in bin/, libexec/, lib/ and share/hadoop/common and share/hadoop/hdfs.

However, including the above still takes up quite a bit of space. Is
there a single JAR I can add to my image to perform operations against HDFS?


Cheers,

Francis


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]