Formatting the namenode

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Formatting the namenode

alakshman
Hi

When I format my namenode it does format the directory specified under
dfs.name.dir. However there is a folder under /tmp called hadoop-alakshman.
What is this for ? Will all blocks be stored under /var/hadoop somewhere.
 Here is a copy of hadoop-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>dfs.name.dir</name>
                <value>/var/hadoop</value>
        </property>

        <property>
                <name>fs.default.name</name>
                <value>hdfs://hadoop001.sf2p.com:9000</value>
        </property>

        <property>
                <name>mapred.job.tracker</name>
                <value>hadoop001.sf2p.com:50029</value>
        </property>

        <property>
                <name>mapred.job.tracker.info.port</name>
                <value>50030</value>
        </property>

        <property>
                <name>mapred.child.java.opts</name>
                <value>-Xmx1024M</value>
        </property>

        <property>
                <name>mapred.min.split.size</name>
                <value>65536</value>
        </property>

        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>

</configuration>
Reply | Threaded
Open this post in threaded view
|

Re: Formatting the namenode

Michael Bieniosek-2
In hadoop-default.xml you should find:

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>dfs.name.dir</name>
  <value>${hadoop.tmp.dir}/dfs/name</value>
  <description>Determines where on the local filesystem the DFS name node
      should store the name table.  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>${hadoop.tmp.dir}/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

and some other properties that use the hadoop.tmp.dir prefix.  What
subdirectories did hadoop create inside /tmp/hadoop-${user.name}?

-Michael

On 6/14/07 10:39 PM, "Phantom" <[hidden email]> wrote:

> Hi
>
> When I format my namenode it does format the directory specified under
> dfs.name.dir. However there is a folder under /tmp called hadoop-alakshman.
> What is this for ? Will all blocks be stored under /var/hadoop somewhere.
>  Here is a copy of hadoop-site.xml
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>         <property>
>                 <name>dfs.name.dir</name>
>                 <value>/var/hadoop</value>
>         </property>
>
>         <property>
>                 <name>fs.default.name</name>
>                 <value>hdfs://hadoop001.sf2p.com:9000</value>
>         </property>
>
>         <property>
>                 <name>mapred.job.tracker</name>
>                 <value>hadoop001.sf2p.com:50029</value>
>         </property>
>
>         <property>
>                 <name>mapred.job.tracker.info.port</name>
>                 <value>50030</value>
>         </property>
>
>         <property>
>                 <name>mapred.child.java.opts</name>
>                 <value>-Xmx1024M</value>
>         </property>
>
>         <property>
>                 <name>mapred.min.split.size</name>
>                 <value>65536</value>
>         </property>
>
>         <property>
>                 <name>dfs.replication</name>
>                 <value>3</value>
>         </property>
>
> </configuration>

Reply | Threaded
Open this post in threaded view
|

RE: Formatting the namenode

Avinash Lakshman-2
Dfs and mapred are the directories created inside the
/tmp/hadoop-<user-name> directory.

Thanks
A

-----Original Message-----
From: Michael Bieniosek [mailto:[hidden email]]
Sent: Friday, June 15, 2007 11:31 AM
To: [hidden email]; Phantom
Subject: Re: Formatting the namenode

In hadoop-default.xml you should find:

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>dfs.name.dir</name>
  <value>${hadoop.tmp.dir}/dfs/name</value>
  <description>Determines where on the local filesystem the DFS name
node
      should store the name table.  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>${hadoop.tmp.dir}/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

and some other properties that use the hadoop.tmp.dir prefix.  What
subdirectories did hadoop create inside /tmp/hadoop-${user.name}?

-Michael

On 6/14/07 10:39 PM, "Phantom" <[hidden email]> wrote:

> Hi
>
> When I format my namenode it does format the directory specified under
> dfs.name.dir. However there is a folder under /tmp called
hadoop-alakshman.
> What is this for ? Will all blocks be stored under /var/hadoop
somewhere.

>  Here is a copy of hadoop-site.xml
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>         <property>
>                 <name>dfs.name.dir</name>
>                 <value>/var/hadoop</value>
>         </property>
>
>         <property>
>                 <name>fs.default.name</name>
>                 <value>hdfs://hadoop001.sf2p.com:9000</value>
>         </property>
>
>         <property>
>                 <name>mapred.job.tracker</name>
>                 <value>hadoop001.sf2p.com:50029</value>
>         </property>
>
>         <property>
>                 <name>mapred.job.tracker.info.port</name>
>                 <value>50030</value>
>         </property>
>
>         <property>
>                 <name>mapred.child.java.opts</name>
>                 <value>-Xmx1024M</value>
>         </property>
>
>         <property>
>                 <name>mapred.min.split.size</name>
>                 <value>65536</value>
>         </property>
>
>         <property>
>                 <name>dfs.replication</name>
>                 <value>3</value>
>         </property>
>
> </configuration>

Reply | Threaded
Open this post in threaded view
|

Re: Formatting the namenode

Michael Bieniosek-2
Go through hadoop-default.xml and look at all the paths that install into
/tmp/hadoop-${user.name}.  You should probably override all of those values
in hadoop-site.xml.

-Michael

On 6/15/07 12:23 PM, "Avinash Lakshman" <[hidden email]> wrote:

> Dfs and mapred are the directories created inside the
> /tmp/hadoop-<user-name> directory.
>
> Thanks
> A
>
> -----Original Message-----
> From: Michael Bieniosek [mailto:[hidden email]]
> Sent: Friday, June 15, 2007 11:31 AM
> To: [hidden email]; Phantom
> Subject: Re: Formatting the namenode
>
> In hadoop-default.xml you should find:
>
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/tmp/hadoop-${user.name}</value>
>   <description>A base for other temporary directories.</description>
> </property>
>
> <property>
>   <name>dfs.name.dir</name>
>   <value>${hadoop.tmp.dir}/dfs/name</value>
>   <description>Determines where on the local filesystem the DFS name
> node
>       should store the name table.  If this is a comma-delimited list
>       of directories then the name table is replicated in all of the
>       directories, for redundancy. </description>
> </property>
>
> <property>
>   <name>dfs.data.dir</name>
>   <value>${hadoop.tmp.dir}/dfs/data</value>
>   <description>Determines where on the local filesystem an DFS data node
>   should store its blocks.  If this is a comma-delimited
>   list of directories, then data will be stored in all named
>   directories, typically on different devices.
>   Directories that do not exist are ignored.
>   </description>
> </property>
>
> and some other properties that use the hadoop.tmp.dir prefix.  What
> subdirectories did hadoop create inside /tmp/hadoop-${user.name}?
>
> -Michael
>
> On 6/14/07 10:39 PM, "Phantom" <[hidden email]> wrote:
>
>> Hi
>>
>> When I format my namenode it does format the directory specified under
>> dfs.name.dir. However there is a folder under /tmp called
> hadoop-alakshman.
>> What is this for ? Will all blocks be stored under /var/hadoop
> somewhere.
>>  Here is a copy of hadoop-site.xml
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>>         <property>
>>                 <name>dfs.name.dir</name>
>>                 <value>/var/hadoop</value>
>>         </property>
>>
>>         <property>
>>                 <name>fs.default.name</name>
>>                 <value>hdfs://hadoop001.sf2p.com:9000</value>
>>         </property>
>>
>>         <property>
>>                 <name>mapred.job.tracker</name>
>>                 <value>hadoop001.sf2p.com:50029</value>
>>         </property>
>>
>>         <property>
>>                 <name>mapred.job.tracker.info.port</name>
>>                 <value>50030</value>
>>         </property>
>>
>>         <property>
>>                 <name>mapred.child.java.opts</name>
>>                 <value>-Xmx1024M</value>
>>         </property>
>>
>>         <property>
>>                 <name>mapred.min.split.size</name>
>>                 <value>65536</value>
>>         </property>
>>
>>         <property>
>>                 <name>dfs.replication</name>
>>                 <value>3</value>
>>         </property>
>>
>> </configuration>
>