Mapred - NDFS Startup issues

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Mapred - NDFS Startup issues

Jon Shoberg

The data node does not want to start has anyone seen this problem
before?  Following are points of data I have.  This is done using the
mapred version as checked out last night.  My onlu goal here is to see
how / what NDFS is about and begin testing with it for my application.

I hope that I am missing something trivial.  My starting point on this
for docs was the mailing list and wiki, please see below.  Yes, I have
iptables turned off.

Thanks
Jon


--- !!! FAILURE OF DATA NODE TO START !!!
------------------------------------

[jon@devbox nutchmapre]$ ./bin/nutch datanode
050728 115952 10 parsing file:/usr/local/nutchmapre/conf/nutch-default.xml
050728 115952 10 parsing file:/usr/local/nutchmapre/conf/nutch-site.xml
050728 115952 10 Opened server at 7000
050728 115952 11 Starting DataNode in: /tmp/nutch/ndfs/data/data
050728 115952 11 Exception: java.util.NoSuchElementException
050728 115952 11 Lost connection to namenode.  Retrying...


--- STARTING OF THE NAME NODE
-----------------------------------------------
[jon@devbox nutchmapre]$ ./bin/nutch namenode
050728 120637 parsing file:/usr/local/nutchmapre/conf/nutch-default.xml
050728 120637 parsing file:/usr/local/nutchmapre/conf/nutch-site.xml
050728 120637 Server listener on port 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting
050728 120637 Server handler on 9000: starting


--- Nutch Site Conf XML
-----------------------------------------------------

[jon@devbox nutchmapre]$ cat conf/nutch-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="nutch-conf.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<nutch-conf>
<!-- file system properties -->
<property>
   <name>fs.default.name</name>
   <value>127.0.0.1:9000</value>
   <description>The name of the default file system.  Either the
   literal string "local" or a host:port for NDFS.</description>
</property>
<property>
   <name>ndfs.name.dir</name>
   <value>/tmp/nutch/ndfs/name</value>
   <description>Determines where on the local filesystem the NDFS name node
       should store the name table.</description>
</property>
<property>
   <name>ndfs.data.dir</name>
   <value>/tmp/nutch/ndfs/data</value>
   <description>Determines where on the local filesystem an NDFS data node
   should store its blocks.  If this is a comma- or space-delimited
   list of directories, then data will be stored in all named
   directories, typically on different devices.</description>
</property>
</nutch-conf>

--- NUTCH NDFS DIRECTORY STRUCTURE -----------------------------------------

[jon@devbox nutchmapre]$ ls -laR /tmp/nutch/ndfs
/tmp/nutch/ndfs:
total 32
drwxrwxr-x  4 jon jon 4096 Jul 28 08:23 .
drwxrwxr-x  4 jon jon 4096 Jul 28 08:23 ..
drwxrwxr-x  4 jon jon 4096 Jul 28 11:59 data
drwxrwxr-x  3 jon jon 4096 Jul 28 08:26 name

/tmp/nutch/ndfs/data:
total 32
drwxrwxr-x  4 jon jon 4096 Jul 28 11:59 .
drwxrwxr-x  4 jon jon 4096 Jul 28 08:23 ..
drwxrwxr-x  2 jon jon 4096 Jul 28 08:26 data
drwxrwxr-x  2 jon jon 4096 Jul 28 11:59 tmp

/tmp/nutch/ndfs/data/data:
total 16
drwxrwxr-x  2 jon jon 4096 Jul 28 08:26 .
drwxrwxr-x  4 jon jon 4096 Jul 28 11:59 ..

/tmp/nutch/ndfs/data/tmp:
total 16
drwxrwxr-x  2 jon jon 4096 Jul 28 11:59 .
drwxrwxr-x  4 jon jon 4096 Jul 28 11:59 ..

/tmp/nutch/ndfs/name:
total 28
drwxrwxr-x  3 jon jon 4096 Jul 28 08:26 .
drwxrwxr-x  4 jon jon 4096 Jul 28 08:23 ..
-rw-rw-r--  1 jon jon    0 Jul 28 11:59 edits
drwxrwxr-x  2 jon jon 4096 Jul 28 08:26 image

/tmp/nutch/ndfs/name/image:
total 16
drwxrwxr-x  2 jon jon 4096 Jul 28 08:26 .
drwxrwxr-x  3 jon jon 4096 Jul 28 08:26 ..