Input directory urls/url-fr.txt in localhost:9000 is invalid with Hadoop 0.4.0patched and Nutch 0.8.1

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Input directory urls/url-fr.txt in localhost:9000 is invalid with Hadoop 0.4.0patched and Nutch 0.8.1

yleny@ifrance.com
Hello,

I can not index website with Nutch and Hadoop.
I spend 15 days to try that nutch 0.8.1 work but with no success.

I use :
* jdk1.5.0_10 or 1.4.2_13 (i have the same problem with these 2 JDK)
* Nutch-0.8.1
* Hadoop-0.4.0 from Nutch-0.8.1
And I made configuration with http://wiki.apache.org/nutch/NutchHadoopTutorial
I have ONLY one server.

I start Hadoop-0.4.0 (start-all.sh) with no errors in the logs.

The directories for crawl and url are created.
I use dfs put command line to put them in NDFS File System.

[nutch-0.8.1]$ bin/hadoop dfs -lsr
/user/webadm/crawls     <dir>
/user/webadm/urls       <dir>
/user/webadm/urls/url-fr.txt        <r 1>   44

And when I crawl with nutch 0.8.1, I have this message error:

[nutch-0.8.1]$ bin/nutch crawl urls/url-fr.txt  -dir crawls/crawl-fr -depth 10  -topN 50
crawl started in: crawls/crawl-fr
rootUrlDir = urls/url-fr.txt
threads = 10
depth = 10
topN = 50
Injector: starting
Injector: crawlDb: crawls/crawl-fr/crawldb
Injector: urlDir: urls/url-fr.txt
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Input directory /user/webadm/urls/url-fr.txt in localhost:9000 is invalid.
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:274)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
        at org.apache.nutch.crawl.Injector.inject(Injector.java:138)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:105)
        
        
*******************************************************************

In the nutch-0.8.1/logs/namenodep.log, I have this error

2007-01-19 20:05:26,522 WARN  fs.FSNamesystem - Replication requested of 2 is larger than cluster size (1). Using cluster size.
2007-01-19 20:05:26,523 WARN  fs.FSNamesystem - Zero targets found, forbidden1.size=1 forbidden2.size()=0... 16 more


In my /opt/nutch-0.8.1/conf/hadoop-env.sh and /opt/nutch-0.8.1/conf/hadoop-env.sh,
I Have:

export HADOOP_HOME=/opt/nutch-0.8.1
export JAVA_HOME=/logiciels/java/jdk1.5.0_10
export HADOOP_HEAPSIZE=1000
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
export HADOOP_PID_DIR=/opt/nutch-0.8.1/pids

****************************************

In my /opt/nutch-0.8.1/conf/hadoop-site.xml
I Have these values with xml tags of course:

hadoop.tmp.dir = /opt/nutch-0.8.1/HDFS
fs.default.name = localhost:9000
dfs.name.dir = /opt/nutch-0.8.1/HDFS/dfs/name
dfs.client.buffer.dir = /opt/nutch-0.8.1/HDFS/dfs/tmp
dfs.data.dir = /opt/nutch-0.8.1/HDFS/dfs/data
dfs.replication = 1
mapred.job.tracker = localhost:9001
mapred.local.dir = /opt/nutch-0.8.1/HDFS/mapred/local
mapred.system.dir = /opt/nutch-0.8.1/HDFS/mapred/system
mapred.temp.dir = /opt/nutch-0.8.1/HDFS/mapred/temp
mapred.map.tasks = 2
mapred.reduce.tasks = 1


In my conf/crawl-urlfilter.txt, I have this line to index:
+^http://www.mywebsite.com

In my urls/url-mywebsite, I have this line to index:
http://www.mywebsite.com/index.htm

In my conf/nutch-site.xml, I have this line:
searcher.dir = crawls/crawl-mywebsite

Thanks in advance.

Yannick LE NY


________________________________________________________________________
iFRANCE, exprimez-vous !
http://web.ifrance.com
Reply | Threaded
Open this post in threaded view
|

Re: Input directory urls/url-fr.txt in localhost:9000 is invalid with Hadoop 0.4.0patched and Nutch 0.8.1

Andrzej Białecki-2
[hidden email] wrote:

> Hello,
>
> I can not index website with Nutch and Hadoop.
> I spend 15 days to try that nutch 0.8.1 work but with no success.
>
> I use :
> * jdk1.5.0_10 or 1.4.2_13 (i have the same problem with these 2 JDK)
> * Nutch-0.8.1
> * Hadoop-0.4.0 from Nutch-0.8.1
> And I made configuration with http://wiki.apache.org/nutch/NutchHadoopTutorial
> I have ONLY one server.
>
> I start Hadoop-0.4.0 (start-all.sh) with no errors in the logs.
>
> The directories for crawl and url are created.
> I use dfs put command line to put them in NDFS File System.
>
> [nutch-0.8.1]$ bin/hadoop dfs -lsr
> /user/webadm/crawls     <dir>
> /user/webadm/urls       <dir>
> /user/webadm/urls/url-fr.txt        <r 1>   44
>
> And when I crawl with nutch 0.8.1, I have this message error:
>
> [nutch-0.8.1]$ bin/nutch crawl urls/url-fr.txt  -dir crawls/crawl-fr -depth 10  -topN 50
> crawl started in: crawls/crawl-fr
> rootUrlDir = urls/url-fr.txt
> threads = 10
> depth = 10
> topN = 50
> Injector: starting
> Injector: crawlDb: crawls/crawl-fr/crawldb
> Injector: urlDir: urls/url-fr.txt
> Injector: Converting injected urls to crawl db entries.
> Exception in thread "main" java.io.IOException: Input directory /user/webadm/urls/url-fr.txt in localhost:9000 is invalid.
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:274)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
>         at org.apache.nutch.crawl.Injector.inject(Injector.java:138)
>         at org.apache.nutch.crawl.Crawl.main(Crawl.java:105)
>        
>  


.. and that's because "urlDir: urls/url-fr.txt" is not a directory, but
a file. You should give only the "urls" as the input directory - Nutch
will read all text files inside the directory.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com