Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

savannah_beckett
I am using the nutch nightly build #741 (Mar 3, 2009 4:01:53 AM).  I am at the final phrase of crawling following the tutorial on Nutch.org website.  I ran the following command, and I got exception in Hadoop.  I double checked the folder path in nutch-site.xml, and they are correct.  I tried multiple times, and I got same problem.  I didn't have same problem in 0.9.   What's wrong?

$ bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
Indexer: starting
Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)


Log from Hadoop:
2009-03-04 14:30:31,531 WARN  mapred.LocalJobRunner - job_local_0001
java.lang.IllegalArgumentException: it doesn't make sense to have a field that is neither indexed nor stored
        at org.apache.lucene.document.Field.<init>(Field.java:279)
        at org.apache.nutch.indexer.lucene.LuceneWriter.createLuceneDoc(LuceneWriter.java:133)
        at org.apache.nutch.indexer.lucene.LuceneWriter.write(LuceneWriter.java:239)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:40)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
2009-03-04 14:30:31,668 FATAL indexer.Indexer - Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

yanky
you can see hadoop log to find a clue

good luck

yanky

2009/3/5 dealmaker <[hidden email]>

>
> I am using the nutch nightly build #741 (Mar 3, 2009 4:01:53 AM).  I am at
> the final phrase of crawling following the tutorial on Nutch.org website.
>  I
> ran the following command, and I got exception in Hadoop.  I double checked
> the folder path in nutch-site.xml, and they are correct.  I tried multiple
> times, and I got same problem.  I didn't have same problem in 0.9.   What's
> wrong?
>
> $ bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
> Indexer: starting
> Indexer: java.io.IOException: Job failed!
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
>
> --
> View this message in context:
> http://www.nabble.com/Hadoop--java.io.IOException%3A-Job-failed%21-at-org.apache.hadoop.mapred.JobClient.runJob%28JobClient.java%3A1232%29-while-indexing.-tp22341554p22341554.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

andy2005cst
In reply to this post by savannah_beckett
I met the same problem as you, have you find a way to solve it?

dealmaker wrote
I am using the nutch nightly build #741 (Mar 3, 2009 4:01:53 AM).  I am at the final phrase of crawling following the tutorial on Nutch.org website.  I ran the following command, and I got exception in Hadoop.  I double checked the folder path in nutch-site.xml, and they are correct.  I tried multiple times, and I got same problem.  I didn't have same problem in 0.9.   What's wrong?

$ bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb crawl/segments/*
Indexer: starting
Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)


Log from Hadoop:
2009-03-04 14:30:31,531 WARN  mapred.LocalJobRunner - job_local_0001
java.lang.IllegalArgumentException: it doesn't make sense to have a field that is neither indexed nor stored
        at org.apache.lucene.document.Field.<init>(Field.java:279)
        at org.apache.nutch.indexer.lucene.LuceneWriter.createLuceneDoc(LuceneWriter.java:133)
        at org.apache.nutch.indexer.lucene.LuceneWriter.write(LuceneWriter.java:239)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:40)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:158)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:50)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
2009-03-04 14:30:31,668 FATAL indexer.Indexer - Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

fishg
In reply to this post by savannah_beckett
this is the solution,maybe can solve your problem.
http://www.txtob.com/bbs/dispbbs.asp?boardID=27&ID=1046&page=1
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

Filipe Antunes
fishg :
> this is the solution,maybe can solve your problem.
> http://www.txtob.com/bbs/dispbbs.asp?boardID=27&ID=1046&page=1
>  
Solution in chinese??
Translation didn't solve the problem.
Does anyone have a solution?


Reply | Threaded
Open this post in threaded view
|

RE: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

Davide.D'ALESSANDRO
I had the same problem, and I fixed it after filling in all the options in the files

nutch-default.xml
nutch-site.xml

I hope it helps.

Davide

-----Original Message-----
From: Filipe Antunes [mailto:[hidden email]]
Sent: Friday, July 31, 2009 11:04 AM
To: [hidden email]
Subject: Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

fishg :
> this is the solution,maybe can solve your problem.
> http://www.txtob.com/bbs/dispbbs.asp?boardID=27&ID=1046&page=1
>  
Solution in chinese??
Translation didn't solve the problem.
Does anyone have a solution?


Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

Filipe Antunes
Thanks for the tip.
Checked all config files, but the error is still the same.
Any other sugestions?


Davide.D'[hidden email] escreveu:

> I had the same problem, and I fixed it after filling in all the options in the files
>
> nutch-default.xml
> nutch-site.xml
>
> I hope it helps.
>
> Davide
>
> -----Original Message-----
> From: Filipe Antunes [mailto:[hidden email]]
> Sent: Friday, July 31, 2009 11:04 AM
> To: [hidden email]
> Subject: Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.
>
> fishg :
>  
>> this is the solution,maybe can solve your problem.
>> http://www.txtob.com/bbs/dispbbs.asp?boardID=27&ID=1046&page=1
>>  
>>    
> Solution in chinese??
> Translation didn't solve the problem.
> Does anyone have a solution?
>
>
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.

Chuan
I encountered the same problem and solved it like this:

In eclipse, add '-Xms64m -Xmx512m' in 'VM arguments' in 'Run As'.


Filipe Antunes wrote
Thanks for the tip.
Checked all config files, but the error is still the same.
Any other sugestions?


Davide.D'ALESSANDRO@ec.europa.eu escreveu:
> I had the same problem, and I fixed it after filling in all the options in the files
>
> nutch-default.xml
> nutch-site.xml
>
> I hope it helps.
>
> Davide
>
> -----Original Message-----
> From: Filipe Antunes [mailto:fantunes@tecnica.cc]
> Sent: Friday, July 31, 2009 11:04 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing.
>
> fishg :
>  
>> this is the solution,maybe can solve your problem.
>> http://www.txtob.com/bbs/dispbbs.asp?boardID=27&ID=1046&page=1
>>  
>>    
> Solution in chinese??
> Translation didn't solve the problem.
> Does anyone have a solution?
>
>
>
>