FW: Job failed!

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

FW: Job failed!

Edward Quick
For info only. I fixed this problem by removing the mapreduce directory in tmp before running another fetch.


From: [hidden email]
To: [hidden email]; [hidden email]
Subject: FW: Job failed!
Date: Sat, 6 Sep 2008 07:10:11 +0000

Hi,

I reran the fetch and got this error again after 5 hours. Any ideas what causes this?


2008-09-06 04:10:23,062 WARN  mapred.LocalJobRunner - job_local_1
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_local_1/job_local_1_map_0000/output/spill0.out in an
y of the configured local directories
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:359)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138)
        at org.apache.hadoop.mapred.MapOutputFile.getSpillFile(MapOutputFile.java:94)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:972)
        at org.apache.hadoop.mapred.MapT ask$MapOutputBuffer.flush(MapTask.java:698)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
2008-09-06 04:10:23,860 FATAL fetcher.Fetcher - Fetcher: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
        at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:552)
        at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:587)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:559)






From: [hidden email]
To: [hidden email]
Subject: FW: Job failed!
Date: Fri, 5 Sep 2008 21:58:13 +0000

Sorry for all these posts. I found the problem. Had a dodgy segment, probably the one which was left after the last fetch bombed out.


From: [hidden email]
To: [hidden email]
Subject: FW: Job failed!
Date: Fri, 5 Sep 2008 21:49:38 +0000

Struth! Here's another problem as well. I'm trying to merge the segments I've created so far:

$ nutch mergesegs crawl/mergesegs_dir -dir crawl/segments
Merging 5 segments to crawl/mergesegs_dir/20080905223155
SegmentMerger:   adding file:/tmp/crawl/segments/20080905141605
SegmentMerger:   adding file:/tmp/crawl/segments/20080905141522
SegmentMerger:   adding file:/tmp/crawl/segments/20080905142231
SegmentMerger:   adding file:/tmp/crawl/segments/20080905153116
SegmentMerger:   adding file:/tmp/crawl/segments/20080905141348
SegmentMerger: using segment data from: crawl_generate

$ find crawl/mergesegs_dir
crawl/mergesegs_dir
crawl/mergesegs_dir/20080905223155
crawl/mergesegs_dir/20080905223155/crawl_generate
crawl/mergesegs_dir/20080905223155/crawl_generate/.part-00000.crc
crawl/mergesegs_dir/20080905223155/crawl_generate/part-00000

But when I run invertlinks, I get an err or about a missing path:

$ mv crawl/segments crawl/BACKUPsegments
$ mv crawl/mergesegs_dir crawl/segments
$ nutch invertlinks crawl/linkdb -dir crawl/segments
LinkDb: starting
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: file:/tmp/crawl/segments/20080905223155
LinkDb: org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : file:/tmp/crawl/segments/20080905223155/parse_data
        at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputFormat.java:215)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
        at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:170)
        at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:285)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:248)




From: [hidden email]
To: [hidden email]
Subject: FW: Job failed!
Date: Fri, 5 Sep 2008 21:09:13 +0000

Sort of figured out how to kickstart the crawl again.

Basically did:

$s1=ls -d crawl/segments/* | tail -1
bin/nutch updatedb crawl/crawldb $1
bin/nutch generate crawl/crawldb crawl/segments
$2=ls -d crawl/segments/* | tail -1
bin/nutch fetch $2

But unfortunately this is fetching the same urls as the previous fetch. :(


From: [hidden email]
To: [hidden email]
Subject: RE: Job failed!
Date: Fri, 5 Sep 2008 09:45:00 +0000

Initially I just did a tail -10 so thought there were no errors, but there are a few actually. The pdf errors are my fault because I updated the pdf plugin with the latest PDFBox and FontBox jars from cvs on sf.net and missed out parse-pdf.jar on the rebuild. I'm not sure that's the reason why the job failed though. The log is 5MB so I can't really attach it all here but hopefully the last 200 lines gives an indication.

By the way, is there a way to kickstart this crawl off again without crawling from the start again?


tail -200 hadoop.log.2008-09-05
2008-09-05 03:41:22,360 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:22,360 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:22,360 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:22,360 WARN  parse. ParserFactory - ... 4 more
2008-09-05 03:41:22,360 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:22,360 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/Premium+Service+Training+insert/$FILE/Premium+training.pdf of type application/pdf
2008-09-05 03:41:22,362 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/CTP+-+Travel+Plan+Objectives?OpenDocument
2008-09-05 03:41:23,616 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptsal.nsf/Content/ctcBA+Home%5CBusTools%5CC%5Ccomp+tckts%5Ccr+comp+tickets?OpenDocument
2008-09-05 03:41:24,745 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/Notes+7+- +93+Rooms?OpenDocument
2008-09-05 03:41:26,033 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/SCCM+January+/$FILE/SCCMonthly+-+NovDec%2C+07+%2808+Jan%2C+08%29.pdf
2008-09-05 03:41:27,215 WARN  parse.ParserFactory - org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:166)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParserFactory.getParsers(ParserFactory.java:133)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:67)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:355)
2008-09-05 03:41:27,216 WARN  parse.ParserF actory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:178)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - Caused by: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at java.security.AccessController.doPrivileged(Native Method)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:27,216 WARN& nbsp; parse.ParserFactory - ... 4 more
2008-09-05 03:41:27,216 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:27,216 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/SCCM+January+/$FILE/SCCMonthly+-+NovDec%2C+07+%2808+Jan%2C+08%29.pdf of type application/pdf
2008-09-05 03:41:27,216 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/bani.nsf/Content/XXXXLS%5FQ1Results%5F030807%5CXXXXLS%5FQ1Resultsvideo%5F030807?opendocument
2008-09-05 03:41:28,451 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptflt.nsf/AttachmentsByTitle/Flight+Ops+News+Aug+2008/$FILE/FLIGHT+OPS_AUGUST_08+intranet+live.pdf
2008-09-05 03:41:29,760 INFO  fetcher.Fetcher - fetching http ://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/Virus+2+questions?OpenDocument
2008-09-05 03:41:30,789 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/Gender+Reass+the+process?OpenDocument
2008-09-05 03:41:32,066 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/LGW+Crew+Responsibilities/$FILE/Crew+Responsibilities.doc
2008-09-05 03:41:33,390 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptflt.nsf/Content/Flight+Ops+Home%5CBusiness+Tools%5CFlight+Technical+Services%5CAircraft+Weights+%26+Evaluation%5CFleet+Weights+-+Aircraft+Weighing+Schedules?OpenDocument
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at org.apache.nutch.plugi n.Extension.getExtensionInstance(Extension.java:166)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParserFactory.getParsers(ParserFactory.java:133)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:67)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:355)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:178)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - Caused by: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at java.security.AccessController.doPrivileged(Native Method)
2008-09-05 03:41:34,562 WARN   parse.ParserFactory - at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:34,562 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:34,563 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:34,563 WARN  parse.ParserFactory - ... 4 more
2008-09-05 03:41:34,563 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:34,563 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptrix.nsf/AttachmentsByTitle/T5+Retail+-+T5+Ground+Level/$FILE/T5_Ground_Level.pdf of type application/ pdf
2008-09-05 03:41:34,564 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/travel/stpg2.nsf/072561aa006322660725618c006b09a0/fc11f85e25deb736802574a30033c99e?OpenDocument
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:166)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParserFactory.getParsers(ParserFactory.java:133)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:67)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:355)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.fe tcher.Fetcher$FetcherThread.run(Fetcher.java:178)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - Caused by: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at java.security.AccessController.doPrivileged(Native Method)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - ... 4 more
2008-09-05 03:41:35,926 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:35,926 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptrix.nsf/AttachmentsByTitle/Diversity+dignity+at+work+booklet/$FILE/Dignity+at+work+booklet.pdf of type application/pdf
2008-09-05 03:41:35,928 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/communications/wtps1.nsf/$lookup/1D94AD9A45B463638025730100263FDF
2008-09-05 03:41:36,988 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/Barplus+Hints+and+Tips/$FILE/Barplus+Hints+and+Tips.pdf
2008-09-05 03:41:38,217 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/apteng.nsf/Content/Engineering+Home%5CDepar tment+Information%5CEngineering+IT+Support+%26+Delivery+Homepage%5CEngineering+Solution+Group+%28ESG%29+Homepage%5CKey+user+Guides?OpenDocument
2008-09-05 03:41:41,143 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/Cultural+Awareness+Photo+Prize+Draw?OpenDocument
2008-09-05 03:41:42,278 WARN  parse.ParserFactory - org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:166)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParserFactory.getParsers(ParserFactory.java:133)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:67)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$Fe tcherThread.output(Fetcher.java:355)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:178)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - Caused by: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at java.security.AccessController.doPrivileged(Native Method)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - at org.apache.nutch.pl ugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - ... 4 more
2008-09-05 03:41:42,279 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:42,279 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptflt.nsf/AttachmentsByTitle/Flight+Ops+News+Aug+2008/$FILE/FLIGHT+OPS_AUGUST_08+intranet+live.pdf of type application/pdf
2008-09-05 03:41:42,313 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptsal.nsf/Content/ctcBA+Home%5CBusTools%5CB%5Cbah%5CPromos+pckge%5CFlrda+08+EBO+WTP+upgde?OpenDocument
2008-09-05 03:41:42,342 INFO  fetcher.Fetcher - fetching http://planetba.baplc.com/general/aptrix/aptrix.nsf/Content/PMA+EG904+timescales?OpenDocument
2008-09-05 03:4 1:52,279 WARN  parse.ParserFactory - org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:166)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParserFactory.getParsers(ParserFactory.java:133)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:67)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:355)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:178)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - Caused by: java.lang.ClassNotFoundException: org.apache.nutch.parse.pdf.PdfParser
2008-09-05 03:41:52,27 9 WARN  parse.ParserFactory - at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at java.security.AccessController.doPrivileged(Native Method)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:156)
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - ... 4 more
2008-09-05 03:41:52,279 WARN  parse.ParserFactory - ParserFactory:PluginRuntimeException when initializing parser plugin parse-pdf instance in getParsers function: attempting to continue instantiating parsers
2008-09-05 03:41:52,279 WARN  parse.ParseUtil - Unable to successfully parse content http://planetba.baplc.com/general/aptrix/aptcsops.nsf/AttachmentsByTitle/Barplus+Hints+and+Tips/$FILE/Barplus+Hints+and+Tips.pdf of type application/pdf
2008-09-05 03:41:55,927 WARN  mapred.LocalJobRunner - job_local_21
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_local_21/job_local_21_map_0000/output/file.out
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
        at org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
        at org.apach e.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:982)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
2008-09-05 09:32:46,906 INFO  searcher.NutchBean - opening indexes in crawl/indexes
2008-09-05 09:32:47,002 INFO  plugin.PluginRepository - Plugins: looking in: /ok/appl/nutch-2008-09-04_04-01-27/plugins
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository - Plugin Auto-activation mode: [true]
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository - Registered Plugins:
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository -         CyberNeko HTML Parser (lib-nekohtml)
2008-09-05 09:32 :47,305 INFO  plugin.PluginRepository -         MSPowerPoint Parse Plug-in (parse-mspowerpoint)
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository -         Site Query Filter (query-site)
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository -         Http / Https Protocol Plug-in (protocol-httpclient)
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository -         MSWord Parse Plug-in (parse-msword)
2008-09-05 09:32:47,305 INFO  plugin.PluginRepository -         Basic URL Normalizer (urlnormalizer-basic)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Pass-through URL Normalizer (urlnormalizer-pass)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -  &n bsp;      Html Parse Plug-in (parse-html)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Regex URL Filter Framework (lib-regex-filter)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Basic Indexing Filter (index-basic)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Pdf Parse Plug-in (parse-pdf)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Basic Summarizer Plug-in (summary-basic)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         MSExcel Parse Plug-in (parse-msexcel)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Text Parse Plug-in (parse-text)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Jakarta POI - Java API To Access Microsoft Format Files (lib-jakarta-poi)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Regex URL Filter (urlfilter-regex)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Basic Query Filter (query-basic)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         HTTP Framework (lib-http)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         URL Query Filter (query-url)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Regex URL Normalizer (urlnormalizer-regex)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -        ;  Parse MS Documents Framework (lib-parsems)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Zip Parse Plug-in (parse-zip)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         the nutch core extension points (nutch-extensionpoints)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         OPIC Scoring Plug-in (scoring-opic)
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository - Registered Extension-Points:
2008-09-05 09:32:47,306 INFO  plugin.PluginRepository -         Nutch Summarizer (org.apache.nutch.searcher.Summarizer)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2008-09-05 09:32:47,307 INFO  plugin.PluginR epository -         Nutch Protocol (org.apache.nutch.protocol.Protocol)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch URL Filter (org.apache.nutch.net.URLFilter)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFi lter)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Content Parser (org.apache.nutch.parse.Parser)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)
2008-09-05 09:32:47,307 INFO  plugin.PluginRepository -         Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)
2008-09-05 09:32:47,342 INFO  searcher.NutchBean - opening segments in crawl/segments
2008-09-05 09:32:47,368 INFO  searcher.SummarizerFactory - Using the first summarizer extension found: Basic Summarizer
2008-09-05 09:32:47,371 INFO  searcher.NutchBean - opening linkdb in crawl/li nkdb
2008-09-05 09:32:52,746 INFO  searcher.NutchBean - opening indexes in crawl/indexes
2008-09-05 09:32:52,791 INFO  plugin.PluginRepository - Plugins: looking in: /ok/appl/nutch-2008-09-04_04-01-27/plugins
2008-09-05 09:32:52,999 INFO  plugin.PluginRepository - Plugin Auto-activation mode: [true]
2008-09-05 09:32:52,999 INFO  plugin.PluginRepository - Registered Plugins:
2008-09-05 09:32:52,999 INFO  plugin.PluginRepository -         CyberNeko HTML Parser (lib-nekohtml)



> Subject: Re: Job failed!
> From: [hidden email]
> To: [hidden email]
> Date: Fri, 5 Sep 2008 17:28:47 +0800
>
> Could you show the whole hdaoop.log?
> 在 2008-09-05五的 08:46 +0000,Edward Quick写道:
> > Hi,
> >
> > I ran a crawl last night
> >
> > bin/nutch crawl urls -dir crawl -de pth 10
> >
> > which collected 10612 pages, and then bailed out with the following error:
> >
> > Exception in thread "main" java.io.IOException: Job failed!
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
> > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:552)
> > at org.apache.nutch.crawl.Crawl.main(Crawl.java:122)
> >
> > I checked there was enough space on the box, and there don't appear to be any errors in hadoop.log or the crawl output, so I'm stuck on what caused this.
> >
> > Also, is there a way to pick up the crawl from where it stopped rather than having to rerun it all over again?
> >
> > Thanks for any help.
> >
> > Ed.
> >
> >
> >
> > _________________________________________________________________
> > Discover Bird's Eye View now with Multimap from Live Search
> > http://clk.atdmt.com/UKM/go/111354026/direct/01/
>
>


Get Hotmail on your mobile from Vodafone Try it Now

Get Hotmail on your mobile from Vodafone Try it Now!

Win £3000 to spend on whatever you want at Uni! Click here to WIN!

Get Hotmail on your mobile from Vodafone Try it Now!

Try Facebook in Windows Live Messenger! Try it Now!

Get Hotmail on your mobile from Vodafone Try it Now