EOF exception while fetching

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

EOF exception while fetching

Ned Rockson-3
Lately I've been getting this error while running Fetcher2:

java.io.EOFException at
java.io.DataInputStream.readFully(DataInputStream.java:178) at
java.io.DataInputStream.readFully(DataInputStream.java:152) at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1383) at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1360) at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1283) at
org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:2866)
at
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2683)
at
org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2392)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:552)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:607)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193) at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

I'm using trunk and thus hadoop 0.15.0.  All of my nodes have the same
number of map/reduce tasks set.  This only happens on about 1/10 of the
nodes during a 100M page crawl and once the job restarts it finishes
most of the time.  Could this be a problem with HDFS or is it a config
that I should modify?

Thanks,
Ned