Problem with partititioning

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with partititioning

karolrybak
I have a problem with partitioning. After selecting phase of generating
which completed succesfully i got a following exception:

Generator: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
Cannot open filename
/tmp/hadoop-nutch/mapred/temp/generate-temp-1194364974836/_task_200711051139_0323_r_000007_0
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at org.apache.hadoop.ipc.Client.call(Client.java:482)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(
DFSClient.java:848)
        at org.apache.hadoop.dfs.DFSClient$DFSInputStream.<init>(
DFSClient.java:840)
        at org.apache.hadoop.dfs.DFSClient.open(DFSClient.java:285)
        at org.apache.hadoop.dfs.DistributedFileSystem.open(
DistributedFileSystem.java:114)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1356)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1349)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1344)
        at org.apache.hadoop.mapred.SequenceFileOutputFormat.getReaders(
SequenceFileOutputFormat.java:87)
        at org.apache.nutch.crawl.Generator.generate(Generator.java:429)
        at org.apache.nutch.crawl.Generator.run(Generator.java:563)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:54)
        at org.apache.nutch.crawl.Generator.main(Generator.java:526)


On my client node in taskctracker log i found a following exception

org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
task_200711051139_0322_m_000064_0/file.out.index in any of the configured
local director
ies
        at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead
(LocalDirAllocator.java:327)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(
LocalDirAllocator.java:138)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(
TaskTracker.java:1923)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java
:427)
        at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
WebApplicationHandler.java:475)
        at org.mortbay.jetty.servlet.ServletHandler.handle(
ServletHandler.java:567)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.jetty.servlet.WebApplicationContext.handle(
WebApplicationContext.java:635)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java
:981)
        at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at org.mortbay.http.SocketListener.handleConnection(
SocketListener.java:244)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)


The weird thing is that second time i run the batch it worked ok, i'm
generating 500 000 urls from a db of about 550 000.

Could it have something to do with open files limit ?

--
Karol Rybak
Programista / Programmer
Sekcja aplikacji / Applications section
Wyższa Szkoła Informatyki i Zarządzania / University of Internet Technology
and Management
+48(17)8661277
Reply | Threaded
Open this post in threaded view
|

Re: Problem with partititioning

karolrybak
I have also noticed that partitioning job generates a large number (150) of
reduce tasks, while i have set them to 9. I have 150 map tasks. Is that
normal or could this be some kind of bug ?
Reply | Threaded
Open this post in threaded view
|

Re: Problem with partititioning

Tomislav Poljak
Hi,
I have the same problem (same exception) after selecting phase of
generating, sometimes it works fine and sometimes this exception occurs.
Why is that and how can I fix it?

Thanks,
     Tomislav
 
On Tue, 2007-11-06 at 15:02 +0100, Karol Rybak wrote:
> ced that partitioning job generates a large number (150) of
> reduce tasks, while i have

Reply | Threaded
Open this post in threaded view
|

Re: Problem with partititioning

Mathijs Homminga
We have had the same problem (I think it is not the partitioning but the
last part of the select which goes wrong). We solved it by turning of
speculative execution.
In hadoop-site.xml:

<property>
  <name>mapred.speculative.execution</name>
  <value>false</value>
  <description>If true, then multiple instances of some map and reduce
tasks
               may be executed in parallel.</description>
</property>

 From the stacktrace, it looks like the tracker tries to copy the output
data from the task which was killed (tip already completed) instead from
the task (same task, other attempt) which completed.  Is this related to
the Generator job in Nutch or an issue in Hadoop?

Mathijs

Tomislav Poljak wrote:

> Hi,
> I have the same problem (same exception) after selecting phase of
> generating, sometimes it works fine and sometimes this exception occurs.
> Why is that and how can I fix it?
>
> Thanks,
>      Tomislav
>  
> On Tue, 2007-11-06 at 15:02 +0100, Karol Rybak wrote:
>  
>> ced that partitioning job generates a large number (150) of
>> reduce tasks, while i have
>>    
>
>  

--
Knowlogy
Helperpark 290 C
9723 ZA Groningen
+31 (0)50 2103567
http://www.knowlogy.nl

[hidden email]
+31 (0)6 15312977