Re: Error on convert to 0.9 during mergesegs step

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Error on convert to 0.9 during mergesegs step

Alan Tanaman
I'm getting a similar problem with 0.9, but during the injector:

2007-01-02 21:30:25,797 INFO  crawl.Injector - Injector: Merging injected
urls into crawl db.

2007-01-02 21:30:25,906 WARN  util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable

2007-01-02 21:34:38,078 INFO  crawl.Injector - Injector: done

 

It doesn't crash, simply doesn't seem to get any injected urls at all.  I'm
using the same filters settings as for 0.8.1

I've set mapred.speculative.execution to false to no avail.

 

Best regards,

Alan

_________________________

Alan Tanaman

iDNA Solutions

 

Reply | Threaded
Open this post in threaded view
|

Re: Error on convert to 0.9 during mergesegs step

Andrzej Białecki-2
Alan Tanaman wrote:

> I'm getting a similar problem with 0.9, but during the injector:
>
> 2007-01-02 21:30:25,797 INFO  crawl.Injector - Injector: Merging injected
> urls into crawl db.
>
> 2007-01-02 21:30:25,906 WARN  util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
>
> 2007-01-02 21:34:38,078 INFO  crawl.Injector - Injector: done
>
>  
>
> It doesn't crash, simply doesn't seem to get any injected urls at all.  I'm
> using the same filters settings as for 0.8.1
>  

Please note that recent versions of trunk/ are incompatible with 0.8.x,
and with earlier versions of trunk// - see the note 17. in CHANGES.txt.
You should also temporarily increase your logging level to DEBUG to see
if there are any problems reported at low level.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

RE: Error on convert to 0.9 during mergesegs step

Alan Tanaman
Thanks for pointing that out Andrzej.  I think that the problem is actually
that our native Java Nutch launcher needs to be rebuilt with the Hadoop 0.9
jar.

Best regards,
Alan
_________________________
Alan Tanaman
iDNA Solutions

-----Original Message-----
From: Andrzej Bialecki [mailto:[hidden email]]
Sent: 02 January 2007 21:52
To: [hidden email]
Subject: Re: Error on convert to 0.9 during mergesegs step

Alan Tanaman wrote:

> I'm getting a similar problem with 0.9, but during the injector:
>
> 2007-01-02 21:30:25,797 INFO  crawl.Injector - Injector: Merging
> injected urls into crawl db.
>
> 2007-01-02 21:30:25,906 WARN  util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
>
> 2007-01-02 21:34:38,078 INFO  crawl.Injector - Injector: done
>
>  
>
> It doesn't crash, simply doesn't seem to get any injected urls at all.  
> I'm using the same filters settings as for 0.8.1
>  

Please note that recent versions of trunk/ are incompatible with 0.8.x, and
with earlier versions of trunk// - see the note 17. in CHANGES.txt.
You should also temporarily increase your logging level to DEBUG to see if
there are any problems reported at low level.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web ___|||__||  \|
||  |  Embedded Unix, System Integration http://www.sigram.com  Contact:
info at sigram dot com