nutch protocol-file

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

nutch protocol-file

Cam Bazz
Hello,

I wanted to index my files so I followed the instructions at

http://www.folge2.de/tp/search/1/crawling-the-local-filesystem-with-nutch

I get : Exception in thread "main" java.io.IOException: Job failed!

and looking at the log file:

2006-09-01 01:49:43,166 WARN  mapred.LocalJobRunner - job_p2pnnk
java.lang.RuntimeException: No scoring plugins - at least one scoring
plugin is required!
        at
org.apache.nutch.scoring.ScoringFilters.<init>(ScoringFilters.java:84)
        at
org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:57)
        at org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)
        at org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:125)
        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:90)


my plugin.includes is like:

<property>
  <name>plugin.includes</name>
 
<value>protocol-file|protocol-http|urlfilter-regex|parse-(xml|text|html|js|pdf)|index-basic|query-(basic|site|url)</value>
</property>

how can I add a scoring plugin. by default, we dont have to add a
scoring plugin, so I dont know where to go.

Any ideas appreciated,

Best Regards,
-C.B.



Reply | Threaded
Open this post in threaded view
|

Re: nutch protocol-file

Thomas Delnoij-3
Just add scoring-opic to your plugin.includes in nutch-site.xml.

Rgrds, Thomas

On 9/1/06, Cam Bazz <[hidden email]> wrote:

> Hello,
>
> I wanted to index my files so I followed the instructions at
>
> http://www.folge2.de/tp/search/1/crawling-the-local-filesystem-with-nutch
>
> I get : Exception in thread "main" java.io.IOException: Job failed!
>
> and looking at the log file:
>
> 2006-09-01 01:49:43,166 WARN  mapred.LocalJobRunner - job_p2pnnk
> java.lang.RuntimeException: No scoring plugins - at least one scoring
> plugin is required!
>         at
> org.apache.nutch.scoring.ScoringFilters.<init>(ScoringFilters.java:84)
>         at
> org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:57)
>         at org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
>         at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)
>         at org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:125)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:90)
>
>
> my plugin.includes is like:
>
> <property>
>   <name>plugin.includes</name>
>
> <value>protocol-file|protocol-http|urlfilter-regex|parse-(xml|text|html|js|pdf)|index-basic|query-(basic|site|url)</value>
> </property>
>
> how can I add a scoring plugin. by default, we dont have to add a
> scoring plugin, so I dont know where to go.
>
> Any ideas appreciated,
>
> Best Regards,
> -C.B.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: nutch protocol-file

Cam Bazz
Hello,

I almost got it to work, it starts to crawl but after sometime it
finishes. I have something like 200000 little html pages, (2-3k) under
/nutch/data.
I am only getting like 100, or 200 pages, then it stops.

The command I am giving is:

# bin/nutch crawl urls -dir crawl -threads 10 -depth 2

directory urls, contain a urls file, which contains:

file:///nutch/data/

Any Ideas?

-Thanks a bunch.

When I look at the logs, I see:

2006-09-03 18:27:04,231 INFO  indexer.Indexer -  Indexing
[file:///nutch/data] with analyzer
org.apache.nutch.analysis.NutchDocumentAnalyzer@6aba4211 (null)
2006-09-03 18:27:04,381 INFO  indexer.Indexer - maxFieldLength 10000
reached, ignoring following tokens
2006-09-03 18:27:04,454 INFO  indexer.Indexer - Optimizing index.
2006-09-03 18:27:04,529 INFO  indexer.Indexer - merging segments _0 (1
docs) into _1 (1 docs)
2006-09-03 18:27:05,442 INFO  indexer.Indexer - Indexer: done
2006-09-03 18:27:05,444 INFO  indexer.DeleteDuplicates - Dedup: starting
2006-09-03 18:27:05,459 INFO  indexer.DeleteDuplicates - Dedup: adding
indexes in: crawl/indexes
2006-09-03 18:27:08,042 INFO  indexer.DeleteDuplicates - Dedup: done
2006-09-03 18:27:08,043 INFO  indexer.IndexMerger - Adding
crawl/indexes/part-00000
2006-09-03 18:27:08,080 INFO  crawl.Crawl - crawl finished: crawl
2006-09-03 18:28:07,452 INFO  crawl.CrawlDbReader - CrawlDb statistics
start: crawl/crawldb
2006-09-03 18:28:09,387 INFO  crawl.CrawlDbReader - Statistics for
CrawlDb: crawl/crawldb
2006-09-03 18:28:09,387 INFO  crawl.CrawlDbReader - TOTAL urls: 101
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - avg score:  1.008
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - max score:  1.009
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - min score:  1.0
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - retry 0:    1
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - retry 1:    100
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - status 1
(DB_unfetched):    100
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - status 2
(DB_fetched):      1
2006-09-03 18:28:09,388 INFO  crawl.CrawlDbReader - CrawlDb statistics: done



Thomas Delnoij wrote:

> Just add scoring-opic to your plugin.includes in nutch-site.xml.
>
> Rgrds, Thomas
>
> On 9/1/06, Cam Bazz <[hidden email]> wrote:
>> Hello,
>>
>> I wanted to index my files so I followed the instructions at
>>
>> http://www.folge2.de/tp/search/1/crawling-the-local-filesystem-with-nutch 
>>
>>
>> I get : Exception in thread "main" java.io.IOException: Job failed!
>>
>> and looking at the log file:
>>
>> 2006-09-01 01:49:43,166 WARN  mapred.LocalJobRunner - job_p2pnnk
>> java.lang.RuntimeException: No scoring plugins - at least one scoring
>> plugin is required!
>>         at
>> org.apache.nutch.scoring.ScoringFilters.<init>(ScoringFilters.java:84)
>>         at
>> org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:57)
>>         at
>> org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
>>         at
>> org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)
>>         at
>> org.apache.hadoop.mapred.JobConf.newInstance(JobConf.java:443)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:125)
>>         at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:90)
>>
>>
>> my plugin.includes is like:
>>
>> <property>
>>   <name>plugin.includes</name>
>>
>> <value>protocol-file|protocol-http|urlfilter-regex|parse-(xml|text|html|js|pdf)|index-basic|query-(basic|site|url)</value>
>>
>> </property>
>>
>> how can I add a scoring plugin. by default, we dont have to add a
>> scoring plugin, so I dont know where to go.
>>
>> Any ideas appreciated,
>>
>> Best Regards,
>> -C.B.
>>
>>
>>
>>
>