Hadoop Nutch Performance problem

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Hadoop Nutch Performance problem

Volkan Ebil
Hi,

 

 I have got a cluster with 3 machines : 1 Master and 2 slaves

 

I set -the mapred max tasks 5

 

       -mapred map tasks 17 and reduce tasks 2

 

 I start crawl with depth 2 topN 2 but it runs approximately 25 minute.

 

I start a local crawl with 1 computer  and it finishes in 2 minutes .

 

 The difference is very big .Is it normal or am i wrong in configurations.

 

 I tryed with different settings of map and reduce tasks but i have observed
that these numbers

 

don't have direct impact on speed  

 

 Help needed urgently.

 

Thanks