Hadoop Nutch Performance problem

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Hadoop Nutch Performance problem

Volkan Ebil


 I have got a cluster with 3 machines : 1 Master and 2 slaves


I set -the mapred max tasks 5


       -mapred map tasks 17 and reduce tasks 2


 I start crawl with depth 2 topN 2 but it runs approximately 25 minute.


I start a local crawl with 1 computer  and it finishes in 2 minutes .


 The difference is very big .Is it normal or am i wrong in configurations.


 I tryed with different settings of map and reduce tasks but i have observed
that these numbers


don't have direct impact on speed  


 Help needed urgently.