Limiting the total number of urls to crawl on a single website

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Limiting the total number of urls to crawl on a single website

Vlador
Is there any way to limit total number of urls per site to crawl (under Nutch 0.8) ?
"generate.max.per.host" setting doesn't seem to work as it is supposed to