Recrawling some pages much more often than others.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Recrawling some pages much more often than others.

Marcin Okraszewski-3
In my search sites there are several pages that change very often. New links at the pages are added every minute. If I won't recrawl those frequently updated pages every 5 minutes, some links will be lost.

Other sites and pages do not change so often. They can be rechecked every several days or a week. Doesn't matter that much.

Is there any way of making Nutch to recrawl the frequently updated pages every 5 minutes, and go to others rarely?

Thanks for help.
Marcin Okraszewski