Creating and updating indexes

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Creating and updating indexes

Jacob Brunson
I'm using the latest checkout
( rev 400163) with
Java 1.5.

I've been trying to follow the tutorial for whole web crawling which
is found at

For the most part, I get through the tutorial just fine, although I
did have to look at the latest command line options for the
invertlinks and index commands to make it work.

Now if I want to update my index, I do: generate, fetch, updatedb,
invertlinks, index.  But I can't run the index command a second time
because it complains that the indexes directory already exists.  How
am I suppose to update the index?

Also, in other tutorials I see a dedup command.  What is this used for
and when should I use it?

Thank you so much for your help.