Multiple Indexes - Merge?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple Indexes - Merge?

João Rodrigues-2
Hello all! I am new to Lucene and I'm using PyLucene's extension in my
project. I'm using it to index a large volume of data (index size estimated
in 75GB) over the web and as my internet connection isn't that good enough,
I am asking you guys, probably more experienced than I am, if there is a way
of, if the index is cut short (my script, if connection fails, will optimize
and close the index, just as if it was done), to start from where it ended.
Another option was to merge several "little" indexes.

Hope you can help me out on this one.

Thanks in advance

João Rodrigues
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Indexes - Merge?

hossman

: in 75GB) over the web and as my internet connection isn't that good enough,
: I am asking you guys, probably more experienced than I am, if there is a way
: of, if the index is cut short (my script, if connection fails, will optimize
: and close the index, just as if it was done), to start from where it ended.
: Another option was to merge several "little" indexes.

Well.. the short answer is "yes" you can merge indexes ... the
IndexReader.addIndexes method will do this for you.  In general though,
your problem statement doesn't seem to really relate to Lucene, or require
merging of indexes ...if your app encounters a problem unrelated to
luncene (ie: a network problem) and closes the index cleanly, then when
the problem goes away your app can open an IndexWriter on the index and
start right back up where it left off.




-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Multiple Indexes - Merge?

João Rodrigues-2
Alright, thanks for the answers. I'd tried the pylucene lists and they've
directed me here. I'm sorry for the trouble.

Again, thanks for the quick reply, it'll be useful indeed :)

João
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Indexes - Merge?

hossman

: Alright, thanks for the answers. I'd tried the pylucene lists and they've
: directed me here. I'm sorry for the trouble.

No trouble at all ... I just wanted to make sure you were aware of all the
resources available to you.


-Hoss