Index Splitter

[Giovanni Fernandez-Kincade]

Hi,
I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?

This seems promising but I'm not sure if there is anything more mature out there:
http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html

Thanks,
Gio.

[Koji Sekiguchi]

Giovanni Fernandez-Kincade wrote:

> Hi,
> I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?
>
> This seems promising but I'm not sure if there is anything more mature out there:
> http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html
>
> Thanks,
> Gio.
>
>  

There are IndexSplitter and MultiPassIndexSplitter tools in 3.0.

https://issues.apache.org/jira/browse/LUCENE-1959

I'd written an article about them before:

http://lucene.jugem.jp/?eid=344

It is Japanese but I think you can read out how to use them from command
lines...

Koji

--
http://www.rondhuit.com/en/

[Giovanni Fernandez-Kincade]

You can't really use this if you have an optimized index, right?

-----Original Message-----
From: Koji Sekiguchi [mailto:[hidden email]]
Sent: Tuesday, November 24, 2009 6:57 PM
To: [hidden email]
Subject: Re: Index Splitter

Giovanni Fernandez-Kincade wrote:

> Hi,
> I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?
>
> This seems promising but I'm not sure if there is anything more mature out there:
> http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html
>
> Thanks,
> Gio.
>
>  

There are IndexSplitter and MultiPassIndexSplitter tools in 3.0.

https://issues.apache.org/jira/browse/LUCENE-1959

I'd written an article about them before:

http://lucene.jugem.jp/?eid=344

It is Japanese but I think you can read out how to use them from command
lines...

Koji

--
http://www.rondhuit.com/en/

[Koji Sekiguchi]

Giovanni Fernandez-Kincade wrote:
> You can't really use this if you have an optimized index, right?
>
>  
For optimized index, I think you can use MultiPassIndexSplitter.

Koji

--
http://www.rondhuit.com/en/

[Andrzej Bialecki]

Koji Sekiguchi wrote:
> Giovanni Fernandez-Kincade wrote:
>> You can't really use this if you have an optimized index, right?
>>
>>  
> For optimized index, I think you can use MultiPassIndexSplitter.

Correct - MultiPassIndexSplitter can handle any index - optimized or
not, with or without deletions, etc. The cost for this flexibility is
that it needs to read index files multiple times (hence "multi-pass").



--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com