[lucy-user] Parallel indexing in same index dir using Lucy

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-user] Parallel indexing in same index dir using Lucy

Shahab Mohammed
Hi,

I am getting lock exception message when trying to run two parallel
processes to index two documents at same index location. As i understand
this is the default behaviour.

Please advice if it will be possible to index two document at same time in
parallel using same index location.

*Kind Regards*
*Shahab*
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Parallel indexing in same index dir using Lucy

Peter Karman
On 9/22/14, 6:13 AM, Shahab Mohammed wrote:
> Hi,
>
> I am getting lock exception message when trying to run two parallel
> processes to index two documents at same index location. As i understand
> this is the default behaviour.
>
> Please advice if it will be possible to index two document at same time in
> parallel using same index location.
>

Only one Indexer object can hold the lock on a single index at a time.

https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Index/Indexer.pod

"In general, only one Indexer at a time may write to an index safely. If
a write lock cannot be secured, new() will throw an exception."

If you want to work in parallel, try creating one-index-per-indexer and
then search them all with PolySearcher:

https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Search/PolySearcher.pod

Or consider the background merging idiom, as here:
https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Docs/Cookbook/FastUpdates.pod

--
Peter Karman  .  http://peknet.com/  .  [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Parallel indexing in same index dir using Lucy

Shahab Mohammed
In reply to this post by Shahab Mohammed
Dear Peter,

Thank you very much for the valueable advice.

*Kind Regards*
*Shahab*
---------------------------------------------------------------------------------

On Mon, Sep 22, 2014 at 7:28 PM, Peter Karman <[hidden email]> wrote:

> On 9/22/14, 6:13 AM, Shahab Mohammed wrote:
> > Hi,
> >
> > I am getting lock exception message when trying to run two parallel
> > processes to index two documents at same index location. As i understand
> > this is the default behaviour.
> >
> > Please advice if it will be possible to index two document at same time
> in
> > parallel using same index location.
> >
>
> Only one Indexer object can hold the lock on a single index at a time.
>
> https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Index/Indexer.pod
>
> "In general, only one Indexer at a time may write to an index safely. If
> a write lock cannot be secured, new() will throw an exception."
>
> If you want to work in parallel, try creating one-index-per-indexer and
> then search them all with PolySearcher:
>
> https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Search/PolySearcher.pod
>
> Or consider the background merging idiom, as here:
>
> https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Docs/Cookbook/FastUpdates.pod
>
> --
> Peter Karman  .  http://peknet.com/  .  [hidden email]
>