Race condition between IndexWriter.commit and IndexWriter.close

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Race condition between IndexWriter.commit and IndexWriter.close

Boris Petrov-2
Hi all,

We're getting the following exception:

java.lang.IllegalStateException: cannot close: prepareCommit was already
called with no corresponding call to commit
    at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1025)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
    ...

We are only calling "commit" on IndexWriter. By reading the code I can
see that if you call IndexWriter.commit in parallel to
IndexWriter.close, it is possible to get this exception. More
specifically, after setting "IndexWriter.pendingCommit" on line 4779
(this is using Lucene 7.5.0) and before setting it to "null" on line
4793 this problem could happen.

Is this by design or is it a bug?

Thanks,

Boris Petrov



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Race condition between IndexWriter.commit and IndexWriter.close

Michael McCandless-2
I think if you call commit and close concurrently the results are undefined
and so this is acceptable.

Mike

On Thu, Nov 29, 2018 at 5:53 AM Boris Petrov <[hidden email]>
wrote:

> Hi all,
>
> We're getting the following exception:
>
> java.lang.IllegalStateException: cannot close: prepareCommit was already
> called with no corresponding call to commit
>     at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1025)
>     at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
>     ...
>
> We are only calling "commit" on IndexWriter. By reading the code I can
> see that if you call IndexWriter.commit in parallel to
> IndexWriter.close, it is possible to get this exception. More
> specifically, after setting "IndexWriter.pendingCommit" on line 4779
> (this is using Lucene 7.5.0) and before setting it to "null" on line
> 4793 this problem could happen.
>
> Is this by design or is it a bug?
>
> Thanks,
>
> Boris Petrov
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> --
Mike McCandless

http://blog.mikemccandless.com
Reply | Threaded
Open this post in threaded view
|

Re: Race condition between IndexWriter.commit and IndexWriter.close

Boris Petrov-2
So you're saying that this race-condition is OK? Nowhere in the
documentation does it say that these two calls should be synchronized...
at least that must be fixed. :)

On 12/1/18 6:25 PM, Michael McCandless wrote:

> I think if you call commit and close concurrently the results are undefined
> and so this is acceptable.
>
> Mike
>
> On Thu, Nov 29, 2018 at 5:53 AM Boris Petrov <[hidden email]>
> wrote:
>
>> Hi all,
>>
>> We're getting the following exception:
>>
>> java.lang.IllegalStateException: cannot close: prepareCommit was already
>> called with no corresponding call to commit
>>     at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1025)
>>     at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
>>     ...
>>
>> We are only calling "commit" on IndexWriter. By reading the code I can
>> see that if you call IndexWriter.commit in parallel to
>> IndexWriter.close, it is possible to get this exception. More
>> specifically, after setting "IndexWriter.pendingCommit" on line 4779
>> (this is using Lucene 7.5.0) and before setting it to "null" on line
>> 4793 this problem could happen.
>>
>> Is this by design or is it a bug?
>>
>> Thanks,
>>
>> Boris Petrov
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>> --
> Mike McCandless
>
> http://blog.mikemccandless.com
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Race condition between IndexWriter.commit and IndexWriter.close

Michael McCandless-2
Yeah I agree we should fix the javadocs to say that you should not try to
close and commit concurrently from different threads.

Wanna open a quick PR or issue with a patch?

Thanks,

Mike

On Wed, Dec 5, 2018, 4:06 AM Boris Petrov <[hidden email] wrote:

> So you're saying that this race-condition is OK? Nowhere in the
> documentation does it say that these two calls should be synchronized...
> at least that must be fixed. :)
>
> On 12/1/18 6:25 PM, Michael McCandless wrote:
> > I think if you call commit and close concurrently the results are
> undefined
> > and so this is acceptable.
> >
> > Mike
> >
> > On Thu, Nov 29, 2018 at 5:53 AM Boris Petrov <[hidden email]>
> > wrote:
> >
> >> Hi all,
> >>
> >> We're getting the following exception:
> >>
> >> java.lang.IllegalStateException: cannot close: prepareCommit was already
> >> called with no corresponding call to commit
> >>     at
> org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1025)
> >>     at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
> >>     ...
> >>
> >> We are only calling "commit" on IndexWriter. By reading the code I can
> >> see that if you call IndexWriter.commit in parallel to
> >> IndexWriter.close, it is possible to get this exception. More
> >> specifically, after setting "IndexWriter.pendingCommit" on line 4779
> >> (this is using Lucene 7.5.0) and before setting it to "null" on line
> >> 4793 this problem could happen.
> >>
> >> Is this by design or is it a bug?
> >>
> >> Thanks,
> >>
> >> Boris Petrov
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >> --
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Race condition between IndexWriter.commit and IndexWriter.close

Boris Petrov-2
I just started doing that and noticed that in "close"'s documentation it
says:

NOTE: You must ensure no other threads are still making changes at the
same time that this method is invoked.

Which I guess is what I've missed and should be enough. Thanks for the
support! :)


On 12/5/18 12:52 PM, Michael McCandless wrote:

> Yeah I agree we should fix the javadocs to say that you should not try
> to close and commit concurrently from different threads.
>
> Wanna open a quick PR or issue with a patch?
>
> Thanks,
>
> Mike
>
> On Wed, Dec 5, 2018, 4:06 AM Boris Petrov <[hidden email]
> <mailto:[hidden email]> wrote:
>
>     So you're saying that this race-condition is OK? Nowhere in the
>     documentation does it say that these two calls should be
>     synchronized...
>     at least that must be fixed. :)
>
>     On 12/1/18 6:25 PM, Michael McCandless wrote:
>     > I think if you call commit and close concurrently the results
>     are undefined
>     > and so this is acceptable.
>     >
>     > Mike
>     >
>     > On Thu, Nov 29, 2018 at 5:53 AM Boris Petrov
>     <[hidden email] <mailto:[hidden email]>>
>     > wrote:
>     >
>     >> Hi all,
>     >>
>     >> We're getting the following exception:
>     >>
>     >> java.lang.IllegalStateException: cannot close: prepareCommit
>     was already
>     >> called with no corresponding call to commit
>     >>     at
>     org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1025)
>     >>     at
>     org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
>     >>     ...
>     >>
>     >> We are only calling "commit" on IndexWriter. By reading the
>     code I can
>     >> see that if you call IndexWriter.commit in parallel to
>     >> IndexWriter.close, it is possible to get this exception. More
>     >> specifically, after setting "IndexWriter.pendingCommit" on line
>     4779
>     >> (this is using Lucene 7.5.0) and before setting it to "null" on
>     line
>     >> 4793 this problem could happen.
>     >>
>     >> Is this by design or is it a bug?
>     >>
>     >> Thanks,
>     >>
>     >> Boris Petrov
>     >>
>     >>
>     >>
>     >>
>     ---------------------------------------------------------------------
>     >> To unsubscribe, e-mail: [hidden email]
>     <mailto:[hidden email]>
>     >> For additional commands, e-mail:
>     [hidden email]
>     <mailto:[hidden email]>
>     >>
>     >> --
>     > Mike McCandless
>     >
>     > http://blog.mikemccandless.com
>     >
>