Interrupting IndexWriter#close(true)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Interrupting IndexWriter#close(true)

Mark Miller-3
Currently, interrupting IndexWriter#close(true) - which should wait for background merges to finish - causes #close to return while background merges are still running.

There is a test that fails occasionally because of this. This is because it's using logic like this:

try {
  IndexWriter#close(true)
} finally {
  Directory#close();
}

Background merges will still try and use that directory though. Part of wanting to wait for them to finish is so that you know when it's safe to close the directory.

So what is the proper way to address this?

Merge threads that throw an exception - even if it's harmless - will fail the test framework.

Should Lucene act differently? Should #close not be interruptible? Should it make sure merge threads are killed before it bails on interrupt?

Or should the test framework be able to ignore some background thread exceptions?

Or is there some option I'm missing?

- Mark Miller
lucidimagination.com












---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Interrupting IndexWriter#close(true)

Uwe Schindler
I think in tests we should always wait for the merges on close, so don't
pass true. That's the easiest to fix, would it be too complicated to make
the crazy tests I disabled and you reenabled silent by this?

If the code inside Solr uses this IW#close(true), we should add some other
workaround, but that is a different story.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]


> -----Original Message-----
> From: Mark Miller [mailto:[hidden email]]
> Sent: Sunday, July 22, 2012 7:16 PM
> To: [hidden email]
> Subject: Interrupting IndexWriter#close(true)
>
> Currently, interrupting IndexWriter#close(true) - which should wait for
> background merges to finish - causes #close to return while background
> merges are still running.
>
> There is a test that fails occasionally because of this. This is because
it's using
> logic like this:
>
> try {
>   IndexWriter#close(true)
> } finally {
>   Directory#close();
> }
>
> Background merges will still try and use that directory though. Part of
wanting
> to wait for them to finish is so that you know when it's safe to close the
> directory.
>
> So what is the proper way to address this?
>
> Merge threads that throw an exception - even if it's harmless - will fail
the test
> framework.
>
> Should Lucene act differently? Should #close not be interruptible? Should
it

> make sure merge threads are killed before it bails on interrupt?
>
> Or should the test framework be able to ignore some background thread
> exceptions?
>
> Or is there some option I'm missing?
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional
> commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Interrupting IndexWriter#close(true)

Uwe Schindler
Sorry I mean we should do IndexWriter.close(true) in tests and not interrupt
the threads. When do we do this?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]


> -----Original Message-----
> From: Uwe Schindler [mailto:[hidden email]]
> Sent: Sunday, July 22, 2012 7:37 PM
> To: [hidden email]
> Subject: RE: Interrupting IndexWriter#close(true)
>
> I think in tests we should always wait for the merges on close, so don't
pass
> true. That's the easiest to fix, would it be too complicated to make the
crazy

> tests I disabled and you reenabled silent by this?
>
> If the code inside Solr uses this IW#close(true), we should add some other
> workaround, but that is a different story.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
>
> > -----Original Message-----
> > From: Mark Miller [mailto:[hidden email]]
> > Sent: Sunday, July 22, 2012 7:16 PM
> > To: [hidden email]
> > Subject: Interrupting IndexWriter#close(true)
> >
> > Currently, interrupting IndexWriter#close(true) - which should wait
> > for background merges to finish - causes #close to return while
> > background merges are still running.
> >
> > There is a test that fails occasionally because of this. This is
> > because
> it's using
> > logic like this:
> >
> > try {
> >   IndexWriter#close(true)
> > } finally {
> >   Directory#close();
> > }
> >
> > Background merges will still try and use that directory though. Part
> > of
> wanting
> > to wait for them to finish is so that you know when it's safe to close
> > the directory.
> >
> > So what is the proper way to address this?
> >
> > Merge threads that throw an exception - even if it's harmless - will
> > fail
> the test
> > framework.
> >
> > Should Lucene act differently? Should #close not be interruptible?
> > Should
> it
> > make sure merge threads are killed before it bails on interrupt?
> >
> > Or should the test framework be able to ignore some background thread
> > exceptions?
> >
> > Or is there some option I'm missing?
> >
> > - Mark Miller
> > lucidimagination.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email] For
> > additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional
> commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Interrupting IndexWriter#close(true)

Mark Miller-3
In reply to this post by Mark Miller-3
https://issues.apache.org/jira/browse/LUCENE-4245 tracks this issue.

On Jul 22, 2012, at 1:15 PM, Mark Miller wrote:

> Currently, interrupting IndexWriter#close(true) - which should wait for background merges to finish - causes #close to return while background merges are still running.
>
> There is a test that fails occasionally because of this. This is because it's using logic like this:
>
> try {
>  IndexWriter#close(true)
> } finally {
>  Directory#close();
> }
>
> Background merges will still try and use that directory though. Part of wanting to wait for them to finish is so that you know when it's safe to close the directory.
>
> So what is the proper way to address this?
>
> Merge threads that throw an exception - even if it's harmless - will fail the test framework.
>
> Should Lucene act differently? Should #close not be interruptible? Should it make sure merge threads are killed before it bails on interrupt?
>
> Or should the test framework be able to ignore some background thread exceptions?
>
> Or is there some option I'm missing?
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>

- Mark Miller
lucidimagination.com












---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]