[jira] Created: (LUCENE-1877) Improve IndexWriter javadoc on locking

classic Classic list List threaded Threaded
53 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
Improve IndexWriter javadoc on locking
--------------------------------------

                 Key: LUCENE-1877
                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Javadocs
            Reporter: Mark Miller
            Priority: Trivial
             Fix For: 2.9


A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.


{code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
  another <code>IndexWriter</code> on the same directory will lead to a
  {@link LockObtainFailedException}. The {@link LockObtainFailedException}
  is also thrown if an IndexReader on the same directory is used to delete documents
  from the index.</p>{code}

Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749301#action_12749301 ]

Uwe Schindler commented on LUCENE-1877:
---------------------------------------

For IndexWriter/IndexReader this hint is no longer needed (in Lucene 2.9), as all methods taking String/File instead of Directory are deprecated and users should create directory instances and then will automatically get to the place where the LockFactory can be supplied.

The note should be added to FSDirectory instead.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749303#action_12749303 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

My initial thought was also that it didn't really belong in IndexWriter - but I sold myself on the fact that IndexWriter talks about locking and offers the force unlock method - so it seems fine to me to mention how to use the optimal locking factory (and generally avoid using the force unlock at all - as an aside I just saw a guy trying to use that the other day as regular code so that they could use two IndexWriters with just commit rather than close - ugg).

I'm not sold either way though - I'd go with whatever. My preference would really be to make it the default.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749303#action_12749303 ]

Mark Miller edited comment on LUCENE-1877 at 8/30/09 11:53 AM:
---------------------------------------------------------------

My initial thought was also that it didn't really belong in IndexWriter - but I sold myself on the fact that IndexWriter talks about locking and offers the force unlock method - so it seems fine to me to mention how to use the optimal locking factory (and generally avoid using the force unlock at all - as an aside I just saw a guy trying to use that the other day as regular code so that they could use two IndexWriters with just commit rather than close - ugg).

I'm not sold either way though - I'd go with whatever. My preference would really be to make it the default (though of course not for 2.9).

      was (Author: [hidden email]):
    My initial thought was also that it didn't really belong in IndexWriter - but I sold myself on the fact that IndexWriter talks about locking and offers the force unlock method - so it seems fine to me to mention how to use the optimal locking factory (and generally avoid using the force unlock at all - as an aside I just saw a guy trying to use that the other day as regular code so that they could use two IndexWriters with just commit rather than close - ugg).

I'm not sold either way though - I'd go with whatever. My preference would really be to make it the default.
 

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749322#action_12749322 ]

Michael McCandless commented on LUCENE-1877:
--------------------------------------------

bq. Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

In my testing (long ago) over NFS, I actually found "native" locks didn't work as well as "simple" locks.  I was also a bit nervous on how well supported "native" locks are across different OSs.

bq. My preference would really be to make it the default (though of course not for 2.9).

+1, I think it's the better default.

People who use Lucene over NFS already have to do special things (eg make a custom deletion policy), and far too many users hit the "leftover lock file" problem.  We could state in the javadocs that this default will change in 3.0?

Maybe just add one sentence in that IndexWriter locking section, referencing the discussion in NativeFSLockFactory's javadocs about not having the "leftover lock file" problem?

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749324#action_12749324 ]

Uwe Schindler commented on LUCENE-1877:
---------------------------------------

Let's do it in the following way:
- deprecated FSDir.getDirectory() methods return the SimpleLockFactory, as it was before.
- The new FSDir.open() methods and also the direct ctors of SimpleFSDir, MMapFSDir, NIOFSDir default to NativeLocakFactory (these ctors/methods are all new in 2.9)

Because of this we have no BW problem.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749330#action_12749330 ]

Marvin Humphrey commented on LUCENE-1877:
-----------------------------------------

> Anyone remember why NativeFSLockFactory is not the default over
> SimpleFSLockFactory?

Wasn't it because native locking is somethings implemented with Fcntl, and
Fcntl locking blows chunks?  Especially for a library rather than an
application.

From the BSD manpage on Fcntl:

{quote}
This interface follows the completely stupid semantics of System V and IEEE
Std 1003.1-1988 (``POSIX.1'') that require that all locks associated with a
file for a given process are removed when any file descriptor for that file is
closed by that process.  This semantic means that applications must be aware
of any files that a subroutine library may access.  For example if an
application for updating the password file locks the password file database
while making the update, and then calls getpwname(3) to retrieve a record, the
lock will be lost because getpwname(3) opens, reads, and closes the password
database.  The database close will release all locks that the process has
associated with the database, even if the library routine never requested a
lock on the database.  Another minor semantic problem with this interface is
that locks are not inherited by a child process created using the fork(2)
function.  The flock(2) interface has much more rational last close
semantics and allows locks to be inherited by child processes.  Flock(2) is
recommended for applications that want to ensure the integrity of their locks
when using library routines or wish to pass locks to their children...
{quote}

The lockfile may be annoying, but at least it's guaranteed safe on all non-shared
volumes when the OS implements atomic file opening.

Are you folks at least able to clean up orphaned lockfiles if the PID it was created
under is no longer active?

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749334#action_12749334 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

{quote}This interface follows the completely stupid semantics of System V and IEEE
Std 1003.1-1988 (``POSIX.1'') that require that all locks associated with a
file for a given process are removed when any file descriptor for that file is
closed by that process. This semantic means that applications must be aware
of any files that a subroutine library may access. For example if an
application for updating the password file locks the password file database
while making the update, and then calls getpwname(3) to retrieve a record, the
lock will be lost because getpwname(3) opens, reads, and closes the password
database. The database close will release all locks that the process has
associated with the database, even if the library routine never requested a
lock on the database. Another minor semantic problem with this interface is
that locks are not inherited by a child process created using the fork(2)
function. The flock(2) interface has much more rational last close
semantics and allows locks to be inherited by child processes. Flock(2) is
recommended for applications that want to ensure the integrity of their locks
when using library routines or wish to pass locks to their children... {quote}

I can see how this is not ideal, but I'm not seeing how any of the mentioned issues apply to our simple lock usage ...

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749336#action_12749336 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

bq. People who use Lucene over NFS already have to do special things (eg make a custom deletion policy), and far too many users hit the "leftover lock file" problem. We could state in the javadocs that this default will change in 3.0?

+1 from me - if it made things work out of the box with NFS, I'd vote to keep as is, but the points you mention were in my head too.

My only worry is current users counting on this default for NFS - but if we put it in the back compat break section (a break in regards to NFS anyway), that should be sufficient warning?

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749363#action_12749363 ]

Marvin Humphrey commented on LUCENE-1877:
-----------------------------------------

> I can see how this is not ideal, but I'm not seeing how any of the
> mentioned issues apply to our simple lock usage ...

"Simple lock usage"?!  You must have a bigger brain than me...

As a matter of fact, I think you're right.   Fcntl locks have two major
drawbacks, and upon review I think NativeFSLockFactory avoids both of them.

The first is that opening and closing a file releases all locks for the entire
process.  Even if you never request a lock on the second filehandle, or if you
request a lock and the request is denied, closing the filehandle releases the
lock on the first filehandle.  But NativeFSLockFactory avoids that problem by
keeping the HashSet of filepaths and ensuring that the same file is never
opened twice.  Furthermore, since the lockfiles are private to Lucene, you can
assume that nobody else is going to open them and inadvertently spoil the lock.

The second is that child processes spawned via fork() do not inherit locks
from the parent process.  If you assume that nobody's ever going to fork a
Java process, that's not relevant.  (Too bad that won't work for Lucy... we
have to support fork().)

I think you're probably safe with Fcntl locks on all non-shared volumes.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749404#action_12749404 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

My brain has never been as large as I'd like it to be, but that's  
never concerned me too greatly - it's my ego that I have the trouble  
with.

- Mark

http://www.lucidimagination.com (mobile)

On Aug 30, 2009, at 11:25 PM, "Marvin Humphrey (JIRA)"  



> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749417#action_12749417 ]

Michael McCandless commented on LUCENE-1877:
--------------------------------------------

{quote}
Let's do it in the following way:
- deprecated FSDir.getDirectory() methods return the SimpleLockFactory, as it was before.
- The new FSDir.open() methods and also the direct ctors of SimpleFSDir, MMapFSDir, NIOFSDir default to NativeLocakFactory (these ctors/methods are all new in 2.9)
{quote}

+1


> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Schindler updated LUCENE-1877:
----------------------------------

    Attachment: LUCENE-1877.patch

Here is a patch, that changes the default for ctor-based / open() based instantiations to use NativeFSLockFactory (in fact, if the supplied param to ctor is NULL). Also change javadocs.

It also deprecates the static FSDirectory.setDisableLocks() [which we should have done already]. One should simple use a ctor/open with NoLockFactory as param to do that.

Currently only TestIndexReader fails here on windows because of some strange lockfile-delete opeartions. Maybe the testcase must be updated. I will look into this.

If we want to go this way, we have to put this in 2.9.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749449#action_12749449 ]

Uwe Schindler commented on LUCENE-1877:
---------------------------------------

I will be happy when 3.0 removes all this FSDirectory deprecated stuff. Its a hell to maintain!

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749459#action_12749459 ]

Michael McCandless commented on LUCENE-1877:
--------------------------------------------

I think we should also fix NativeLockFactory so that if the write lock is in the index dir it doesn't generate the large digest in the file name.  That digest is problematic when two different machines access the same physical dir via different mount names, since that results in different lock file names.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749475#action_12749475 ]

Uwe Schindler commented on LUCENE-1877:
---------------------------------------

With the above patch some more tests are failing, mostly because of the strange lock file names. I think we should fix the tests, at least hardcode the simplelock factory, when it should be tested.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Issue Comment Edited: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749475#action_12749475 ]

Uwe Schindler edited comment on LUCENE-1877 at 8/31/09 5:17 AM:
----------------------------------------------------------------

With the above patch some more tests are failing, mostly because of the strange lock file names. I think we should fix the tests, at least hardcode the simplelock factory, when it should be tested.

The backwards-tests seem to pass, as they only use FSDir.getDirectory() which defaults to old standard. That's good.

      was (Author: thetaphi):
    With the above patch some more tests are failing, mostly because of the strange lock file names. I think we should fix the tests, at least hardcode the simplelock factory, when it should be tested.
 

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749502#action_12749502 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

bq. If we want to go this way, we have to put this in 2.9.

I'd personally be a little (to a lot) afraid to change the default to native during freeze -

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749505#action_12749505 ]

Uwe Schindler commented on LUCENE-1877:
---------------------------------------

It's only the default for new code, clearly documented; deprecated code stays as it is.

If we will not get this into 2.9, 3.0 will remove the deprecated parts and the new code (new in 2.9) will change its defaults.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1877) Improve IndexWriter javadoc on locking

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749506#action_12749506 ]

Mark Miller commented on LUCENE-1877:
-------------------------------------

Oh, okay, cool - that makes me feel a little better.

Though new users seeing it as the default now - thats not the worst situation, but I would almost prefer the change go through a dev cycle as the default.

If others are not feeling as cautious, I wouldn't vote against.

> Improve IndexWriter javadoc on locking
> --------------------------------------
>
>                 Key: LUCENE-1877
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1877
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Javadocs
>            Reporter: Mark Miller
>            Priority: Trivial
>             Fix For: 2.9
>
>         Attachments: LUCENE-1877.patch
>
>
> A user requested we add a note in IndexWriter alerting the availability of NativeFSLockFactory (allowing you to avoid retaining locks on abnormal jvm exit). Seems reasonable to me - we want users to be able to easily stumble upon this class. The below code looks like a good spot to add a note - could also improve whats there a bit - opening an IndexWriter does not necessarily create a lock file - that would depend on the LockFactory used.
> {code}  <p>Opening an <code>IndexWriter</code> creates a lock file for the directory in use. Trying to open
>   another <code>IndexWriter</code> on the same directory will lead to a
>   {@link LockObtainFailedException}. The {@link LockObtainFailedException}
>   is also thrown if an IndexReader on the same directory is used to delete documents
>   from the index.</p>{code}
> Anyone remember why NativeFSLockFactory is not the default over SimpleFSLockFactory?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

123