[lucy-dev] [GitHub] lucy pull request #54: Improve locking code (v2)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-dev] [GitHub] lucy pull request #54: Improve locking code (v2)

s1monw
GitHub user nwellnhof opened a pull request:

    https://github.com/apache/lucy/pull/54

    Improve locking code (v2)

    [Diff to previous pull request](https://gist.github.com/nwellnhof/f262bcd53c3d69281748571410a0bdbd).
   
    Fixes [LUCY-323](https://issues.apache.org/jira/browse/LUCY-323).
   
    - Major rewrite of FilePurger, making it do less work.
    - Eliminate global deletion lock, making PolyReader and FilePurger never wait for a lock.
    - Redesign Lock API in preparation for native locks.
    - Remove LockFactory.
    - Fix locking bug with non-default merge lock timeout.
    - Fix locking bug with concurrent BackgroundMergers.
    - Port some tests to C.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nwellnhof/lucy improve-locking-v2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/lucy/pull/54.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #54
   
----
commit cb6004330a8e211c2944bb470bdea936d9d69feb
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-15T04:10:07Z

    Separate purging of snapshots and aborted merges
   
    Purge aborted merge when starting an Indexer/BGMerger session and
    purge snapshots at the end of Commit.
   
    Rework FilePurger to use a single hash for purged and spared files
    when purging snapshots. Optimize Folder handling assuming that
    Snapshots only contain entries local to the index folder.
   
    Make sure to list the contents of the real directory, not the
    virtual compound file directory. CFReaderDirHandle lists both real
    and virtual files. Close the CFReader before deleting a segment to
    allow immediate deletion on Windows.
   
    Port BGMerger tests to C.
   
    TODO: Move Perl BGMerge test to t/binding.

commit 38d296557923e07c3717b54cf271f43daa55dba2
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T14:58:45Z

    Merge SharedLock into LockFileLock
   
    Supplying the lock type when requesting a lock better matches other
    file lock APIs. Merging the implementations allows to convert
    Maybe_Delete_File into a static function.
   
    Make shared and exclusive locks check for locks of the other type
    except for exclusive locks created with `exclusive_only`.
   
    Clear_Stale is broken for shared locks but will be removed with the
    following commit.

commit a99bbdb44a67240f8759e9e38a3e607e657dbafc
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T15:47:03Z

    Always check for stale locks in Is_Locked
   
    Note that checking for stale locks isn't necessary when requesting
    shared locks.

commit d9c661d070e34a5150e0d4513d01575bbc9d6ccf
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T16:47:33Z

    Remove LockFactory and make Lock a private class
   
    If there are users who really use custom locks, this can be reverted.

commit 09e8393bf60fbe211cd19ec0c7bfb8be5041f86e
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T12:53:59Z

    Don't allow double obtain/release

commit 571bb9b771acc1f82dc31925d90e143d35f9dd4c
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T16:19:43Z

    Rework snapshot and deletion locks
   
    Request exclusive or shared snapshot locks instead of the global
    deletion lock. This removes the need to retry lock requests and
    reduces contention.

commit 53cbc1088e816229a4c106d8eefd24e189c0a6cb
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T17:25:35Z

    Improve merge lock handling
   
    Don't check the merge lock with Is_Locked in
    FilePurger_Purge_Aborted_Merge. This prevented background mergers from
    cleaning up because they already held the merge lock. Require that
    callers hold the merge lock instead.
   
    Don't check the merge lock with Is_Locked in Indexer_init. Since we
    hold the write lock, and the merge.json file from aborted merge has
    been removed, the presence of a merge.json file is a reliable indicator
    that it belongs to an active background merger. Move the merge lock
    request form S_maybe_merge to Indexer_init.
   
    Also, S_maybe_merge used to Obtain the merge lock instead of merely
    Requesting it. Waiting for a potentially long-running background merger
    slows things down unnecessarily and increases the chance for write lock
    contention. This didn't affect the default configuration which has a
    zero merge lock timeout.

commit 1924910df5d7629aed736d6a2766cb6650ab432a
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T17:34:31Z

    Change lock acquitision order in BGMerger_init
   
    Obtain the merge lock before the write lock so that an already running
    background merger won't abort when failing to acquire the write lock
    during commit.
   
    Release the merge lock at the very end of BGMerger_Commit. It doesn't
    really matter when we release the merge lock after we acquired the
    write lock in BGMerger_Prepare_Commit. I think it makes the code
    clearer to release it at the very end of BGMerger_Commit.

commit 7c7b560388e97b778ddeb83ef46fd3ff4ac6b548
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T18:13:59Z

    Remove Lock_Is_Locked

commit 0674cdd78844152bef89c15898f4550b7dc2d1d8
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T19:26:13Z

    Don't hide I/O errors behind LockErr
   
    Only return a LockErr if there's lock contention, not on unrelated
    I/O errors.

commit 7232b5807cc97b14335ddd158ff33b32f034322e
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T20:38:56Z

    Port LockFileLock tests to C

commit f7dd386e06f327a12b282a6e5bb78f3b2c6b7b82
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T13:09:02Z

    Release locks on destruction

commit 240a1fbfa2fa88048d17daca57ef47f786cea092
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T15:21:00Z

    Move ivars from Lock to LockFileLock

commit 87652fb7ad7e363e41f4f08dd38ead57cbc24a1f
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T16:01:50Z

    Move LockFileLock to separate file

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[lucy-dev] [GitHub] lucy pull request #54: Improve locking code (v2)

s1monw
Github user nwellnhof closed the pull request at:

    https://github.com/apache/lucy/pull/54


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---