[lucy-dev] [GitHub] lucy pull request #53: Improve locking code

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-dev] [GitHub] lucy pull request #53: Improve locking code

jpountz
GitHub user nwellnhof opened a pull request:

    https://github.com/apache/lucy/pull/53

    Improve locking code

    Fixes [LUCY-323](https://issues.apache.org/jira/browse/LUCY-323).
   
    - Major rewrite of FilePurger, making it do less work.
    - Eliminate global deletion lock, making PolyReader and FilePurger never wait for a lock.
    - Redesign Lock API in preparation for native locks.
    - Remove LockFactory.
    - Fix locking bug with non-default merge lock timeout.
    - Fix locking bug with concurrent BackgroundMergers.
    - Port some tests to C.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nwellnhof/lucy improve-locking

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/lucy/pull/53.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #53
   
----
commit 2051451de456f51e35345f12206ca4cb0040dd7c
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-15T04:10:07Z

    Separate purging of snapshots and aborted merges
   
    Purge aborted merge when starting an Indexer/BGMerger session and
    purge snapshots at the end of Commit.
   
    Rework FilePurger to use a single hash for purged and spared files
    when purging snapshots. Optimize Folder handling assuming that
    Snapshots only contain entries local to the index folder.
   
    Make sure to list the contents of the real directory, not the
    virtual compound file directory. CFReaderDirHandle lists both real
    and virtual files which might be a bug in itself.
   
    Port BGMerger tests to C.
   
    TODO: Move Perl BGMerge test to t/binding.

commit bb83d0d77bf99ed09b2bbad782c3c1d3b94f04a9
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T14:58:45Z

    Merge SharedLock into LockFileLock
   
    Supplying the lock type when requesting a lock better matches other
    file lock APIs. Merging the implementations allows to convert
    Maybe_Delete_File into a static function.
   
    Make shared and exclusive locks check for locks of the other type
    except for exclusive locks created with `exclusive_only`.
   
    Clear_Stale is broken for shared locks but will be removed with the
    following commit.

commit 1a8e02df0a0257e4ba70de4a41504507b0ba8a47
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T15:47:03Z

    Always check for stale locks in Is_Locked
   
    Note that checking for stale locks isn't necessary when requesting
    shared locks.

commit 3e2298650d8fc0213ce25078cadeb4f22d45bb5f
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-16T16:47:33Z

    Remove LockFactory and make Lock a private class
   
    If there are users who really use custom locks, this can be reverted.

commit 79161a4b233ce3570ecf4c48745b469ccb50aec7
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T12:53:59Z

    Don't allow double obtain/release

commit 5270b988c477a2e1c110e2069c66bb2d3f27a60f
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T16:19:43Z

    Rework snapshot and deletion locks
   
    Request exclusive or shared snapshot locks instead of the global
    deletion lock. This removes the need to retry lock requests and
    reduces contention.

commit 4e6e5bbf2814e8e2a824b5ae1b91d068233cd248
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T17:25:35Z

    Improve merge lock handling
   
    Don't check the merge lock with Is_Locked in
    FilePurger_Purge_Aborted_Merge. This prevented background mergers from
    cleaning up because they already held the merge lock. Require that
    callers hold the merge lock instead.
   
    Don't check the merge lock with Is_Locked in Indexer_init. Since we
    hold the write lock, and the merge.json file from aborted merge has
    been removed, the presence of a merge.json file is a reliable indicator
    that it belongs to an active background merger. Move the merge lock
    request form S_maybe_merge to Indexer_init.
   
    Also, S_maybe_merge used to Obtain the merge lock instead of merely
    Requesting it. Waiting for a potentially long-running background merger
    slows things down unnecessarily and increases the chance for write lock
    contention. This didn't affect the default configuration which has a
    zero merge lock timeout.

commit de14d4ca31ab27fc1569966c1b182fa4829afd40
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T17:34:31Z

    Change lock acquitision order in BGMerger_init
   
    Obtain the merge lock before the write lock so that an already running
    background merger won't abort when failing to acquire the write lock
    during commit.
   
    Release the merge lock at the very end of BGMerger_Commit. It doesn't
    really matter when we release the merge lock after we acquired the
    write lock in BGMerger_Prepare_Commit. I think it makes the code
    clearer to release it at the very end of BGMerger_Commit.

commit 38a33badb310ba9fea5bf9a923277da4a82ec369
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T18:13:59Z

    Remove Lock_Is_Locked

commit e06cdbf63d3036c3a33e5265444d827065539e92
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T19:26:13Z

    Don't hide I/O errors behind LockErr
   
    Only return a LockErr if there's lock contention, not on unrelated
    I/O errors.

commit 35388cd22600cb3c202cf11c8c9afd661c8f4fe4
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-17T20:38:56Z

    Port LockFileLock tests to C

commit d23b560dc34d1c1cbc731fdc745b4e72324d8717
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T13:09:02Z

    Release locks on destruction

commit 2bd2bc611325ee643f325ecfd11d5227319949a8
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T15:21:00Z

    Move host ivar from Lock to LockFileLock

commit 6e8538a8834a49da7e8cce8b94decc130467b0f3
Author: Nick Wellnhofer <[hidden email]>
Date:   2017-02-19T16:01:50Z

    Move LockFileLock to separate file

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[lucy-dev] [GitHub] lucy pull request #53: Improve locking code

jpountz
Github user nwellnhof closed the pull request at:

    https://github.com/apache/lucy/pull/53


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---