[jira] Created: (LUCENE-1322) Remove synchronization in CompoundFileReader

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
Remove synchronization in CompoundFileReader
--------------------------------------------

                 Key: LUCENE-1322
                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
    Affects Versions: 2.3.1
            Reporter: Jason Rutherglen
            Priority: Trivial


Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609235#action_12609235 ]

robert engels commented on LUCENE-1322:
---------------------------------------

This comment is very misleading.

Since there is IO involved, any operation can block, which will lower CPU utilization.

You just need to make sure that the locks control different physical resources.

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Priority: Trivial
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1322:
-------------------------------------

    Attachment: lucene-1322.patch

lucene-1322.patch

CSIndexInput.clone() added that clones the base IndexInput removing the synchronization overhead from CompoundFileReader.

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Priority: Trivial
>         Attachments: lucene-1322.patch
>
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless reassigned LUCENE-1322:
------------------------------------------

    Assignee: Michael McCandless

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Assignee: Michael McCandless
>            Priority: Trivial
>         Attachments: lucene-1322.patch
>
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless resolved LUCENE-1322.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 2.4

Thanks Jason!

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Assignee: Michael McCandless
>            Priority: Trivial
>             Fix For: 2.4
>
>         Attachments: lucene-1322.patch
>
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612911#action_12612911 ]

Jason Rutherglen commented on LUCENE-1322:
------------------------------------------

Seeing a possible bug in this patch:

{noformat}
1) testSystem(org.apache.lucene.ocean.TestSystem)java.io.IOException: Negative seek offset
at java.io.RandomAccessFile.seek(Native Method)
at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:578)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:131)
at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:68)
at org.apache.lucene.store.IndexInput.readLong(IndexInput.java:91)
at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:212)
at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:760)
at org.apache.lucene.index.IndexReader.document(IndexReader.java:572)
at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:92)
{noformat}

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Assignee: Michael McCandless
>            Priority: Trivial
>             Fix For: 2.4
>
>         Attachments: lucene-1322.patch
>
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-1322) Remove synchronization in CompoundFileReader

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1322:
-------------------------------------

    Comment: was deleted

> Remove synchronization in CompoundFileReader
> --------------------------------------------
>
>                 Key: LUCENE-1322
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1322
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Assignee: Michael McCandless
>            Priority: Trivial
>             Fix For: 2.4
>
>         Attachments: lucene-1322.patch
>
>
> Currently there is what seems to be unnecessary synchronization in CompoundFileReader.  This is solved by cloning the base IndexInput.  Synchronization in low level IO classes creates lock contention on highly multi threaded Lucene installations, so much so that in many cases the CPU utilization never reaches the maximum without using something like ParallelMultiSearcher.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]