[jira] Created: (LUCENE-2555) Remove shared doc stores

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
Remove shared doc stores
------------------------

                 Key: LUCENE-2555
                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
            Reporter: Michael Busch
            Assignee: Michael Busch
            Priority: Minor
             Fix For: Realtime Branch


With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.

See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891414#action_12891414 ]

Michael Busch commented on LUCENE-2555:
---------------------------------------

What shall we do about index backward-compatibility?

I guess 4.0 has to be able to read shared doc stores?  So a lot of that code we can't remove? :(

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891422#action_12891422 ]

Jason Rutherglen commented on LUCENE-2555:
------------------------------------------

Maybe we should break backwards-compatibility for the RT branch?  Or just ship an RT specific JAR to keep things simple?

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891433#action_12891433 ]

Michael McCandless commented on LUCENE-2555:
--------------------------------------------

The reading side of shared doc stores is quite trivial; I think we should keep it (keep back compat).

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891558#action_12891558 ]

Shai Erera commented on LUCENE-2555:
------------------------------------

What are the performance implications of removing shared doc stores? From what I understand, if several segments share the same doc store, then when they are merged, the doc stores aren't merged. Which is a great benefit, especially if you intend to store large fields.

I understand (mostly from the discussion on the PTDW) that with the move to a per-thread approach, the doc stores cannot be shared between segments created by different threads, but what about segments created by the same thread? Are we losing that functionality?

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891654#action_12891654 ]

Jason Rutherglen commented on LUCENE-2555:
------------------------------------------

Shai, I think Mike has outlined the pros and cons in other
postings see:

https://issues.apache.org/jira/browse/LUCENE-2324?focusedCommentId=12891256&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acommenttabpanel#action_12891256 

Basically when going to DWPTs we're losing shared doc stores
completely, and sharing between threads probably doesn't make
sense. However we can keep the reading of shared doc stores for
back compat. I think the confusing part in the code is the
writing of shared doc stores, and I'm glad that's going away. In
addition the DWPT code completely streamlines some of the most
confusing parts of the IndexWriter class tree (the wait notify,
and per thread logic in particular). Overall this will help
future folks when they're trying to customize IndexWriter, and
in addition, remove a layer of complexity, as we add yet another
layer of complexity with the RT code.


> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891655#action_12891655 ]

Michael Busch commented on LUCENE-2555:
---------------------------------------

{quote}
I understand (mostly from the discussion on the PTDW) that with the move to a per-thread approach, the doc stores cannot be shared between segments created by different threads, but what about segments created by the same thread? Are we losing that functionality?
{quote}

We discussed that in LUCENE-2324 (close to the bottom).  The problem is that doc stores only help you if you merge segments that all share the same store.  With DWPT that's extremely unlikely.  


{quote}
What are the performance implications of removing shared doc stores?
{quote}

I agree we have to test this when this patch is complete.  My hope is that we save in other places (removing the interleaving step of the per-thread postings, no wait queue that serializes writing to doc stores) so that overall we won't be slower.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891659#action_12891659 ]

Michael McCandless commented on LUCENE-2555:
--------------------------------------------

{quote}

bq. What are the performance implications of removing shared doc stores?

I agree we have to test this when this patch is complete. My hope is that we save in other places (removing the interleaving step of the per-thread postings, no wait queue that serializes writing to doc stores) so that overall we won't be slower.
{quote}

Also, remember that shared doc stores is not as good an opto as it used to be, because we are now able to bulk-copy both stored fields and term vectors during merging.

However, bulk merging only happens if the field name -> number mapping is congruent, b/w the merged segment and the one segment being merged.

Unfortunately, you can easily unexpectedly break this (see LUCENE-1737) but eg adding diff't fields to your docs, or adding same fields just in a different order.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891668#action_12891668 ]

Shai Erera commented on LUCENE-2555:
------------------------------------

Thanks for the explanation. Let's remember though that not all apps are multi-threaded, but I think most are, so designing for the most is better than making the other few more performing. I'm generally ok with that, just wanted to better understand the reasons.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Busch updated LUCENE-2555:
----------------------------------

    Attachment: lucene-2555.patch

Checkpointing what I have so far:

- Removed writing part of shared doc stores
- Reading part is kept for backwards compatibility.  TestBackwardsCompatibility passes.
- Removed DocumentsWriterPerThread.DocWriter and all its subclasses

Some testcases fail still.  Next I will look into removing PerDocBuffer, exception handling,
and then fix the failing tests.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>         Attachments: lucene-2555.patch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891709#action_12891709 ]

Jason Rutherglen commented on LUCENE-2555:
------------------------------------------

Michael, nice!  A lot is cleaned up.  

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>         Attachments: lucene-2555.patch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Busch updated LUCENE-2555:
----------------------------------

    Attachment: lucene-2555.patch

Changed the patch to also remove PerDocBuffer.  It changes StoredFieldsWriter and TermVectorsTermsWriter to write the data directly to the final IndexOutput, without buffering it in a temporary PerDocBuffer.

Several tests still fail due to exception handling or thread-safety problems (which is expected - haven't tried very hard to fix them yet).  I will commit this patch to the realtime branch and work on fixing the tests with a separate issue.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>         Attachments: lucene-2555.patch, lucene-2555.patch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (LUCENE-2555) Remove shared doc stores

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Busch resolved LUCENE-2555.
-----------------------------------

    Resolution: Fixed

Committed revision 978805.

> Remove shared doc stores
> ------------------------
>
>                 Key: LUCENE-2555
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2555
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>            Priority: Minor
>             Fix For: Realtime Branch
>
>         Attachments: lucene-2555.patch, lucene-2555.patch
>
>
> With per-thread DocumentsWriters sharing doc stores across segments doesn't make much sense anymore.
> See also LUCENE-2324.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]