[jira] Created: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
IndexReader.close should forcefully evict entries from FieldCache
-----------------------------------------------------------------

                 Key: LUCENE-2135
                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
             Project: Lucene - Java
          Issue Type: Bug
          Components: Index
            Reporter: Michael McCandless
            Assignee: Michael McCandless
             Fix For: 3.1


Spinoff of java-user thread "heap memory issues when sorting by a string field".

We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.

WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.

To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787406#action_12787406 ]

Michael McCandless commented on LUCENE-2135:
--------------------------------------------

This is not unlike what we had to do in creating CloseableThreadLocal... that was another case where  the underlying impl failed to free things as immediately as we'd like.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787505#action_12787505 ]

Earwin Burrfoot commented on LUCENE-2135:
-----------------------------------------

A better approach is to don IR-keyed weakHashMaps completely and bind everything you need onto IR itself. That's how I do it and it works like a charm.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Uwe Schindler
LUCENE-831...

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [hidden email]


> -----Original Message-----
> From: Earwin Burrfoot (JIRA) [mailto:[hidden email]]
> Sent: Tuesday, December 08, 2009 4:40 PM
> To: [hidden email]
> Subject: [jira] Commented: (LUCENE-2135) IndexReader.close should
> forcefully evict entries from FieldCache
>
>
>     [ https://issues.apache.org/jira/browse/LUCENE-
> 2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=12787505#action_12787505 ]
>
> Earwin Burrfoot commented on LUCENE-2135:
> -----------------------------------------
>
> A better approach is to don IR-keyed weakHashMaps completely and bind
> everything you need onto IR itself. That's how I do it and it works like a
> charm.
>
> > IndexReader.close should forcefully evict entries from FieldCache
> > -----------------------------------------------------------------
> >
> >                 Key: LUCENE-2135
> >                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
> >             Project: Lucene - Java
> >          Issue Type: Bug
> >          Components: Index
> >            Reporter: Michael McCandless
> >            Assignee: Michael McCandless
> >             Fix For: 3.1
> >
> >
> > Spinoff of java-user thread "heap memory issues when sorting by a string
> field".
> > We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But
> this lacks immediacy on releasing the reference, after a reader is closed.
> > WeakHashMap can't free the key until the reader is no longer referenced
> by the app. And, apparently, WeakHashMap has a further impl detail that
> requires invoking one of its methods for it to notice that a key has just
> become only weakly reachable.
> > To fix this, I think on IR.close we should evict entries from the
> FieldCache, as long as the sub-readers are truly closed (refCount dropped
> to 0).
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787509#action_12787509 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

LUCENE-831...

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787514#action_12787514 ]

Yonik Seeley commented on LUCENE-2135:
--------------------------------------

bq. A better approach is to don IR-keyed weakHashMaps completely and bind everything you need onto IR itself. That's how I do it and it works like a charm.

That would be nice...  I'd love to see a
Map<Object,Object> IndexReader.getInfo()
That was usable by anyone (not just the field cache) to associate stuff with a reader.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787552#action_12787552 ]

Earwin Burrfoot commented on LUCENE-2135:
-----------------------------------------

bq. I'd love to see a Map<Object,Object> IndexReader.getInfo()
I'm currently using - <T> T IndexReader.component(Class<T> key)
Plus a bundle of factories passed to IR on construction. Factories are called after IR is initialized, and also for child IRs and reopens. In case of reopens, besides new IR they are handed the component they produced for the current one (probably better just to pass old IR).

I can try to conjure a patch this weekend.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787555#action_12787555 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

bq. I'm currently using - <T> T IndexReader.component(Class<T> key)

Thats much better than untyped. Same like AttributeSource.

+1 for a patch.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787561#action_12787561 ]

Christian Kohlschütter commented on LUCENE-2135:
------------------------------------------------

Please see LUCENE-2133 for a refactoring of FieldCache, which also addresses these problems.


> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787585#action_12787585 ]

Earwin Burrfoot commented on LUCENE-2135:
-----------------------------------------

bq. Please see LUCENE-2133 for a refactoring of FieldCache, which also addresses these problems.
It doesn't address a problem of adding custom components to IR. It also does complicate IR beyond that unholy mess it already is.

I think it's better to have an ability to add 'any' kind of component to IR, and then implement whateverCaches over it.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787599#action_12787599 ]

Christian Kohlschütter commented on LUCENE-2135:
------------------------------------------------

I haven't followed the aforementioned discussion on the mailing list, but I think this issue covers a few things that are not mentioned explicitly here. Maybe it is a good idea to summarize the actual problems/challenges/benefits in a few sentences?

What I understand is that you plan to add arbitrary (cacheable) attributes to IndexReader. I suggest to move these features to the IndexCache proposed in LUCENE-2133. Especially when using decorating IndexReaders (things like "ReadOnlyIndexReader") you would not want to store attributes separately from the decorated IndexReader. The same probably applies to SegmentReader with all its clones.

IndexCache would provide a common base for the extensions you mentioned. (i.e. you are welcome to apply your patches on top of LUCENE-2133).

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2135:
---------------------------------------

    Attachment: LUCENE-2135.patch

I would love to see a bigger solution here, but in the interim, I
think we should fix the current FieldCache (patch attached).

The patch adds FieldCache.purge to the interface.  This is technically
a break in back-compat, to any external impls of FieldCache, but
that's such an insanely expert & difficult thing that I think it's
fine to make an exception.

A few tests (incl back-compat) needed fixing, because they were
closing the reader in-between to calls to getInts and then incorrectly
asserting the int[]'s were the same.


> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787610#action_12787610 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

In 2.9. we wrote in the BW section, that FieldCache interface is no BW problem as nobody ever can implement it (because the FileCacheImpl singleton is the only used one). Ok, you can implement it without any use. :-)

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787613#action_12787613 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

In my opinion, all IndexReaders should call purge, mabe put it on toplevel IR.close default impl? Because if you request FieldCache from Top-level (which you should not do, but you can), it should also be purged.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787624#action_12787624 ]

Michael McCandless commented on LUCENE-2135:
--------------------------------------------

bq. all IndexReaders should call purge, mabe put it on toplevel IR.close default impl?

Hmm... this actually gets tricky to get right, because of the FieldCacheKey.

EG on closing a SegmentReader that's a clone of another, you don't want to evict it from the FieldCache.

I guess I could fix each of the IndexReader subclasses to evict themselves from the cache.  Let me look into that...

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787626#action_12787626 ]

Yonik Seeley commented on LUCENE-2135:
--------------------------------------

bq. Hmm... this actually gets tricky to get right, because of the FieldCacheKey.

It's almost like we want two caches... one with entries that are independent of any changes in deleted docs (like the current FieldCache), and one that isn't.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

     [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-2135:
---------------------------------------

    Attachment: LUCENE-2135.patch

New patch, also evicts the other subclasses of IR from FieldCache.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch, LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787644#action_12787644 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

+1

I just noticed, it is even possible to retrieve a field cache from the FilterIndexReader and that would be a duplicate of the dlegate's cache. Very ugly.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch, LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787650#action_12787650 ]

Michael McCandless commented on LUCENE-2135:
--------------------------------------------

bq. it is even possible to retrieve a field cache from the FilterIndexReader and that would be a duplicate of the dlegate's cache.

Yeah, not good.  Should we default getFieldCacheKey to delegate?  A subclass of FIR would presumably need to then override if their filtering altered what's in the field cache.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch, LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (LUCENE-2135) IndexReader.close should forcefully evict entries from FieldCache

Tim Allison (Jira)
In reply to this post by Tim Allison (Jira)

    [ https://issues.apache.org/jira/browse/LUCENE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787652#action_12787652 ]

Uwe Schindler commented on LUCENE-2135:
---------------------------------------

+1 Good idea, just add a note to the method javadocs, that you have to override this, if you change the contents by the filter.

> IndexReader.close should forcefully evict entries from FieldCache
> -----------------------------------------------------------------
>
>                 Key: LUCENE-2135
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2135
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1
>
>         Attachments: LUCENE-2135.patch, LUCENE-2135.patch
>
>
> Spinoff of java-user thread "heap memory issues when sorting by a string field".
> We rely on WeakHashMap to hold our FieldCache, keyed by reader.  But this lacks immediacy on releasing the reference, after a reader is closed.
> WeakHashMap can't free the key until the reader is no longer referenced by the app. And, apparently, WeakHashMap has a further impl detail that requires invoking one of its methods for it to notice that a key has just become only weakly reachable.
> To fix this, I think on IR.close we should evict entries from the FieldCache, as long as the sub-readers are truly closed (refCount dropped to 0).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12