ThreadLocal in SegmentReader

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

ThreadLocal in SegmentReader

Roman Puchkovskiy-2
Hi.

There's a ThreadLocal field in SegmentReader (it's called termVectorsLocal). Some value is put to it, but it's never cleared.
Is it ok? It looks like sometimes this behavior may lead to leaks.

This is the same in lucene-2.2.0 and lucene-2.3.2.

Thanks in advance.
Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Michael McCandless-2

Well ... if the thread dies, the value in its ThreadLocal should be  
GC'd.

If the thread does not die (eg thread pool in an app server) then the  
ThreadLocal value remains, but that value is a shallow clone of the  
original TermVectorsReader and should not be consuming that much RAM  
per thread.

So I think it's OK?

Mike

Roman Puchkovskiy wrote:

>
> Hi.
>
> There's a ThreadLocal field in SegmentReader (it's called  
> termVectorsLocal).
> Some value is put to it, but it's never cleared.
> Is it ok? It looks like sometimes this behavior may lead to leaks.
>
> This is the same in lucene-2.2.0 and lucene-2.3.2.
>
> Thanks in advance.
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Roman Puchkovskiy-2
Unfortunately, it's not ok sometimes. For instance, when Lucene is loaded by a web-application from its WEB-INF/lib and SegmentReader is initialized during the application start-up (i.e. it's initialized in the thread which will never die), this causes problems with unloading of a classloader of the web-application. When web-application is undeployed, there's still a ThreadLocal in a thread which is external to webapp classloader, and this ThreadLocal contains an object which references its class, and this class was loaded through web-app classloader and hence references it... so we have a chain of hard links from an alive thread to our classloader. In the result, the classloader cannot be unloaded together will all its classes, so memory waste is considerable.

I've found a way to work this around by creating a new thread during webapp start-up and executing code which eventually initializes Lucene indices from this thread, so all spawned ThreadLocals go to this short-lived thread and get garbage-collected shortly after the webapp start-up is finished. But this does not seem to be a pretty solution. Besides, one has to guess that ThreadLocals are the problem to invent such a work-around :)

Michael McCandless-2 wrote
Well ... if the thread dies, the value in its ThreadLocal should be  
GC'd.

If the thread does not die (eg thread pool in an app server) then the  
ThreadLocal value remains, but that value is a shallow clone of the  
original TermVectorsReader and should not be consuming that much RAM  
per thread.

So I think it's OK?

Mike

Roman Puchkovskiy wrote:

>
> Hi.
>
> There's a ThreadLocal field in SegmentReader (it's called  
> termVectorsLocal).
> Some value is put to it, but it's never cleared.
> Is it ok? It looks like sometimes this behavior may lead to leaks.
>
> This is the same in lucene-2.2.0 and lucene-2.3.2.
>
> Thanks in advance.
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Michael McCandless-2

Hmmm, I see... you're right.  ThreadLocal is dangerous.

So how would you recommend fixing it?

One thing we can do, in SegmentReader.close, is to call  
termVectorsLocal.set(null).  We do this eg in FieldsReader.close,  
which uses a ThreadLocal to hold thread-private clones of the  
fieldsStream

However, that only works if the very same thread that had opened the  
reader also calls close... which likely often is the case but in  
general is not guaranteed and we should not expect/require.

How about if we set termVectorsLocal itself to null?  Will GC then "do  
the right thing", ie, recognize (eventually) that this ThreadLocal  
instance is no longer referenced, and clear all Objects for all  
threads that were held in it?

Mike

Roman Puchkovskiy wrote:

>
> Unfortunately, it's not ok sometimes. For instance, when Lucene is  
> loaded by
> a web-application from its WEB-INF/lib and SegmentReader is  
> initialized
> during the application start-up (i.e. it's initialized in the thread  
> which
> will never die), this causes problems with unloading of a  
> classloader of the
> web-application. When web-application is undeployed, there's still a
> ThreadLocal in a thread which is external to webapp classloader, and  
> this
> ThreadLocal contains an object which references its class, and this  
> class
> was loaded through web-app classloader and hence references it... so  
> we have
> a chain of hard links from an alive thread to our classloader. In the
> result, the classloader cannot be unloaded together will all its  
> classes, so
> memory waste is considerable.
>
> I've found a way to work this around by creating a new thread during  
> webapp
> start-up and executing code which eventually initializes Lucene  
> indices from
> this thread, so all spawned ThreadLocals go to this short-lived  
> thread and
> get garbage-collected shortly after the webapp start-up is finished.  
> But
> this does not seem to be a pretty solution. Besides, one has to  
> guess that
> ThreadLocals are the problem to invent such a work-around :)
>
>
> Michael McCandless-2 wrote:
>>
>>
>> Well ... if the thread dies, the value in its ThreadLocal should be
>> GC'd.
>>
>> If the thread does not die (eg thread pool in an app server) then the
>> ThreadLocal value remains, but that value is a shallow clone of the
>> original TermVectorsReader and should not be consuming that much RAM
>> per thread.
>>
>> So I think it's OK?
>>
>> Mike
>>
>> Roman Puchkovskiy wrote:
>>
>>>
>>> Hi.
>>>
>>> There's a ThreadLocal field in SegmentReader (it's called
>>> termVectorsLocal).
>>> Some value is put to it, but it's never cleared.
>>> Is it ok? It looks like sometimes this behavior may lead to leaks.
>>>
>>> This is the same in lucene-2.2.0 and lucene-2.3.2.
>>>
>>> Thanks in advance.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
>>> Sent from the Lucene - Java Users mailing list archive at  
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18314310.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Roman Puchkovskiy-2
Yes, calling set(null) does not seem a good fix. As for setting a reference to termVectorsLocal to null, not sure could this help or not, as this ThreadLocal will still be referenced by the thread (or threads).
Anyway, I will try to test this approach and post the results here.

Michael McCandless-2 wrote
Hmmm, I see... you're right.  ThreadLocal is dangerous.

So how would you recommend fixing it?

One thing we can do, in SegmentReader.close, is to call  
termVectorsLocal.set(null).  We do this eg in FieldsReader.close,  
which uses a ThreadLocal to hold thread-private clones of the  
fieldsStream

However, that only works if the very same thread that had opened the  
reader also calls close... which likely often is the case but in  
general is not guaranteed and we should not expect/require.

How about if we set termVectorsLocal itself to null?  Will GC then "do  
the right thing", ie, recognize (eventually) that this ThreadLocal  
instance is no longer referenced, and clear all Objects for all  
threads that were held in it?

Mike

Roman Puchkovskiy wrote:

>
> Unfortunately, it's not ok sometimes. For instance, when Lucene is  
> loaded by
> a web-application from its WEB-INF/lib and SegmentReader is  
> initialized
> during the application start-up (i.e. it's initialized in the thread  
> which
> will never die), this causes problems with unloading of a  
> classloader of the
> web-application. When web-application is undeployed, there's still a
> ThreadLocal in a thread which is external to webapp classloader, and  
> this
> ThreadLocal contains an object which references its class, and this  
> class
> was loaded through web-app classloader and hence references it... so  
> we have
> a chain of hard links from an alive thread to our classloader. In the
> result, the classloader cannot be unloaded together will all its  
> classes, so
> memory waste is considerable.
>
> I've found a way to work this around by creating a new thread during  
> webapp
> start-up and executing code which eventually initializes Lucene  
> indices from
> this thread, so all spawned ThreadLocals go to this short-lived  
> thread and
> get garbage-collected shortly after the webapp start-up is finished.  
> But
> this does not seem to be a pretty solution. Besides, one has to  
> guess that
> ThreadLocals are the problem to invent such a work-around :)
>
>
> Michael McCandless-2 wrote:
>>
>>
>> Well ... if the thread dies, the value in its ThreadLocal should be
>> GC'd.
>>
>> If the thread does not die (eg thread pool in an app server) then the
>> ThreadLocal value remains, but that value is a shallow clone of the
>> original TermVectorsReader and should not be consuming that much RAM
>> per thread.
>>
>> So I think it's OK?
>>
>> Mike
>>
>> Roman Puchkovskiy wrote:
>>
>>>
>>> Hi.
>>>
>>> There's a ThreadLocal field in SegmentReader (it's called
>>> termVectorsLocal).
>>> Some value is put to it, but it's never cleared.
>>> Is it ok? It looks like sometimes this behavior may lead to leaks.
>>>
>>> This is the same in lucene-2.2.0 and lucene-2.3.2.
>>>
>>> Thanks in advance.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
>>> Sent from the Lucene - Java Users mailing list archive at  
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18314310.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Roman Puchkovskiy-2
In reply to this post by Michael McCandless-2
I've tested a little, and it seems that assigning a null is not sufficient. As expected...
I don't see other ways how to fix this, but I'm not the Lucene developer :)
Fortunately, there's the work-around with temporary thread.

Michael McCandless-2 wrote
Hmmm, I see... you're right.  ThreadLocal is dangerous.

So how would you recommend fixing it?

One thing we can do, in SegmentReader.close, is to call  
termVectorsLocal.set(null).  We do this eg in FieldsReader.close,  
which uses a ThreadLocal to hold thread-private clones of the  
fieldsStream

However, that only works if the very same thread that had opened the  
reader also calls close... which likely often is the case but in  
general is not guaranteed and we should not expect/require.

How about if we set termVectorsLocal itself to null?  Will GC then "do  
the right thing", ie, recognize (eventually) that this ThreadLocal  
instance is no longer referenced, and clear all Objects for all  
threads that were held in it?

Mike

Roman Puchkovskiy wrote:

>
> Unfortunately, it's not ok sometimes. For instance, when Lucene is  
> loaded by
> a web-application from its WEB-INF/lib and SegmentReader is  
> initialized
> during the application start-up (i.e. it's initialized in the thread  
> which
> will never die), this causes problems with unloading of a  
> classloader of the
> web-application. When web-application is undeployed, there's still a
> ThreadLocal in a thread which is external to webapp classloader, and  
> this
> ThreadLocal contains an object which references its class, and this  
> class
> was loaded through web-app classloader and hence references it... so  
> we have
> a chain of hard links from an alive thread to our classloader. In the
> result, the classloader cannot be unloaded together will all its  
> classes, so
> memory waste is considerable.
>
> I've found a way to work this around by creating a new thread during  
> webapp
> start-up and executing code which eventually initializes Lucene  
> indices from
> this thread, so all spawned ThreadLocals go to this short-lived  
> thread and
> get garbage-collected shortly after the webapp start-up is finished.  
> But
> this does not seem to be a pretty solution. Besides, one has to  
> guess that
> ThreadLocals are the problem to invent such a work-around :)
>
>
> Michael McCandless-2 wrote:
>>
>>
>> Well ... if the thread dies, the value in its ThreadLocal should be
>> GC'd.
>>
>> If the thread does not die (eg thread pool in an app server) then the
>> ThreadLocal value remains, but that value is a shallow clone of the
>> original TermVectorsReader and should not be consuming that much RAM
>> per thread.
>>
>> So I think it's OK?
>>
>> Mike
>>
>> Roman Puchkovskiy wrote:
>>
>>>
>>> Hi.
>>>
>>> There's a ThreadLocal field in SegmentReader (it's called
>>> termVectorsLocal).
>>> Some value is put to it, but it's never cleared.
>>> Is it ok? It looks like sometimes this behavior may lead to leaks.
>>>
>>> This is the same in lucene-2.2.0 and lucene-2.3.2.
>>>
>>> Thanks in advance.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
>>> Sent from the Lucene - Java Users mailing list archive at  
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18314310.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Michael McCandless-2

So now I'm confused: the SegmentReader itself should no longer be  
reachable, assuming you are not holding any references to your  
IndexReader.

Which means the ThreadLocal instance should no longer be reachable.

Which means it should be GC'd and everything it's holding should be  
GC'd as well, even if some threads under it are still alive.

I think?

So it seems like what we currently do should be working fine, yet it's  
apparently not.  Are you certain you don't keep a reference to your  
IndexReader somewhere?  Also, are you sure that you waited long enough  
for GC to reclaim these objects?

Mike

Roman Puchkovskiy wrote:

>
> I've tested a little, and it seems that assigning a null is not  
> sufficient.
> As expected...
> I don't see other ways how to fix this, but I'm not the Lucene  
> developer :)
> Fortunately, there's the work-around with temporary thread.
>
>
> Michael McCandless-2 wrote:
>>
>>
>> Hmmm, I see... you're right.  ThreadLocal is dangerous.
>>
>> So how would you recommend fixing it?
>>
>> One thing we can do, in SegmentReader.close, is to call
>> termVectorsLocal.set(null).  We do this eg in FieldsReader.close,
>> which uses a ThreadLocal to hold thread-private clones of the
>> fieldsStream
>>
>> However, that only works if the very same thread that had opened the
>> reader also calls close... which likely often is the case but in
>> general is not guaranteed and we should not expect/require.
>>
>> How about if we set termVectorsLocal itself to null?  Will GC then  
>> "do
>> the right thing", ie, recognize (eventually) that this ThreadLocal
>> instance is no longer referenced, and clear all Objects for all
>> threads that were held in it?
>>
>> Mike
>>
>> Roman Puchkovskiy wrote:
>>
>>>
>>> Unfortunately, it's not ok sometimes. For instance, when Lucene is
>>> loaded by
>>> a web-application from its WEB-INF/lib and SegmentReader is
>>> initialized
>>> during the application start-up (i.e. it's initialized in the thread
>>> which
>>> will never die), this causes problems with unloading of a
>>> classloader of the
>>> web-application. When web-application is undeployed, there's still a
>>> ThreadLocal in a thread which is external to webapp classloader, and
>>> this
>>> ThreadLocal contains an object which references its class, and this
>>> class
>>> was loaded through web-app classloader and hence references it... so
>>> we have
>>> a chain of hard links from an alive thread to our classloader. In  
>>> the
>>> result, the classloader cannot be unloaded together will all its
>>> classes, so
>>> memory waste is considerable.
>>>
>>> I've found a way to work this around by creating a new thread during
>>> webapp
>>> start-up and executing code which eventually initializes Lucene
>>> indices from
>>> this thread, so all spawned ThreadLocals go to this short-lived
>>> thread and
>>> get garbage-collected shortly after the webapp start-up is finished.
>>> But
>>> this does not seem to be a pretty solution. Besides, one has to
>>> guess that
>>> ThreadLocals are the problem to invent such a work-around :)
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>>
>>>> Well ... if the thread dies, the value in its ThreadLocal should be
>>>> GC'd.
>>>>
>>>> If the thread does not die (eg thread pool in an app server) then  
>>>> the
>>>> ThreadLocal value remains, but that value is a shallow clone of the
>>>> original TermVectorsReader and should not be consuming that much  
>>>> RAM
>>>> per thread.
>>>>
>>>> So I think it's OK?
>>>>
>>>> Mike
>>>>
>>>> Roman Puchkovskiy wrote:
>>>>
>>>>>
>>>>> Hi.
>>>>>
>>>>> There's a ThreadLocal field in SegmentReader (it's called
>>>>> termVectorsLocal).
>>>>> Some value is put to it, but it's never cleared.
>>>>> Is it ok? It looks like sometimes this behavior may lead to leaks.
>>>>>
>>>>> This is the same in lucene-2.2.0 and lucene-2.3.2.
>>>>>
>>>>> Thanks in advance.
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18306230.html
>>>>> Sent from the Lucene - Java Users mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18314310.html
>>> Sent from the Lucene - Java Users mailing list archive at  
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ThreadLocal-in-SegmentReader-tp18306230p18317640.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Yonik Seeley-2
On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless
<[hidden email]> wrote:
> So now I'm confused: the SegmentReader itself should no longer be reachable,
> assuming you are not holding any references to your IndexReader.
>
> Which means the ThreadLocal instance should no longer be reachable.

It will still be referenced from the Thread(s) ThreadLocalMap
The key (the ThreadLocal) will be weakly referenced, but the values
(now stale) are strongly referenced and won't be actually removed
until the table is resized (under the Java6 impl at least).
Nice huh?

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Michael McCandless-2

Ugh!  I'll move this to java-dev to brainstorm fixes...

Mike

Yonik Seeley wrote:

> On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless
> <[hidden email]> wrote:
>> So now I'm confused: the SegmentReader itself should no longer be  
>> reachable,
>> assuming you are not holding any references to your IndexReader.
>>
>> Which means the ThreadLocal instance should no longer be reachable.
>
> It will still be referenced from the Thread(s) ThreadLocalMap
> The key (the ThreadLocal) will be weakly referenced, but the values
> (now stale) are strongly referenced and won't be actually removed
> until the table is resized (under the Java6 impl at least).
> Nice huh?
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Michael McCandless-2
In reply to this post by Yonik Seeley-2

After discussing this on java-dev:

     http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200807.mbox/%3CF5FC94B2-E5C7-40C0-8B73-E12245B91CEE@...%3E

it seems that this is not in fact a leak but rather a delayed GC  
issue.  The objects are eventually freed, on Sun 1.4, 1.5 and 1.6.

When a ThreadLocal instance becomes unreferenced & GC'd, Java does not  
immediately reclaim the now-unreferenced objects, if the thread  
remains alive.  However, over time, as other ThreadLocals are used  
with that thread, those stale objects do eventually get reclaimed.  So  
my feeling for now is it's OK for Lucene to continue to use non-static  
ThreadLocals.

Mike

Yonik Seeley wrote:

> On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless
> <[hidden email]> wrote:
>> So now I'm confused: the SegmentReader itself should no longer be  
>> reachable,
>> assuming you are not holding any references to your IndexReader.
>>
>> Which means the ThreadLocal instance should no longer be reachable.
>
> It will still be referenced from the Thread(s) ThreadLocalMap
> The key (the ThreadLocal) will be weakly referenced, but the values
> (now stale) are strongly referenced and won't be actually removed
> until the table is resized (under the Java6 impl at least).
> Nice huh?
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ThreadLocal in SegmentReader

Roman Puchkovskiy-2
Well, this 'replacement' of the ThreadLocal does not solve the initial problem. As there's always at least one ThreadLocal which binds the object loaded by the web-app to the Thread which is _not_ loaded by the web-app, the classloader never may be unloaded.

You are right, this is not the 'leak' in the usual sense, but this is not the 'delayed GC' problem, this is the 'never-happening-GC' problem when you have the classloader which needs to be unloaded.

Michael McCandless-2 wrote
After discussing this on java-dev:

     http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200807.mbox/%3CF5FC94B2-E5C7-40C0-8B73-E12245B91CEE@mikemccandless.com%3E

it seems that this is not in fact a leak but rather a delayed GC  
issue.  The objects are eventually freed, on Sun 1.4, 1.5 and 1.6.

When a ThreadLocal instance becomes unreferenced & GC'd, Java does not  
immediately reclaim the now-unreferenced objects, if the thread  
remains alive.  However, over time, as other ThreadLocals are used  
with that thread, those stale objects do eventually get reclaimed.  So  
my feeling for now is it's OK for Lucene to continue to use non-static  
ThreadLocals.

Mike

Yonik Seeley wrote:

> On Mon, Jul 7, 2008 at 2:43 PM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> So now I'm confused: the SegmentReader itself should no longer be  
>> reachable,
>> assuming you are not holding any references to your IndexReader.
>>
>> Which means the ThreadLocal instance should no longer be reachable.
>
> It will still be referenced from the Thread(s) ThreadLocalMap
> The key (the ThreadLocal) will be weakly referenced, but the values
> (now stale) are strongly referenced and won't be actually removed
> until the table is resized (under the Java6 impl at least).
> Nice huh?
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org