LUCENE-436

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

LUCENE-436

Otis Gospodnetic-2
I'm not at home with some of the things mentioned in LUCENE-436, so I'm not applying any of the various patches provided there, but it looks like something that deserves attention.  I think it has been brought up a while back, too.
http://issues.apache.org/jira/browse/LUCENE-436

Otis
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: LUCENE-436

Fernando Padilla-3
so... what do you think?

We just took the patch through QA and there was a noticeable memory
increase through time, and once we applied the patch, the memory didn't
increase..

So if you don't like the solution.. what are some alternatives?

fernando

ps - www.protrade.com


Otis Gospodnetic wrote:

> I'm not at home with some of the things mentioned in LUCENE-436, so I'm not applying any of the various patches provided there, but it looks like something that deserves attention.  I think it has been brought up a while back, too.
> http://issues.apache.org/jira/browse/LUCENE-436
>
> Otis
>  
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: LUCENE-436

Robert Engels
As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
entire indexes. If not, then it is not such a big deal.

Rather than using FixedThreadLocal, a more involved solution using a runtime
property to determine which thread local impl to use is possible. In lieu of
that, RAMDirectories are either broken, or everyone takes a performance hit.


-----Original Message-----
From: Fernando Padilla [mailto:[hidden email]]
Sent: Thursday, May 11, 2006 9:53 PM
To: [hidden email]
Subject: Re: LUCENE-436


so... what do you think?

We just took the patch through QA and there was a noticeable memory
increase through time, and once we applied the patch, the memory didn't
increase..

So if you don't like the solution.. what are some alternatives?

fernando

ps - www.protrade.com


Otis Gospodnetic wrote:
> I'm not at home with some of the things mentioned in LUCENE-436, so I'm
not applying any of the various patches provided there, but it looks like
something that deserves attention.  I think it has been brought up a while
back, too.

> http://issues.apache.org/jira/browse/LUCENE-436
>
> Otis
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: LUCENE-436

Fernando Padilla-3
hmm. So what you're saying is that there is a "memory leak", but very
very noticeable with large RAMDirectories (like what we have)...

With a 5M directories, it seems to be leaking atleast 5M per hour,
depending on queries..  even on our 1500M VM they run our of memory over
24 hours.

So I guess we have no choice but to use FSDirectories?


The FixedThreadLocal patch doesn't seem to have helped afterall..



Robert Engels wrote:

> As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
> entire indexes. If not, then it is not such a big deal.
>
> Rather than using FixedThreadLocal, a more involved solution using a runtime
> property to determine which thread local impl to use is possible. In lieu of
> that, RAMDirectories are either broken, or everyone takes a performance hit.
>
>
> -----Original Message-----
> From: Fernando Padilla [mailto:[hidden email]]
> Sent: Thursday, May 11, 2006 9:53 PM
> To: [hidden email]
> Subject: Re: LUCENE-436
>
>
> so... what do you think?
>
> We just took the patch through QA and there was a noticeable memory
> increase through time, and once we applied the patch, the memory didn't
> increase..
>
> So if you don't like the solution.. what are some alternatives?
>
> fernando
>
> ps - www.protrade.com
>
>
> Otis Gospodnetic wrote:
>
>>I'm not at home with some of the things mentioned in LUCENE-436, so I'm
>
> not applying any of the various patches provided there, but it looks like
> something that deserves attention.  I think it has been brought up a while
> back, too.
>
>>http://issues.apache.org/jira/browse/LUCENE-436
>>
>>Otis
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: [hidden email]
>>For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: LUCENE-436

Robert Engels
There is no "memory leak" per se - just the propensity to use more memory
than would seem to be needed (based on index size).

Using the FixedThreadLocal along with the modified TermInfosReader (that
uses it), the memory problem is resolved. If you are not seeing that, then
you have some other memory leak in your application.

-----Original Message-----
From: Fernando Padilla [mailto:[hidden email]]
Sent: Friday, May 12, 2006 1:35 PM
To: [hidden email]
Subject: Re: LUCENE-436


hmm. So what you're saying is that there is a "memory leak", but very
very noticeable with large RAMDirectories (like what we have)...

With a 5M directories, it seems to be leaking atleast 5M per hour,
depending on queries..  even on our 1500M VM they run our of memory over
24 hours.

So I guess we have no choice but to use FSDirectories?


The FixedThreadLocal patch doesn't seem to have helped afterall..



Robert Engels wrote:
> As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
> entire indexes. If not, then it is not such a big deal.
>
> Rather than using FixedThreadLocal, a more involved solution using a
runtime
> property to determine which thread local impl to use is possible. In lieu
of
> that, RAMDirectories are either broken, or everyone takes a performance
hit.

>
>
> -----Original Message-----
> From: Fernando Padilla [mailto:[hidden email]]
> Sent: Thursday, May 11, 2006 9:53 PM
> To: [hidden email]
> Subject: Re: LUCENE-436
>
>
> so... what do you think?
>
> We just took the patch through QA and there was a noticeable memory
> increase through time, and once we applied the patch, the memory didn't
> increase..
>
> So if you don't like the solution.. what are some alternatives?
>
> fernando
>
> ps - www.protrade.com
>
>
> Otis Gospodnetic wrote:
>
>>I'm not at home with some of the things mentioned in LUCENE-436, so I'm
>
> not applying any of the various patches provided there, but it looks like
> something that deserves attention.  I think it has been brought up a while
> back, too.
>
>>http://issues.apache.org/jira/browse/LUCENE-436
>>
>>Otis
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: [hidden email]
>>For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]