FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"

Doron Cohen-2
Hi, this test consistently fails on Windows with an IBM JDK, with this error:

> java.lang.IllegalArgumentException: At least 0.5MB RAM buffer is needed: 432472
at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:159)
at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:150)
at org.apache.lucene.search.suggest.fst.FSTCompletionLookup.build(FSTCompletionLookup.java:181)
at org.apache.lucene.search.suggest.fst.FSTCompletionTest.testLargeInputConstantWeights(FSTCompletionTest.java:164)
> NOTE: reproduce with: ant test -Dtestcase=FSTCompletionTest -Dtestmethod=testLargeInputConstantWeights -Dtests.seed=63069fe8e90d25f1:-4459dd4f7ddf2b26:71f954eeb3888217 -Dargs="-> Dfile.encoding=UTF-8"
> NOTE: test params are: locale=et_EE, timezone=America/Argentina/La_Rioja
> NOTE: all tests run in this JVM:
> [FSTCompletionTest]
> NOTE: Windows 7 6.1 build 7601 Service Pack 1 amd64/IBM Corporation 1.6.0 (64-bit)/cpus=2,threads=4,free=330928,total=6291456

The memory provided to sort is computed in contrib/spell/.../suggest.fst.Sort.automatic():

{code}
    public static BufferSize automatic() {
      long freeHeap = Runtime.getRuntime().freeMemory();
      return new BufferSize(Math.min(MIN_BUFFER_SIZE_MB * MB, freeHeap / 2));
    }
{code}

With Oracle's Java 6 the test passed. 

With IBM JDK, the fails even with -Xmx700m.   (Allow allocating at most 177M.)
But It will pass with just -Xms10m.   (Allow allocating 10M at start.)

So, if in a certain moment in a JVM's life the currently allocated memory is almost exhausted, Sort will fail, even if the settings in effect allow to allocate more heap.

It seems "nice" that Sort attempts to behave "nice" - use as much as half of the currently free heap.
This makes sense.
But perhaps in the situation that there's not enough free memory but the max memory settings allow to allocate more, a reasonable minimum should be returned, even the minimum of 0.5M. This will cause additional memory allocation for heap, but I think in this case it is justified?

Doron
Reply | Threaded
Open this post in threaded view
|

Re: FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"

Robert Muir
Doron, this sounds like something we should fix: I think we should
open a JIRA issue for it.

On Mon, Jan 30, 2012 at 12:34 PM, Doron Cohen <[hidden email]> wrote:

> Hi, this test consistently fails on Windows with an IBM JDK, with this
> error:
>
>> java.lang.IllegalArgumentException: At least 0.5MB RAM buffer is needed:
>> 432472
>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:159)
>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:150)
>>  at
>> org.apache.lucene.search.suggest.fst.FSTCompletionLookup.build(FSTCompletionLookup.java:181)
>>  at
>> org.apache.lucene.search.suggest.fst.FSTCompletionTest.testLargeInputConstantWeights(FSTCompletionTest.java:164)
>>
>> NOTE: reproduce with: ant test -Dtestcase=FSTCompletionTest
>> -Dtestmethod=testLargeInputConstantWeights
>> -Dtests.seed=63069fe8e90d25f1:-4459dd4f7ddf2b26:71f954eeb3888217 -Dargs="->
>> Dfile.encoding=UTF-8"
>> NOTE: test params are: locale=et_EE, timezone=America/Argentina/La_Rioja
>> NOTE: all tests run in this JVM:
>> [FSTCompletionTest]
>> NOTE: Windows 7 6.1 build 7601 Service Pack 1 amd64/IBM Corporation 1.6.0
>> (64-bit)/cpus=2,threads=4,free=330928,total=6291456
>
> The memory provided to sort is computed in
> contrib/spell/.../suggest.fst.Sort.automatic():
>
> {code}
>     public static BufferSize automatic() {
>       long freeHeap = Runtime.getRuntime().freeMemory();
>       return new BufferSize(Math.min(MIN_BUFFER_SIZE_MB * MB, freeHeap /
> 2));
>     }
> {code}
>
> With Oracle's Java 6 the test passed.
>
> With IBM JDK, the fails even with -Xmx700m.   (Allow allocating at most
> 177M.)
> But It will pass with just -Xms10m.   (Allow allocating 10M at start.)
>
> So, if in a certain moment in a JVM's life the currently allocated memory is
> almost exhausted, Sort will fail, even if the settings in effect allow to
> allocate more heap.
>
> It seems "nice" that Sort attempts to behave "nice" - use as much as half of
> the currently free heap.
> This makes sense.
> But perhaps in the situation that there's not enough free memory but the max
> memory settings allow to allocate more, a reasonable minimum should be
> returned, even the minimum of 0.5M. This will cause additional memory
> allocation for heap, but I think in this case it is justified?
>
> Doron



--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"

Dawid Weiss
long freeHeap = Runtime.getRuntime().freeMemory();

Indeed, this doesn't look right; it'd have to be used in combination
with (maxMemory - totalMemory) because that's how much the heap can
grow? The problem is none of this is atomic, so the result can
unpredictable. There are other methods in management interface that
permit a somewhat more detailed checks. Don't know if they guarantee
atomicity of the returned snapshot, but I doubt it.

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryMXBean.html#getHeapMemoryUsage()
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryPoolMXBean.html#getPeakUsage()

Dawid

On Thu, Feb 2, 2012 at 4:58 PM, Robert Muir <[hidden email]> wrote:

> Doron, this sounds like something we should fix: I think we should
> open a JIRA issue for it.
>
> On Mon, Jan 30, 2012 at 12:34 PM, Doron Cohen <[hidden email]> wrote:
>> Hi, this test consistently fails on Windows with an IBM JDK, with this
>> error:
>>
>>> java.lang.IllegalArgumentException: At least 0.5MB RAM buffer is needed:
>>> 432472
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:159)
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:150)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionLookup.build(FSTCompletionLookup.java:181)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionTest.testLargeInputConstantWeights(FSTCompletionTest.java:164)
>>>
>>> NOTE: reproduce with: ant test -Dtestcase=FSTCompletionTest
>>> -Dtestmethod=testLargeInputConstantWeights
>>> -Dtests.seed=63069fe8e90d25f1:-4459dd4f7ddf2b26:71f954eeb3888217 -Dargs="->
>>> Dfile.encoding=UTF-8"
>>> NOTE: test params are: locale=et_EE, timezone=America/Argentina/La_Rioja
>>> NOTE: all tests run in this JVM:
>>> [FSTCompletionTest]
>>> NOTE: Windows 7 6.1 build 7601 Service Pack 1 amd64/IBM Corporation 1.6.0
>>> (64-bit)/cpus=2,threads=4,free=330928,total=6291456
>>
>> The memory provided to sort is computed in
>> contrib/spell/.../suggest.fst.Sort.automatic():
>>
>> {code}
>>     public static BufferSize automatic() {
>>       long freeHeap = Runtime.getRuntime().freeMemory();
>>       return new BufferSize(Math.min(MIN_BUFFER_SIZE_MB * MB, freeHeap /
>> 2));
>>     }
>> {code}
>>
>> With Oracle's Java 6 the test passed.
>>
>> With IBM JDK, the fails even with -Xmx700m.   (Allow allocating at most
>> 177M.)
>> But It will pass with just -Xms10m.   (Allow allocating 10M at start.)
>>
>> So, if in a certain moment in a JVM's life the currently allocated memory is
>> almost exhausted, Sort will fail, even if the settings in effect allow to
>> allocate more heap.
>>
>> It seems "nice" that Sort attempts to behave "nice" - use as much as half of
>> the currently free heap.
>> This makes sense.
>> But perhaps in the situation that there's not enough free memory but the max
>> memory settings allow to allocate more, a reasonable minimum should be
>> returned, even the minimum of 0.5M. This will cause additional memory
>> allocation for heap, but I think in this case it is justified?
>>
>> Doron
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: FSTCompletionTest failure "At least 0.5MB RAM buffer is needed"

Doron Cohen-2
Great, I thought so too just waited for a other opinions and almost forgot about it!
Doron

On Thu, Feb 2, 2012 at 6:08 PM, Dawid Weiss <[hidden email]> wrote:
long freeHeap = Runtime.getRuntime().freeMemory();

Indeed, this doesn't look right; it'd have to be used in combination
with (maxMemory - totalMemory) because that's how much the heap can
grow? The problem is none of this is atomic, so the result can
unpredictable. There are other methods in management interface that
permit a somewhat more detailed checks. Don't know if they guarantee
atomicity of the returned snapshot, but I doubt it.

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryMXBean.html#getHeapMemoryUsage()
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryPoolMXBean.html#getPeakUsage()

Dawid

On Thu, Feb 2, 2012 at 4:58 PM, Robert Muir <[hidden email]> wrote:
> Doron, this sounds like something we should fix: I think we should
> open a JIRA issue for it.
>
> On Mon, Jan 30, 2012 at 12:34 PM, Doron Cohen <[hidden email]> wrote:
>> Hi, this test consistently fails on Windows with an IBM JDK, with this
>> error:
>>
>>> java.lang.IllegalArgumentException: At least 0.5MB RAM buffer is needed:
>>> 432472
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:159)
>>>  at org.apache.lucene.search.suggest.fst.Sort.<init>(Sort.java:150)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionLookup.build(FSTCompletionLookup.java:181)
>>>  at
>>> org.apache.lucene.search.suggest.fst.FSTCompletionTest.testLargeInputConstantWeights(FSTCompletionTest.java:164)
>>>
>>> NOTE: reproduce with: ant test -Dtestcase=FSTCompletionTest
>>> -Dtestmethod=testLargeInputConstantWeights
>>> -Dtests.seed=63069fe8e90d25f1:-4459dd4f7ddf2b26:71f954eeb3888217 -Dargs="->
>>> Dfile.encoding=UTF-8"
>>> NOTE: test params are: locale=et_EE, timezone=America/Argentina/La_Rioja
>>> NOTE: all tests run in this JVM:
>>> [FSTCompletionTest]
>>> NOTE: Windows 7 6.1 build 7601 Service Pack 1 amd64/IBM Corporation 1.6.0
>>> (64-bit)/cpus=2,threads=4,free=330928,total=6291456
>>
>> The memory provided to sort is computed in
>> contrib/spell/.../suggest.fst.Sort.automatic():
>>
>> {code}
>>     public static BufferSize automatic() {
>>       long freeHeap = Runtime.getRuntime().freeMemory();
>>       return new BufferSize(Math.min(MIN_BUFFER_SIZE_MB * MB, freeHeap /
>> 2));
>>     }
>> {code}
>>
>> With Oracle's Java 6 the test passed.
>>
>> With IBM JDK, the fails even with -Xmx700m.   (Allow allocating at most
>> 177M.)
>> But It will pass with just -Xms10m.   (Allow allocating 10M at start.)
>>
>> So, if in a certain moment in a JVM's life the currently allocated memory is
>> almost exhausted, Sort will fail, even if the settings in effect allow to
>> allocate more heap.
>>
>> It seems "nice" that Sort attempts to behave "nice" - use as much as half of
>> the currently free heap.
>> This makes sense.
>> But perhaps in the situation that there's not enough free memory but the max
>> memory settings allow to allocate more, a reasonable minimum should be
>> returned, even the minimum of 0.5M. This will cause additional memory
>> allocation for heap, but I think in this case it is justified?
>>
>> Doron
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]