too many instances of "org.tartarus.snowball.Among" in the heap

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

too many instances of "org.tartarus.snowball.Among" in the heap

roz dev
Hi All

I am trying to find out the reason for very high memory use and ran JMAP
-hist

It is showing that i have too many instances of org.tartarus.snowball.Among

Any ideas what is this for and why am I getting so many of them

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
*1:              46728110        1869124400      org.tartarus.snowball.Among
*
2:              5244210 1840458960      byte[]
3:              52651949        5969839368      char[]
4:              10008928        864769280       int[]
5:              10250527        410021080
java.util.LinkedHashMap$Entry
6:              4672811 268474232       org.tartarus.snowball.Among[]
*7:              8072312 258313984       java.util.HashMap$Entry*
8:              466514  246319392       org.apache.lucene.util.fst.FST$Arc[]
9:              1828542 237600432       java.util.HashMap$Entry[]
10:             3834312 153372480       java.util.TreeMap$Entry
11:             2684700 128865600
org.apache.lucene.util.fst.Builder$UnCompiledNode
12:             4712425 113098200       org.apache.lucene.util.BytesRef
13:             3484836 111514752       java.lang.String
14:             2636045 105441800       org.apache.lucene.index.FieldInfo
15:             1813561 101559416       java.util.LinkedHashMap
16:             6291619 100665904       java.lang.Integer
17:             2684700 85910400
org.apache.lucene.util.fst.Builder$Arc
18:             956998  84215824
org.apache.lucene.index.TermsHashPerField
19:             2892957 69430968
org.apache.lucene.util.AttributeSource$State
20:             2684700 64432800
org.apache.lucene.util.fst.Builder$Arc[]
21:             685595  60332360        org.apache.lucene.util.fst.FST
22:             933451  59210944        java.lang.Object[]
23:             957043  53594408        org.apache.lucene.util.BytesRefHash
24:             591463  42585336
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader
25:             424801  40780896
org.tartarus.snowball.ext.EnglishStemmer
26:             424801  40780896
org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter
27:             1549670 37192080        org.apache.lucene.index.Term
28:             849602  33984080
org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter$WordDelimiterConcatenation
29:             424801  27187264
org.apache.lucene.analysis.core.WhitespaceTokenizer
30:             478499  26795944
org.apache.lucene.index.FreqProxTermsWriterPerField
31:             535521  25705008
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray
32:             219081  24537072
org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter
33:             478499  22967952
org.apache.lucene.index.FieldInvertState
34:             956998  22967952
org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray
35:             478499  22967952
org.apache.lucene.index.TermVectorsConsumerPerField
36:             478499  22967952
org.apache.lucene.index.NormsConsumerPerField
37:             316582  22793904
org.apache.lucene.store.MMapDirectory$MMapIndexInput
38:             906708  21760992
org.apache.lucene.util.AttributeSource$State[]
39:             906708  21760992
org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl
40:             883588  21206112        java.util.ArrayList
41:             438192  21033216
org.apache.lucene.store.RAMOutputStream
42:             860601  20654424        java.lang.StringBuilder
43:             424801  20390448
org.apache.lucene.analysis.miscellaneous.WordDelimiterIterator
44:             424801  20390448
org.apache.lucene.analysis.core.StopFilter
45:             424801  20390448
org.apache.lucene.analysis.miscellaneous.KeywordMarkerFilter
46:             424801  20390448
org.apache.lucene.analysis.snowball.SnowballFilter
47:             839390  20145360
org.apache.lucene.index.DocumentsWriterDeleteQueue$TermNode


-Saroj
Reply | Threaded
Open this post in threaded view
|

Re: too many instances of "org.tartarus.snowball.Among" in the heap

Bernd Fehling
It is something from internally of the snowball analyzer (stemmer).

To find out more you should take a heapdump and look into it with
Memory Analyzer (MAT) http://www.eclipse.org/mat/

Regards,
Bernd


Am 27.07.2012 09:53, schrieb roz dev:

> Hi All
>
> I am trying to find out the reason for very high memory use and ran JMAP
> -hist
>
> It is showing that i have too many instances of org.tartarus.snowball.Among
>
> Any ideas what is this for and why am I getting so many of them
>
> num       #instances    #bytes  Class description
> --------------------------------------------------------------------------
> *1:              46728110        1869124400      org.tartarus.snowball.Among
> *
> 2:              5244210 1840458960      byte[]
> 3:              52651949        5969839368      char[]
> 4:              10008928        864769280       int[]
> 5:              10250527        410021080
> java.util.LinkedHashMap$Entry
> 6:              4672811 268474232       org.tartarus.snowball.Among[]
> *7:              8072312 258313984       java.util.HashMap$Entry*
> 8:              466514  246319392       org.apache.lucene.util.fst.FST$Arc[]
> 9:              1828542 237600432       java.util.HashMap$Entry[]
> 10:             3834312 153372480       java.util.TreeMap$Entry
> 11:             2684700 128865600
> org.apache.lucene.util.fst.Builder$UnCompiledNode
> 12:             4712425 113098200       org.apache.lucene.util.BytesRef
> 13:             3484836 111514752       java.lang.String
> 14:             2636045 105441800       org.apache.lucene.index.FieldInfo
> 15:             1813561 101559416       java.util.LinkedHashMap
> 16:             6291619 100665904       java.lang.Integer
> 17:             2684700 85910400
> org.apache.lucene.util.fst.Builder$Arc
> 18:             956998  84215824
> org.apache.lucene.index.TermsHashPerField
> 19:             2892957 69430968
> org.apache.lucene.util.AttributeSource$State
> 20:             2684700 64432800
> org.apache.lucene.util.fst.Builder$Arc[]
> 21:             685595  60332360        org.apache.lucene.util.fst.FST
> 22:             933451  59210944        java.lang.Object[]
> 23:             957043  53594408        org.apache.lucene.util.BytesRefHash
> 24:             591463  42585336
> org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader
> 25:             424801  40780896
> org.tartarus.snowball.ext.EnglishStemmer
> 26:             424801  40780896
> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter
> 27:             1549670 37192080        org.apache.lucene.index.Term
> 28:             849602  33984080
> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter$WordDelimiterConcatenation
> 29:             424801  27187264
> org.apache.lucene.analysis.core.WhitespaceTokenizer
> 30:             478499  26795944
> org.apache.lucene.index.FreqProxTermsWriterPerField
> 31:             535521  25705008
> org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray
> 32:             219081  24537072
> org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter
> 33:             478499  22967952
> org.apache.lucene.index.FieldInvertState
> 34:             956998  22967952
> org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray
> 35:             478499  22967952
> org.apache.lucene.index.TermVectorsConsumerPerField
> 36:             478499  22967952
> org.apache.lucene.index.NormsConsumerPerField
> 37:             316582  22793904
> org.apache.lucene.store.MMapDirectory$MMapIndexInput
> 38:             906708  21760992
> org.apache.lucene.util.AttributeSource$State[]
> 39:             906708  21760992
> org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl
> 40:             883588  21206112        java.util.ArrayList
> 41:             438192  21033216
> org.apache.lucene.store.RAMOutputStream
> 42:             860601  20654424        java.lang.StringBuilder
> 43:             424801  20390448
> org.apache.lucene.analysis.miscellaneous.WordDelimiterIterator
> 44:             424801  20390448
> org.apache.lucene.analysis.core.StopFilter
> 45:             424801  20390448
> org.apache.lucene.analysis.miscellaneous.KeywordMarkerFilter
> 46:             424801  20390448
> org.apache.lucene.analysis.snowball.SnowballFilter
> 47:             839390  20145360
> org.apache.lucene.index.DocumentsWriterDeleteQueue$TermNode
>
>
> -Saroj
>

--
*************************************************************
Bernd Fehling                Universit├Ątsbibliothek Bielefeld
Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
Universit├Ątsstr. 25                     und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************


Reply | Threaded
Open this post in threaded view
|

Re: too many instances of "org.tartarus.snowball.Among" in the heap

Alexandre Rafalovitch
In reply to this post by roz dev
Try taking a couple of thread dumps and see where in the stack the
snowball classes show up. That might give you a clue.

Did you customize the parameters to the stemmer? If so, maybe it has
problems with the file you gave it.

Just some generic thoughts that might help.

Regards,
   Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
book)


On Fri, Jul 27, 2012 at 3:53 AM, roz dev <[hidden email]> wrote:

> Hi All
>
> I am trying to find out the reason for very high memory use and ran JMAP
> -hist
>
> It is showing that i have too many instances of org.tartarus.snowball.Among
>
> Any ideas what is this for and why am I getting so many of them
>
> num       #instances    #bytes  Class description
> --------------------------------------------------------------------------
> *1:              46728110        1869124400      org.tartarus.snowball.Among
> *
> 2:              5244210 1840458960      byte[]
Reply | Threaded
Open this post in threaded view
|

Re: too many instances of "org.tartarus.snowball.Among" in the heap

roz dev
I did take couple of thread dumps and they seem to be fine

Heap dump is huge - close to 15GB

I am having hard time to analyze that heap dump

2012-07-30 16:07:32
Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode):

"RMI TCP Connection(33)-10.8.21.124" - Thread t@190
   java.lang.Thread.State: RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
    at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
    at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
    at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
    at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
    at javax.management.StandardMBean.invoke(StandardMBean.java:391)
    at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
    at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
    at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
    at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
    at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
    at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
    at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
    at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - locked <49cbecf2> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

"JMX server connection timeout 189" - Thread t@189
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <b75fa27> (a [I)
    at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-77" - Thread t@186
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-76" - Thread t@185
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-75" - Thread t@184
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-74" - Thread t@183
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-73" - Thread t@182
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-72" - Thread t@181
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-71" - Thread t@180
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"JMX server connection timeout 178" - Thread t@178
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <1170ca7f> (a [I)
    at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-62" - Thread t@163
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"web-60" - Thread t@161
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"RMI TCP Connection(29)-10.8.21.124" - Thread t@159
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    - locked <34d20b72> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - locked <49ed8475> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

"RMI TCP Connection(27)-10.8.21.124" - Thread t@158
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    - locked <7c239fd7> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - locked <6d31bed> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

"web-59" - Thread t@156
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5ab03cb6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"RMI TCP Connection(28)-10.8.21.124" - Thread t@134
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <47e2d0b1> (a
com.sun.jmx.remote.internal.ArrayNotificationBuffer)
    at
com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
    at
com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
    at
com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
    at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
    at
javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
    at
javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
    at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - locked <dce681b> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

"RMI Scheduler(0)" - Thread t@73
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <2ff3450> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"http-0.0.0.0-12061-Acceptor-0" - Thread t@32
   java.lang.Thread.State: RUNNABLE
    at org.apache.tomcat.jni.Socket.accept(Native Method)
    at
org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1233)

   Locked ownable synchronizers:
    - None

"http-0.0.0.0-12061-CometPoller-0" - Thread t@31
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <539e488e> (a
org.apache.tomcat.util.net.AprEndpoint$Poller)
    at java.lang.Object.wait(Object.java:485)
    at
org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1416)

   Locked ownable synchronizers:
    - None

"http-0.0.0.0-12061-Poller-0" - Thread t@30
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <3cad8d85> (a
org.apache.tomcat.util.net.AprEndpoint$Poller)
    at java.lang.Object.wait(Object.java:485)
    at
org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1416)

   Locked ownable synchronizers:
    - None

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@29
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1627)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-25-thread-1" - Thread t@28
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <262635e2> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-22-thread-1" - Thread t@27
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <169b9bff> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-19-thread-1" - Thread t@26
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <421e8df1> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-16-thread-1" - Thread t@25
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <63f8eb20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-13-thread-1" - Thread t@24
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <6b14b526> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-10-thread-1" - Thread t@23
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <1c8108b6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-7-thread-1" - Thread t@22
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <5b62f358> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-4-thread-1" - Thread t@21
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <3d564547> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"pool-1-thread-1" - Thread t@20
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <128d542b> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"RMI TCP Accept-0" - Thread t@19
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <4f141721> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"RMI TCP Accept-20061" - Thread t@18
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <2ffaa016> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"RMI TCP Accept-0" - Thread t@17
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <6f241cf5> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
    at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"jcollectd" - Thread t@13
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <6ade57c6> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Agent ServerConnection" - Thread t@11
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    - locked <1f772f96> (a java.net.SocksSocketImpl)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.<init>(Socket.java:375)
    at java.net.Socket.<init>(Socket.java:189)
    at
com.wily.isengard.postofficehub.link.net.DefaultSocketFactory.createSocket(DefaultSocketFactory.java:39)
    at
com.wily.isengard.api.ServerInstanceLocator.createSocket2(ServerInstanceLocator.java:353)
    at
com.wily.isengard.api.ServerInstanceLocator.createSocket(ServerInstanceLocator.java:320)
    at
com.wily.isengard.postofficehub.link.OutgoingConnection.connect(OutgoingConnection.java:71)
    at
com.wily.isengard.postoffice.PostOfficeHub.connectToServerHub(PostOfficeHub.java:378)
    at
com.wily.isengard.postoffice.PostOfficeHub.connectToServerHub(PostOfficeHub.java:337)
    at
com.wily.isengard.api.IsengardClient.getConnectedHub(IsengardClient.java:275)
    at com.wily.isengard.api.IsengardClient.connect(IsengardClient.java:156)
    at com.wily.isengard.api.IsengardClient.connect(IsengardClient.java:143)
    at
com.wily.introscope.agent.connection.IsengardClientConnection.connect(IsengardClientConnection.java:67)
    at
com.wily.introscope.agent.connection.IsengardServerConnectionManager.connectInternal(IsengardServerConnectionManager.java:646)
    at
com.wily.introscope.agent.connection.ConnectionThread.attemptConnection(ConnectionThread.java:141)
    at
com.wily.introscope.agent.connection.ConnectionThread.connect(ConnectionThread.java:260)
    at
com.wily.introscope.agent.connection.ConnectionThread.run(ConnectionThread.java:64)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Agent Execution" - Thread t@10
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <2b54befa> (a com.wily.util.adt.BlockingQueue)
    at java.lang.Object.wait(Object.java:485)
    at
com.wily.util.adt.BlockingQueue.interruptableDequeue(BlockingQueue.java:123)
    at
com.wily.util.task.AsynchExecutionQueue.doTask(AsynchExecutionQueue.java:200)
    at com.wily.util.task.ATask$CoreTask.run(ATask.java:132)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Agent Heartbeat" - Thread t@5
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at
com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Remove Metric Data Watch Heartbeat Heartbeat" - Thread t@7
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at
com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Configuration Watch Heartbeat Heartbeat" - Thread t@6
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at
com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - None

"Signal Dispatcher" - Thread t@4
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Finalizer" - Thread t@3
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <48c6254f> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

   Locked ownable synchronizers:
    - None

"Reference Handler" - Thread t@2
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    - waiting on <48bb8adc> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

   Locked ownable synchronizers:
    - None

"main" - Thread t@1
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <11dacd96> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at
com.wily.introscope.agent.probe.net.ManagedServerSocket.com_wily_accept14(ManagedServerSocket.java:362)
    at
com.wily.introscope.agent.probe.net.ManagedServerSocket.accept(ManagedServerSocket.java:267)
    at
org.apache.catalina.core.StandardServer.await(StandardServer.java:431)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:628)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

   Locked ownable synchronizers:
    - None



On Fri, Jul 27, 2012 at 5:19 AM, Alexandre Rafalovitch
<[hidden email]>wrote:

> Try taking a couple of thread dumps and see where in the stack the
> snowball classes show up. That might give you a clue.
>
> Did you customize the parameters to the stemmer? If so, maybe it has
> problems with the file you gave it.
>
> Just some generic thoughts that might help.
>
> Regards,
>    Alex.
> Personal blog: http://blog.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all
> at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> book)
>
>
> On Fri, Jul 27, 2012 at 3:53 AM, roz dev <[hidden email]> wrote:
> > Hi All
> >
> > I am trying to find out the reason for very high memory use and ran JMAP
> > -hist
> >
> > It is showing that i have too many instances of
> org.tartarus.snowball.Among
> >
> > Any ideas what is this for and why am I getting so many of them
> >
> > num       #instances    #bytes  Class description
> >
> --------------------------------------------------------------------------
> > *1:              46728110        1869124400
>  org.tartarus.snowball.Among
> > *
> > 2:              5244210 1840458960      byte[]
>
Reply | Threaded
Open this post in threaded view
|

Re: too many instances of "org.tartarus.snowball.Among" in the heap

roz dev
is it some kind of memory leak with Lucene's use of Snowball Stemmer?

I tried to google for Snowball Stemmer but could not find any recent info
about memory leak

this old link does indicate some memory leak but it is from 2004

http://snowball.tartarus.org/archives/snowball-discuss/0631.html

Any inputs are welcome

-Saroj




On Mon, Jul 30, 2012 at 4:39 PM, roz dev <[hidden email]> wrote:

> I did take couple of thread dumps and they seem to be fine
>
> Heap dump is huge - close to 15GB
>
> I am having hard time to analyze that heap dump
>
> 2012-07-30 16:07:32
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode):
>
> "RMI TCP Connection(33)-10.8.21.124" - Thread t@190
>    java.lang.Thread.State: RUNNABLE
>     at sun.management.ThreadImpl.dumpThreads0(Native Method)
>     at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
>     at
> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
>     at
> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
>     at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>     at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>     at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>     at javax.management.StandardMBean.invoke(StandardMBean.java:391)
>     at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>     at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>     at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>     at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>     at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>     at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>     at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>     at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>     at sun.rmi.transport.Transport$1.run(Transport.java:159)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>     at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - locked <49cbecf2> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "JMX server connection timeout 189" - Thread t@189
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <b75fa27> (a [I)
>     at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-77" - Thread t@186
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-76" - Thread t@185
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-75" - Thread t@184
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-74" - Thread t@183
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-73" - Thread t@182
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-72" - Thread t@181
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-71" - Thread t@180
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "JMX server connection timeout 178" - Thread t@178
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <1170ca7f> (a [I)
>     at
> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-62" - Thread t@163
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "web-60" - Thread t@161
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "RMI TCP Connection(29)-10.8.21.124" - Thread t@159
>    java.lang.Thread.State: RUNNABLE
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:129)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>     - locked <34d20b72> (a java.io.BufferedInputStream)
>     at java.io.FilterInputStream.read(FilterInputStream.java:66)
>     at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - locked <49ed8475> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "RMI TCP Connection(27)-10.8.21.124" - Thread t@158
>    java.lang.Thread.State: RUNNABLE
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:129)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>     - locked <7c239fd7> (a java.io.BufferedInputStream)
>     at java.io.FilterInputStream.read(FilterInputStream.java:66)
>     at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - locked <6d31bed> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "web-59" - Thread t@156
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5ab03cb6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "RMI TCP Connection(28)-10.8.21.124" - Thread t@134
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <47e2d0b1> (a
> com.sun.jmx.remote.internal.ArrayNotificationBuffer)
>     at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
>     at
> com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
>     at
> com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
>     at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
>     at
> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
>     at
> javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231)
>     at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>     at sun.rmi.transport.Transport$1.run(Transport.java:159)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>     at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - locked <dce681b> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "RMI Scheduler(0)" - Thread t@73
>    java.lang.Thread.State: TIMED_WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <2ff3450> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>     at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "http-0.0.0.0-12061-Acceptor-0" - Thread t@32
>    java.lang.Thread.State: RUNNABLE
>     at org.apache.tomcat.jni.Socket.accept(Native Method)
>     at
> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1233)
>
>    Locked ownable synchronizers:
>     - None
>
> "http-0.0.0.0-12061-CometPoller-0" - Thread t@31
>    java.lang.Thread.State: WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <539e488e> (a
> org.apache.tomcat.util.net.AprEndpoint$Poller)
>     at java.lang.Object.wait(Object.java:485)
>     at
> org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1416)
>
>    Locked ownable synchronizers:
>     - None
>
> "http-0.0.0.0-12061-Poller-0" - Thread t@30
>    java.lang.Thread.State: WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <3cad8d85> (a
> org.apache.tomcat.util.net.AprEndpoint$Poller)
>     at java.lang.Object.wait(Object.java:485)
>     at
> org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1416)
>
>    Locked ownable synchronizers:
>     - None
>
> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@29
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Thread.sleep(Native Method)
>     at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1627)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-25-thread-1" - Thread t@28
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <262635e2> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-22-thread-1" - Thread t@27
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <169b9bff> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-19-thread-1" - Thread t@26
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <421e8df1> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-16-thread-1" - Thread t@25
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <63f8eb20> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-13-thread-1" - Thread t@24
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <6b14b526> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-10-thread-1" - Thread t@23
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <1c8108b6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-7-thread-1" - Thread t@22
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <5b62f358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-4-thread-1" - Thread t@21
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <3d564547> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "pool-1-thread-1" - Thread t@20
>    java.lang.Thread.State: WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <128d542b> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "RMI TCP Accept-0" - Thread t@19
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>     - locked <4f141721> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at java.net.ServerSocket.accept(ServerSocket.java:430)
>     at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "RMI TCP Accept-20061" - Thread t@18
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>     - locked <2ffaa016> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at java.net.ServerSocket.accept(ServerSocket.java:430)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "RMI TCP Accept-0" - Thread t@17
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>     - locked <6f241cf5> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at java.net.ServerSocket.accept(ServerSocket.java:430)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>     at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "jcollectd" - Thread t@13
>    java.lang.Thread.State: TIMED_WAITING
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for <6ade57c6> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>     at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>     at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>     at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Agent ServerConnection" - Thread t@11
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>     - locked <1f772f96> (a java.net.SocksSocketImpl)
>     at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>     at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>     at java.net.Socket.connect(Socket.java:529)
>     at java.net.Socket.connect(Socket.java:478)
>     at java.net.Socket.<init>(Socket.java:375)
>     at java.net.Socket.<init>(Socket.java:189)
>     at
> com.wily.isengard.postofficehub.link.net.DefaultSocketFactory.createSocket(DefaultSocketFactory.java:39)
>     at
> com.wily.isengard.api.ServerInstanceLocator.createSocket2(ServerInstanceLocator.java:353)
>     at
> com.wily.isengard.api.ServerInstanceLocator.createSocket(ServerInstanceLocator.java:320)
>     at
> com.wily.isengard.postofficehub.link.OutgoingConnection.connect(OutgoingConnection.java:71)
>     at
> com.wily.isengard.postoffice.PostOfficeHub.connectToServerHub(PostOfficeHub.java:378)
>     at
> com.wily.isengard.postoffice.PostOfficeHub.connectToServerHub(PostOfficeHub.java:337)
>     at
> com.wily.isengard.api.IsengardClient.getConnectedHub(IsengardClient.java:275)
>     at
> com.wily.isengard.api.IsengardClient.connect(IsengardClient.java:156)
>     at
> com.wily.isengard.api.IsengardClient.connect(IsengardClient.java:143)
>     at
> com.wily.introscope.agent.connection.IsengardClientConnection.connect(IsengardClientConnection.java:67)
>     at
> com.wily.introscope.agent.connection.IsengardServerConnectionManager.connectInternal(IsengardServerConnectionManager.java:646)
>     at
> com.wily.introscope.agent.connection.ConnectionThread.attemptConnection(ConnectionThread.java:141)
>     at
> com.wily.introscope.agent.connection.ConnectionThread.connect(ConnectionThread.java:260)
>     at
> com.wily.introscope.agent.connection.ConnectionThread.run(ConnectionThread.java:64)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Agent Execution" - Thread t@10
>    java.lang.Thread.State: WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <2b54befa> (a com.wily.util.adt.BlockingQueue)
>     at java.lang.Object.wait(Object.java:485)
>     at
> com.wily.util.adt.BlockingQueue.interruptableDequeue(BlockingQueue.java:123)
>     at
> com.wily.util.task.AsynchExecutionQueue.doTask(AsynchExecutionQueue.java:200)
>     at com.wily.util.task.ATask$CoreTask.run(ATask.java:132)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Agent Heartbeat" - Thread t@5
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Thread.sleep(Native Method)
>     at
> com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Remove Metric Data Watch Heartbeat Heartbeat" - Thread t@7
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Thread.sleep(Native Method)
>     at
> com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Configuration Watch Heartbeat Heartbeat" - Thread t@6
>    java.lang.Thread.State: TIMED_WAITING
>     at java.lang.Thread.sleep(Native Method)
>     at
> com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:670)
>     at java.lang.Thread.run(Thread.java:662)
>
>    Locked ownable synchronizers:
>     - None
>
> "Signal Dispatcher" - Thread t@4
>    java.lang.Thread.State: RUNNABLE
>
>    Locked ownable synchronizers:
>     - None
>
> "Finalizer" - Thread t@3
>    java.lang.Thread.State: WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <48c6254f> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>    Locked ownable synchronizers:
>     - None
>
> "Reference Handler" - Thread t@2
>    java.lang.Thread.State: WAITING
>     at java.lang.Object.wait(Native Method)
>     - waiting on <48bb8adc> (a java.lang.ref.Reference$Lock)
>     at java.lang.Object.wait(Object.java:485)
>     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>
>    Locked ownable synchronizers:
>     - None
>
> "main" - Thread t@1
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>     - locked <11dacd96> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at
> com.wily.introscope.agent.probe.net.ManagedServerSocket.com_wily_accept14(ManagedServerSocket.java:362)
>     at
> com.wily.introscope.agent.probe.net.ManagedServerSocket.accept(ManagedServerSocket.java:267)
>     at
> org.apache.catalina.core.StandardServer.await(StandardServer.java:431)
>     at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
>     at org.apache.catalina.startup.Catalina.start(Catalina.java:628)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>     at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>
>    Locked ownable synchronizers:
>     - None
>
>
>
>
> On Fri, Jul 27, 2012 at 5:19 AM, Alexandre Rafalovitch <[hidden email]
> > wrote:
>
>> Try taking a couple of thread dumps and see where in the stack the
>> snowball classes show up. That might give you a clue.
>>
>> Did you customize the parameters to the stemmer? If so, maybe it has
>> problems with the file you gave it.
>>
>> Just some generic thoughts that might help.
>>
>> Regards,
>>    Alex.
>> Personal blog: http://blog.outerthoughts.com/
>> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
>> - Time is the quality of nature that keeps events from happening all
>> at once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
>> book)
>>
>>
>> On Fri, Jul 27, 2012 at 3:53 AM, roz dev <[hidden email]> wrote:
>> > Hi All
>> >
>> > I am trying to find out the reason for very high memory use and ran JMAP
>> > -hist
>> >
>> > It is showing that i have too many instances of
>> org.tartarus.snowball.Among
>> >
>> > Any ideas what is this for and why am I getting so many of them
>> >
>> > num       #instances    #bytes  Class description
>> >
>> --------------------------------------------------------------------------
>> > *1:              46728110        1869124400
>>  org.tartarus.snowball.Among
>> > *
>> > 2:              5244210 1840458960      byte[]
>>
>
>