<no subject>

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

<no subject>

Jeff Newburn
We are having an intermittent problem with replication. We reindex nightly which usually means there are 2 commits during replication then a final commit/optimize at the end.  For some reason the replication will hang occasionally with the following screenshot.  This is frustrating as it will completely stall out any further replications. Additionally, it seems to only happen on reindex and it will strike 1 server randomly but not always the same server.


In case the screen shot doesn’t come through:

Master        http://10.66.209.38:8080/solr/zeta-main/replication
    Latest Index Version:1233423827699, Generation: 6237
    Replicatable Index Version:0, Generation: 0
Poll Interval     00:05:00
Local Index     Index Version: 1233423827684, Generation: 6222
    Location: /opt/solr-data/zeta-main/index
    Size: 1.29 GB
    Times Replicated Since Startup: 3591
    Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
    Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
    Config Files Replicated: [synonyms.txt]
    Times Config Files Replicated Since Startup: 4
    Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
    Files Downloaded: 12 / 163
    Downloaded: 4.12 MB / 1.41 GB [0.0%]
    Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
    Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163 bytes/s



--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562
Reply | Threaded
Open this post in threaded view
|

Re: <no subject>

Yonik Seeley-2-2
So this is only one slave that hangs up and not the master?
Can you get thread dumps on both the master and the slave during a hang?


-Yonik
http://www.lucidimagination.com


On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]> wrote:

> We are having an intermittent problem with replication. We reindex nightly
> which usually means there are 2 commits during replication then a final
> commit/optimize at the end.  For some reason the replication will hang
> occasionally with the following screenshot.  This is frustrating as it will
> completely stall out any further replications. Additionally, it seems to
> only happen on reindex and it will strike 1 server randomly but not always
> the same server.
>
>
> In case the screen shot doesn’t come through:
>
> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>     Latest Index Version:1233423827699, Generation: 6237
>     Replicatable Index Version:0, Generation: 0
> Poll Interval     00:05:00
> Local Index     Index Version: 1233423827684, Generation: 6222
>     Location: /opt/solr-data/zeta-main/index
>     Size: 1.29 GB
>     Times Replicated Since Startup: 3591
>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>     Config Files Replicated: [synonyms.txt]
>     Times Config Files Replicated Since Startup: 4
>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>     Files Downloaded: 12 / 163
>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
> bytes/s
>
>
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
Reply | Threaded
Open this post in threaded view
|

Re: <no subject>

Shalin Shekhar Mangar
We should obviously get to the bottom of this. But I was thinking, should we
have some sort of timeouts on the SnapPuller in the slave to avoid such
scenarios? Locking out snap pulls forever is not a good idea.

On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley <[hidden email]>wrote:

> So this is only one slave that hangs up and not the master?
> Can you get thread dumps on both the master and the slave during a hang?
>
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
> wrote:
> > We are having an intermittent problem with replication. We reindex
> nightly
> > which usually means there are 2 commits during replication then a final
> > commit/optimize at the end.  For some reason the replication will hang
> > occasionally with the following screenshot.  This is frustrating as it
> will
> > completely stall out any further replications. Additionally, it seems to
> > only happen on reindex and it will strike 1 server randomly but not
> always
> > the same server.
> >
> >
> > In case the screen shot doesn’t come through:
> >
> > Master        http://10.66.209.38:8080/solr/zeta-main/replication
> >     Latest Index Version:1233423827699, Generation: 6237
> >     Replicatable Index Version:0, Generation: 0
> > Poll Interval     00:05:00
> > Local Index     Index Version: 1233423827684, Generation: 6222
> >     Location: /opt/solr-data/zeta-main/index
> >     Size: 1.29 GB
> >     Times Replicated Since Startup: 3591
> >     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
> >     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
> >     Config Files Replicated: [synonyms.txt]
> >     Times Config Files Replicated Since Startup: 4
> >     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
> > Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
> >     Files Downloaded: 12 / 163
> >     Downloaded: 4.12 MB / 1.41 GB [0.0%]
> >     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
> >     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
> > bytes/s
> >
> >
> >
> > --
> > Jeff Newburn
> > Software Engineer, Zappos.com
> > [hidden email] - 702-943-7562
> >
>



--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: <no subject>

Noble Paul നോബിള്‍  नोब्ळ्
We do not set an conn_timeoout,read_timeout for the httpclient in snappuller.

I guess it should be set to some very high value say 1hr for
read-timeout and say 1 minute for conn_timeout and we can make it
configurable .

--Noble

On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
<[hidden email]> wrote:

> We should obviously get to the bottom of this. But I was thinking, should we
> have some sort of timeouts on the SnapPuller in the slave to avoid such
> scenarios? Locking out snap pulls forever is not a good idea.
>
> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley <[hidden email]>wrote:
>
>> So this is only one slave that hangs up and not the master?
>> Can you get thread dumps on both the master and the slave during a hang?
>>
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>> wrote:
>> > We are having an intermittent problem with replication. We reindex
>> nightly
>> > which usually means there are 2 commits during replication then a final
>> > commit/optimize at the end.  For some reason the replication will hang
>> > occasionally with the following screenshot.  This is frustrating as it
>> will
>> > completely stall out any further replications. Additionally, it seems to
>> > only happen on reindex and it will strike 1 server randomly but not
>> always
>> > the same server.
>> >
>> >
>> > In case the screen shot doesn’t come through:
>> >
>> > Master        http://10.66.209.38:8080/solr/zeta-main/replication
>> >     Latest Index Version:1233423827699, Generation: 6237
>> >     Replicatable Index Version:0, Generation: 0
>> > Poll Interval     00:05:00
>> > Local Index     Index Version: 1233423827684, Generation: 6222
>> >     Location: /opt/solr-data/zeta-main/index
>> >     Size: 1.29 GB
>> >     Times Replicated Since Startup: 3591
>> >     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>> >     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>> >     Config Files Replicated: [synonyms.txt]
>> >     Times Config Files Replicated Since Startup: 4
>> >     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>> > Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>> >     Files Downloaded: 12 / 163
>> >     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>> >     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>> >     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>> > bytes/s
>> >
>> >
>> >
>> > --
>> > Jeff Newburn
>> > Software Engineer, Zappos.com
>> > [hidden email] - 702-943-7562
>> >
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Jeff Newburn
I apologize for the delay.  The replication stalling out doesn't happen
daily.  I will paste the thread dump below to try to help.  This is on a
server that is currently locked on replication for a few hours.  Any more
information please let me know. There are no errors in the logs either so
very strange.

Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
Thread Count: current=73, peak=125, daemon=70
Full Thread Dump:
'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
user time=23940.0000ms
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at
org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
a:182)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
.java:108)
at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
22)
at
org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
808)
at
org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
)
at
org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
at
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
a:224)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
at java.lang.Thread.run(Thread.java:619)
'pool-16-thread-1' Id=237, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-190' Id=232, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-189' Id=231, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-188' Id=230, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-187' Id=229, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-186' Id=228, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-182' Id=224, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-181' Id=223, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-180' Id=222, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-179' Id=221, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-178' Id=220, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-177' Id=219, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-175' Id=217, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-173' Id=215, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-171' Id=213, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-170' Id=212, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-169' Id=211, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-168' Id=210, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-167' Id=209, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-166' Id=208, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-165' Id=207, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-164' Id=206, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-161' Id=203, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-160' Id=202, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-159' Id=201, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-158' Id=200, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-157' Id=199, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-156' Id=198, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
time=1800610.0000ms user time=1743610.0000ms
at sun.management.ThreadImpl.getThreadInfo0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
74)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
java:630)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
patcher.java:436)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:374)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
.java:302)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
273)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
879)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
ttp11NioProtocol.java:719)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
2080)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-153' Id=195, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-152' Id=194, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-150' Id=192, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-149' Id=191, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-147' Id=189, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-146' Id=188, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-145' Id=187, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-144' Id=186, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-143' Id=185, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-140' Id=182, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-137' Id=179, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-136' Id=178, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-133' Id=175, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-132' Id=174, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-131' Id=173, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-128' Id=170, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-127' Id=169, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-126' Id=165, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-125' Id=164, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-exec-5' Id=152, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
34ad83f, total cpu time=110.0000ms user time=80.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-exec-4' Id=151, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
34ad83f, total cpu time=120.0000ms user time=80.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-exec-3' Id=150, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
34ad83f, total cpu time=120.0000ms user time=80.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-exec-2' Id=149, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
34ad83f, total cpu time=110.0000ms user time=80.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-exec-1' Id=148, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
34ad83f, total cpu time=140.0000ms user time=110.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-107' Id=129, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-88' Id=110, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-65' Id=87, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-6' Id=28, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
user time=0.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
time=31410.0000ms user time=24650.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
user time=250.0000ms
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
time=764410.0000ms user time=394670.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
time=128740.0000ms user time=95550.0000ms
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
ainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
time=17960.0000ms user time=11850.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
time=261720.0000ms user time=120120.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
SocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:619)
'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:619)
'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:619)
'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
'Finalizer' Id=3, WAITING on
lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
user time=15430.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
'Reference Handler' Id=2, WAITING on
lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
time=3390.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
time=1570.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562


> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
> Reply-To: <[hidden email]>
> Date: Tue, 24 Mar 2009 14:33:50 +0530
> To: <[hidden email]>
> Subject: Re: <no subject>
>
> We do not set an conn_timeoout,read_timeout for the httpclient in snappuller.
>
> I guess it should be set to some very high value say 1hr for
> read-timeout and say 1 minute for conn_timeout and we can make it
> configurable .
>
> --Noble
>
> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
> <[hidden email]> wrote:
>> We should obviously get to the bottom of this. But I was thinking, should we
>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>> scenarios? Locking out snap pulls forever is not a good idea.
>>
>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>> <[hidden email]>wrote:
>>
>>> So this is only one slave that hangs up and not the master?
>>> Can you get thread dumps on both the master and the slave during a hang?
>>>
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>> wrote:
>>>> We are having an intermittent problem with replication. We reindex
>>> nightly
>>>> which usually means there are 2 commits during replication then a final
>>>> commit/optimize at the end.  For some reason the replication will hang
>>>> occasionally with the following screenshot.  This is frustrating as it
>>> will
>>>> completely stall out any further replications. Additionally, it seems to
>>>> only happen on reindex and it will strike 1 server randomly but not
>>> always
>>>> the same server.
>>>>
>>>>
>>>> In case the screen shot doesn’t come through:
>>>>
>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>     Replicatable Index Version:0, Generation: 0
>>>> Poll Interval     00:05:00
>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>     Location: /opt/solr-data/zeta-main/index
>>>>     Size: 1.29 GB
>>>>     Times Replicated Since Startup: 3591
>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>     Config Files Replicated: [synonyms.txt]
>>>>     Times Config Files Replicated Since Startup: 4
>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>     Files Downloaded: 12 / 163
>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>> bytes/s
>>>>
>>>>
>>>>
>>>> --
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> [hidden email] - 702-943-7562
>>>>
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>
>
> --
> --Noble Paul

Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्
I guess if we add a read-timeout on the http connection, it should
release the lock.

I will give  a patch

On Mon, Mar 30, 2009 at 7:49 PM, Jeff Newburn <[hidden email]> wrote:

> I apologize for the delay.  The replication stalling out doesn't happen
> daily.  I will paste the thread dump below to try to help.  This is on a
> server that is currently locked on replication for a few hours.  Any more
> information please let me know. There are no errors in the logs either so
> very strange.
>
> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
> Thread Count: current=73, peak=125, daemon=70
> Full Thread Dump:
> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
> user time=23940.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
> a:182)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
> .java:108)
> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
> 22)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
> 808)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
> )
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
> at
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
> a:224)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
> 101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
> odic(ScheduledThreadPoolExecutor.java:181)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
> eduledThreadPoolExecutor.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-16-thread-1' Id=237, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-190' Id=232, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-189' Id=231, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-188' Id=230, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-187' Id=229, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-186' Id=228, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-182' Id=224, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-181' Id=223, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-180' Id=222, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-179' Id=221, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-178' Id=220, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-177' Id=219, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-175' Id=217, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-173' Id=215, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-171' Id=213, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-170' Id=212, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-169' Id=211, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-168' Id=210, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-167' Id=209, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-166' Id=208, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-165' Id=207, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-164' Id=206, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-161' Id=203, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-160' Id=202, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-159' Id=201, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-158' Id=200, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-157' Id=199, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-156' Id=198, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
> time=1800610.0000ms user time=1743610.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-153' Id=195, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-152' Id=194, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-150' Id=192, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-149' Id=191, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-147' Id=189, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-146' Id=188, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-145' Id=187, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-144' Id=186, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-143' Id=185, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-140' Id=182, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-137' Id=179, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-136' Id=178, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-133' Id=175, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-132' Id=174, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-131' Id=173, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-128' Id=170, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-127' Id=169, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-126' Id=165, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-125' Id=164, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-5' Id=152, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-4' Id=151, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-3' Id=150, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-2' Id=149, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-1' Id=148, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-107' Id=129, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-88' Id=110, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-65' Id=87, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
> user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
> time=31410.0000ms user time=24650.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
> user time=250.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> time=764410.0000ms user time=394670.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
> time=128740.0000ms user time=95550.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
> time=17960.0000ms user time=11850.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
> time=261720.0000ms user time=120120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
> SocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
> user time=15430.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
> time=3390.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
> time=1570.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>> To: <[hidden email]>
>> Subject: Re: <no subject>
>>
>> We do not set an conn_timeoout,read_timeout for the httpclient in snappuller.
>>
>> I guess it should be set to some very high value say 1hr for
>> read-timeout and say 1 minute for conn_timeout and we can make it
>> configurable .
>>
>> --Noble
>>
>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>> <[hidden email]> wrote:
>>> We should obviously get to the bottom of this. But I was thinking, should we
>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>
>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>> <[hidden email]>wrote:
>>>
>>>> So this is only one slave that hangs up and not the master?
>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>> wrote:
>>>>> We are having an intermittent problem with replication. We reindex
>>>> nightly
>>>>> which usually means there are 2 commits during replication then a final
>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>> will
>>>>> completely stall out any further replications. Additionally, it seems to
>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>> always
>>>>> the same server.
>>>>>
>>>>>
>>>>> In case the screen shot doesn’t come through:
>>>>>
>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>     Replicatable Index Version:0, Generation: 0
>>>>> Poll Interval     00:05:00
>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>     Size: 1.29 GB
>>>>>     Times Replicated Since Startup: 3591
>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>     Files Downloaded: 12 / 163
>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>> bytes/s
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> [hidden email] - 702-943-7562
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
>>
>> --
>> --Noble Paul
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Yonik Seeley-2-2
In reply to this post by Jeff Newburn
Next time a hang happens, could you get a thread dump of the master also?

-Yonik
http://www.lucidimagination.com


On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:

> I apologize for the delay.  The replication stalling out doesn't happen
> daily.  I will paste the thread dump below to try to help.  This is on a
> server that is currently locked on replication for a few hours.  Any more
> information please let me know. There are no errors in the logs either so
> very strange.
>
> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
> Thread Count: current=73, peak=125, daemon=70
> Full Thread Dump:
> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
> user time=23940.0000ms
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
> a:182)
> at java.io.FilterInputStream.read(FilterInputStream.java:116)
> at
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
> .java:108)
> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
> 22)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
> 808)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
> )
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
> at
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
> a:224)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
> 101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
> odic(ScheduledThreadPoolExecutor.java:181)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
> eduledThreadPoolExecutor.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-16-thread-1' Id=237, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-190' Id=232, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-189' Id=231, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-188' Id=230, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-187' Id=229, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-186' Id=228, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-182' Id=224, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-181' Id=223, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-180' Id=222, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-179' Id=221, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-178' Id=220, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-177' Id=219, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-175' Id=217, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-173' Id=215, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-171' Id=213, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-170' Id=212, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-169' Id=211, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-168' Id=210, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-167' Id=209, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-166' Id=208, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-165' Id=207, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-164' Id=206, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-161' Id=203, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-160' Id=202, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-159' Id=201, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-158' Id=200, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-157' Id=199, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-156' Id=198, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
> time=1800610.0000ms user time=1743610.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-153' Id=195, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-152' Id=194, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-150' Id=192, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-149' Id=191, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-147' Id=189, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-146' Id=188, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-145' Id=187, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-144' Id=186, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-143' Id=185, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-140' Id=182, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-137' Id=179, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-136' Id=178, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-133' Id=175, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-132' Id=174, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-131' Id=173, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-128' Id=170, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-127' Id=169, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-126' Id=165, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-125' Id=164, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-5' Id=152, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-4' Id=151, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-3' Id=150, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-2' Id=149, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-exec-1' Id=148, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-107' Id=129, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-88' Id=110, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-65' Id=87, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
> user time=0.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
> time=31410.0000ms user time=24650.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
> user time=250.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> time=764410.0000ms user time=394670.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
> time=128740.0000ms user time=95550.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
> time=17960.0000ms user time=11850.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
> time=261720.0000ms user time=120120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
> SocketFactory.java:34)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
> .java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:619)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on
> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
> user time=15430.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
> time=3390.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
> time=1570.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>> To: <[hidden email]>
>> Subject: Re: <no subject>
>>
>> We do not set an conn_timeoout,read_timeout for the httpclient in snappuller.
>>
>> I guess it should be set to some very high value say 1hr for
>> read-timeout and say 1 minute for conn_timeout and we can make it
>> configurable .
>>
>> --Noble
>>
>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>> <[hidden email]> wrote:
>>> We should obviously get to the bottom of this. But I was thinking, should we
>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>
>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>> <[hidden email]>wrote:
>>>
>>>> So this is only one slave that hangs up and not the master?
>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>> wrote:
>>>>> We are having an intermittent problem with replication. We reindex
>>>> nightly
>>>>> which usually means there are 2 commits during replication then a final
>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>> will
>>>>> completely stall out any further replications. Additionally, it seems to
>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>> always
>>>>> the same server.
>>>>>
>>>>>
>>>>> In case the screen shot doesn’t come through:
>>>>>
>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>     Replicatable Index Version:0, Generation: 0
>>>>> Poll Interval     00:05:00
>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>     Size: 1.29 GB
>>>>>     Times Replicated Since Startup: 3591
>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>     Files Downloaded: 12 / 163
>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>> bytes/s
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> [hidden email] - 702-943-7562
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
>>
>> --
>> --Noble Paul
>
>
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Jeff Newburn
Absolutely.  I have been monitoring but it seems to only happen when a full
reindex occurs and even then intermittently.
--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562


> From: Yonik Seeley <[hidden email]>
> Reply-To: <[hidden email]>
> Date: Thu, 2 Apr 2009 15:00:28 -0400
> To: <[hidden email]>
> Subject: Re: <no subject> aka Replication Stall
>
> Next time a hang happens, could you get a thread dump of the master also?
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>> I apologize for the delay.  The replication stalling out doesn't happen
>> daily.  I will paste the thread dump below to try to help.  This is on a
>> server that is currently locked on replication for a few hours.  Any more
>> information please let me know. There are no errors in the logs either so
>> very strange.
>>
>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>> Thread Count: current=73, peak=125, daemon=70
>> Full Thread Dump:
>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>> user time=23940.0000ms
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> at
>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>> a:182)
>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>> at
>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>> .java:108)
>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>> at
>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>> 22)
>> at
>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>> 808)
>> at
>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>> )
>> at
>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>> at
>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>> a:224)
>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>> 101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>> odic(ScheduledThreadPoolExecutor.java:181)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>> eduledThreadPoolExecutor.java:205)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>> va:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 07)
>> at java.lang.Thread.run(Thread.java:619)
>> 'pool-16-thread-1' Id=237, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>> time=1800610.0000ms user time=1743610.0000ms
>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>> 74)
>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>> FilterChain.java:290)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>> ain.java:206)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>> java:630)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>> patcher.java:436)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>> er.java:374)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>> .java:302)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>> 273)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>> FilterChain.java:235)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>> ain.java:206)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>> va:233)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>> va:175)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>> )
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>> )
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>> :109)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>> at
>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>> 879)
>> at
>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>> ttp11NioProtocol.java:719)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>> 2080)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>> va:885)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 07)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-5' Id=152, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-4' Id=151, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-3' Id=150, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-2' Id=149, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-exec-1' Id=148, WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>> AbstractQueuedSynchronizer.java:1925)
>> at
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>> at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>> at java.lang.Thread.run(Thread.java:619)
>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>> user time=0.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>> at
>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>> time=31410.0000ms user time=24650.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>> user time=250.0000ms
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>> time=764410.0000ms user time=394670.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>> at java.lang.Thread.run(Thread.java:619)
>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>> time=128740.0000ms user time=95550.0000ms
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>> at java.lang.Thread.run(Thread.java:619)
>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>> ainerBase.java:1579)
>> at java.lang.Thread.run(Thread.java:619)
>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>> time=17960.0000ms user time=11850.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>> lector.java:305)
>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>> time=261720.0000ms user time=120120.0000ms
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>> lector.java:305)
>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>> SocketFactory.java:34)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>> .java:369)
>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>> at java.lang.Thread.run(Thread.java:619)
>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>> time=0.0000ms
>> 'Finalizer' Id=3, WAITING on
>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>> user time=15430.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>> 'Reference Handler' Id=2, WAITING on
>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>> time=3390.0000ms
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:485)
>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>> time=1570.0000ms
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>> )
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>> .java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>
>> --
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> [hidden email] - 702-943-7562
>>
>>
>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>> Reply-To: <[hidden email]>
>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>> To: <[hidden email]>
>>> Subject: Re: <no subject>
>>>
>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>> snappuller.
>>>
>>> I guess it should be set to some very high value say 1hr for
>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>> configurable .
>>>
>>> --Noble
>>>
>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>> <[hidden email]> wrote:
>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>> we
>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>
>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>> <[hidden email]>wrote:
>>>>
>>>>> So this is only one slave that hangs up and not the master?
>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>
>>>>>
>>>>> -Yonik
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>> wrote:
>>>>>> We are having an intermittent problem with replication. We reindex
>>>>> nightly
>>>>>> which usually means there are 2 commits during replication then a final
>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>> will
>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>> always
>>>>>> the same server.
>>>>>>
>>>>>>
>>>>>> In case the screen shot doesn’t come through:
>>>>>>
>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>> Poll Interval     00:05:00
>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>     Size: 1.29 GB
>>>>>>     Times Replicated Since Startup: 3591
>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>     Files Downloaded: 12 / 163
>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>> bytes/s
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jeff Newburn
>>>>>> Software Engineer, Zappos.com
>>>>>> [hidden email] - 702-943-7562
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्
I have raised an issue and have given a patch too
https://issues.apache.org/jira/browse/SOLR-1096

On Fri, Apr 3, 2009 at 2:00 AM, Jeff Newburn <[hidden email]> wrote:

> Absolutely.  I have been monitoring but it seems to only happen when a full
> reindex occurs and even then intermittently.
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Yonik Seeley <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>> To: <[hidden email]>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> Next time a hang happens, could you get a thread dump of the master also?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>>> I apologize for the delay.  The replication stalling out doesn't happen
>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>> server that is currently locked on replication for a few hours.  Any more
>>> information please let me know. There are no errors in the logs either so
>>> very strange.
>>>
>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>> Thread Count: current=73, peak=125, daemon=70
>>> Full Thread Dump:
>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>> user time=23940.0000ms
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> at
>>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>>> a:182)
>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>> at
>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>>> .java:108)
>>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>> at
>>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>>> 22)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>>> 808)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>>> )
>>> at
>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>>> a:224)
>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>>> odic(ScheduledThreadPoolExecutor.java:181)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:205)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'pool-16-thread-1' Id=237, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>> time=1800610.0000ms user time=1743610.0000ms
>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>>> 74)
>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:290)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>>> java:630)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>>> patcher.java:436)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>>> er.java:374)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>>> .java:302)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>> 273)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:235)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>> va:233)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>> va:175)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>> )
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>> )
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>> :109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>> at
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>> 879)
>>> at
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>> ttp11NioProtocol.java:719)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>> 2080)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-5' Id=152, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-4' Id=151, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-3' Id=150, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-2' Id=149, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-1' Id=148, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>> user time=0.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>> time=31410.0000ms user time=24650.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>> user time=250.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>> time=764410.0000ms user time=394670.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>> time=128740.0000ms user time=95550.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>>> ainerBase.java:1579)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>> time=17960.0000ms user time=11850.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>> time=261720.0000ms user time=120120.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>>> SocketFactory.java:34)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> 'Finalizer' Id=3, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>>> user time=15430.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> 'Reference Handler' Id=2, WAITING on
>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>> time=3390.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.Object.wait(Object.java:485)
>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>> time=1570.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>> )
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>> .java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>
>>> --
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> [hidden email] - 702-943-7562
>>>
>>>
>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>>> Reply-To: <[hidden email]>
>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>> To: <[hidden email]>
>>>> Subject: Re: <no subject>
>>>>
>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>> snappuller.
>>>>
>>>> I guess it should be set to some very high value say 1hr for
>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>> configurable .
>>>>
>>>> --Noble
>>>>
>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>> <[hidden email]> wrote:
>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>> we
>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>
>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>> <[hidden email]>wrote:
>>>>>
>>>>>> So this is only one slave that hangs up and not the master?
>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>>> wrote:
>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>> nightly
>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>> will
>>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>> always
>>>>>>> the same server.
>>>>>>>
>>>>>>>
>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>
>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>> Poll Interval     00:05:00
>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>     Size: 1.29 GB
>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>>> bytes/s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Newburn
>>>>>>> Software Engineer, Zappos.com
>>>>>>> [hidden email] - 702-943-7562
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shalin Shekhar Mangar.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>
>>>
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Yonik Seeley-2-2
In reply to this post by Jeff Newburn
On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <[hidden email]> wrote:
> Absolutely.  I have been monitoring but it seems to only happen when a full
> reindex occurs and even then intermittently.

Any update on this?  Have you seen any additional hangs and been able
to get a thread dump of the master?

-Yonik
http://www.lucidimagination.com



> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Yonik Seeley <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>> To: <[hidden email]>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> Next time a hang happens, could you get a thread dump of the master also?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>>> I apologize for the delay.  The replication stalling out doesn't happen
>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>> server that is currently locked on replication for a few hours.  Any more
>>> information please let me know. There are no errors in the logs either so
>>> very strange.
>>>
>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>> Thread Count: current=73, peak=125, daemon=70
>>> Full Thread Dump:
>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>> user time=23940.0000ms
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> at
>>> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.jav
>>> a:182)
>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>> at
>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream
>>> .java:108)
>>> at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>> at
>>> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:1
>>> 22)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java:
>>> 808)
>>> at
>>> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:764
>>> )
>>> at
>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.jav
>>> a:224)
>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
>>> 101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
>>> odic(ScheduledThreadPoolExecutor.java:181)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
>>> eduledThreadPoolExecutor.java:205)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'pool-16-thread-1' Id=237, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>> time=1800610.0000ms user time=1743610.0000ms
>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:3
>>> 74)
>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:290)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.
>>> java:630)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDis
>>> patcher.java:436)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
>>> er.java:374)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher
>>> .java:302)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>> 273)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>> FilterChain.java:235)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>> ain.java:206)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>> va:233)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>> va:175)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>> )
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>> )
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>> :109)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>> at
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>> 879)
>>> at
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>> ttp11NioProtocol.java:719)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>> 2080)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>> va:885)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 07)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-5' Id=152, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-4' Id=151, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-3' Id=150, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-2' Id=149, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-exec-1' Id=148, WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1
>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
>>> AbstractQueuedSynchronizer.java:1925)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5
>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>>> anos(AbstractQueuedSynchronizer.java:1963)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>> 06)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>> user time=0.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>> time=31410.0000ms user time=24650.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>> user time=250.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>> time=764410.0000ms user time=394670.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>> time=128740.0000ms user time=95550.0000ms
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
>>> ainerBase.java:1579)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>> time=17960.0000ms user time=11850.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>> time=261720.0000ms user time=120120.0000ms
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
>>> lector.java:305)
>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServer
>>> SocketFactory.java:34)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport
>>> .java:369)
>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:619)
>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>> time=0.0000ms
>>> 'Finalizer' Id=3, WAITING on
>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu time=16460.0000ms
>>> user time=15430.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>> 'Reference Handler' Id=2, WAITING on
>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>> time=3390.0000ms
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.Object.wait(Object.java:485)
>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>> time=1570.0000ms
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>> )
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>> .java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>
>>> --
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> [hidden email] - 702-943-7562
>>>
>>>
>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>>> Reply-To: <[hidden email]>
>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>> To: <[hidden email]>
>>>> Subject: Re: <no subject>
>>>>
>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>> snappuller.
>>>>
>>>> I guess it should be set to some very high value say 1hr for
>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>> configurable .
>>>>
>>>> --Noble
>>>>
>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>> <[hidden email]> wrote:
>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>> we
>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>
>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>> <[hidden email]>wrote:
>>>>>
>>>>>> So this is only one slave that hangs up and not the master?
>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>>> wrote:
>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>> nightly
>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>> will
>>>>>>> completely stall out any further replications. Additionally, it seems to
>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>> always
>>>>>>> the same server.
>>>>>>>
>>>>>>>
>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>
>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>> Poll Interval     00:05:00
>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>     Size: 1.29 GB
>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed: 163
>>>>>>> bytes/s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Newburn
>>>>>>> Software Engineer, Zappos.com
>>>>>>> [hidden email] - 702-943-7562
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shalin Shekhar Mangar.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>
>>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Jeff Newburn
It does still stall.  This time I have included the master server thread
dump.

SOLR
Solr Amin ()
Thread Dump
6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
Thread Count: current=27, peak=29, daemon=23
Full Thread Dump:
'http-8080-exec-13' Id=36, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
a sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
user time=1010960.0000ms
at sun.management.ThreadImpl.getThreadInfo0(Native Method)
at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
74)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
t
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
java:630)
at
org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
patcher.java:436)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
er.java:374)
at
org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
.java:302)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
273)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:175)
at
org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
879)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
ttp11NioProtocol.java:719)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
2080)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
at java.lang.Thread.run(Thread.jav:619)
'http-8080-exec-11' Id=34, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-10' Id=33, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
at sun.mis.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbsractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.ang.Thread.run(Thread.java:619)
'http-8080-exec-9' Id=32, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
at sun.misc.Unsaf.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-8' Id=29, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-7' Id=28, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-6' Id=26, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-5' Id=24, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-4' Id=23, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
at sun.miscUnsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-3' Id=22, TIMED_WAITING on
lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-2' Id=21, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8080-exec-1' Id=19, TIMED_WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
tim=6480.0000ms user time=2900.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
user time=0.0000ms
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
time=83050.0000ms user time=36360.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.jav:619)
'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
user time=11940.0000s
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
ainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
'pool-5-thread-1' Id=13, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'pool-3-thread-1' Id=12, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'pool-1-thread-1' Id=11, WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
time=5580.0000ms user time=2120.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
time=333280.0000ms user time=107520.0000ms
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
lector.java:305)
'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
time=0.0000ms
'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
total cpu time=4900.0000ms user time=940.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
'Reference Handler' Id=2, WAITING on
lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
time=440.0000ms
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
time=11180.0000ms
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Return to Admin Page
--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562


> From: Yonik Seeley <[hidden email]>
> Reply-To: <[hidden email]>
> Date: Mon, 4 May 2009 16:07:50 -0400
> To: <[hidden email]>
> Subject: Re: <no subject> aka Replication Stall
>
> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <[hidden email]> wrote:
>> Absolutely.  I have been monitoring but it seems to only happen when a full
>> reindex occurs and even then intermittently.
>
> Any update on this?  Have you seen any additional hangs and been able
> to get a thread dump of the master?
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> [hidden email] - 702-943-7562
>>
>>
>>> From: Yonik Seeley <[hidden email]>
>>> Reply-To: <[hidden email]>
>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>> To: <[hidden email]>
>>> Subject: Re: <no subject> aka Replication Stall
>>>
>>> Next time a hang happens, could you get a thread dump of the master also?
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>>>> I apologize for the delay.  The replication stalling out doesn't happen
>>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>>> server that is currently locked on replication for a few hours.  Any more
>>>> information please let me know. There are no errors in the logs either so
>>>> very strange.
>>>>
>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>> Thread Count: current=73, peak=125, daemon=70
>>>> Full Thread Dump:
>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>> user time=23940.0000ms
>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>> at
>>>>
org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
v
>>>> a:182)
>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>> at
>>>>
org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
m
>>>> .java:108)
>>>> at
>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>> at
>>>>
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
1
>>>> 22)
>>>> at
>>>>
org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
:
>>>> 808)
>>>> at
>>>>
org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
4
>>>> )
>>>> at
>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>> at
>>>>
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
v
>>>> a:224)
>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
$
>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
i
>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>> at
>>>>
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
h
>>>> eduledThreadPoolExecutor.java:205)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
a
>>>> va:885)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 07)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9

>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>> time=1800610.0000ms user time=1743610.0000ms
>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>> at
>>>>
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
3
>>>> 74)
>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
n
>>>> FilterChain.java:290)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
h
>>>> ain.java:206)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
.
>>>> java:630)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
s
>>>> patcher.java:436)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
h
>>>> er.java:374)
>>>> at
>>>>
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
r
>>>> .java:302)
>>>> at
>>>>
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
:
>>>> 273)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
n
>>>> FilterChain.java:235)
>>>> at
>>>>
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
h
>>>> ain.java:206)
>>>> at
>>>>
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
a
>>>> va:233)
>>>> at
>>>>
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
a
>>>> va:175)
>>>> at
>>>>
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
8
>>>> )
>>>> at
>>>>
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
2
>>>> )
>>>> at
>>>>
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
a
>>>> :109)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>> at
>>>>
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
:
>>>> 879)
>>>> at
>>>>
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
H
>>>> ttp11NioProtocol.java:719)
>>>> at
>>>>
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
:
>>>> 2080)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
a
>>>> va:885)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 07)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
1
>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
(
>>>> AbstractQueuedSynchronizer.java:1925)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9
>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
5
>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>> at
>>>>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
N
>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
)
>>>> at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
9

>>>> 06)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>> user time=0.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>> at
>>>>
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
u

>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>> time=31410.0000ms user time=24650.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>> user time=250.0000ms
>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>> time=764410.0000ms user time=394670.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>> time=128740.0000ms user time=95550.0000ms
>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
t

>>>> ainerBase.java:1579)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>> time=17960.0000ms user time=11850.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at
>>>>
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
e

>>>> lector.java:305)
>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>> time=261720.0000ms user time=120120.0000ms
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at
>>>>
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
e
>>>> lector.java:305)
>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
r
>>>> SocketFactory.java:34)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t

>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>> user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t

>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at
>>>>
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
t

>>>> .java:369)
>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>> time=0.0000ms
>>>> 'Finalizer' Id=3, WAITING on
>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>> time=16460.0000ms
>>>> user time=15430.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>> 'Reference Handler' Id=2, WAITING on
>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>> time=3390.0000ms
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>> time=1570.0000ms
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
9
>>>> )
>>>> at
>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
l

>>>> .java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>
>>>> --
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> [hidden email] - 702-943-7562
>>>>
>>>>
>>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>>>> Reply-To: <[hidden email]>
>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>> To: <[hidden email]>
>>>>> Subject: Re: <no subject>
>>>>>
>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>> snappuller.
>>>>>
>>>>> I guess it should be set to some very high value say 1hr for
>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>> configurable .
>>>>>
>>>>> --Noble
>>>>>
>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>> <[hidden email]> wrote:
>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>> we
>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>
>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>> <[hidden email]>wrote:
>>>>>>
>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>
>>>>>>>
>>>>>>> -Yonik
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>>>> wrote:
>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>> nightly
>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>>> will
>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>> to
>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>> always
>>>>>>>> the same server.
>>>>>>>>
>>>>>>>>
>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>
>>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>>> Poll Interval     00:05:00
>>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>>     Size: 1.29 GB
>>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>> 163
>>>>>>>> bytes/s
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jeff Newburn
>>>>>>>> Software Engineer, Zappos.com
>>>>>>>> [hidden email] - 702-943-7562
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>
>>>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्
hi,
Jeff , did you use the fix for SOLR-1096 annd you still see that it hangs?

is the client stack trace same then?

We have set it to timeout after 10 secs

--Noble

On Tue, May 5, 2009 at 9:12 PM, Jeff Newburn <[hidden email]> wrote:

> It does still stall.  This time I have included the master server thread
> dump.
>
> SOLR
> Solr Amin ()
> Thread Dump
> 6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
> Thread Count: current=27, peak=29, daemon=23
> Full Thread Dump:
> 'http-8080-exec-13' Id=36, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
> a sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
> user time=1010960.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-exec-11' Id=34, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
> d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-10' Id=33, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
> d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
> at sun.mis.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbsractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.ang.Thread.run(Thread.java:619)
> 'http-8080-exec-9' Id=32, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
> at sun.misc.Unsaf.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-8' Id=29, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
> d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-7' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=26, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-5' Id=24, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-4' Id=23, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
> at sun.miscUnsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-3' Id=22, TIMED_WAITING on
> lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-2' Id=21, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-1' Id=19, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> tim=6480.0000ms user time=2900.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
> time=83050.0000ms user time=36360.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
> user time=11940.0000s
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
> TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-5-thread-1' Id=13, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
> 5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-3-thread-1' Id=12, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
> 5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-1-thread-1' Id=11, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
> 6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
> time=5580.0000ms user time=2120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
> time=333280.0000ms user time=107520.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
> total cpu time=4900.0000ms user time=940.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
> time=440.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
> time=11180.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Return to Admin Page
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Yonik Seeley <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Mon, 4 May 2009 16:07:50 -0400
>> To: <[hidden email]>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <[hidden email]> wrote:
>>> Absolutely.  I have been monitoring but it seems to only happen when a full
>>> reindex occurs and even then intermittently.
>>
>> Any update on this?  Have you seen any additional hangs and been able
>> to get a thread dump of the master?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> [hidden email] - 702-943-7562
>>>
>>>
>>>> From: Yonik Seeley <[hidden email]>
>>>> Reply-To: <[hidden email]>
>>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>>> To: <[hidden email]>
>>>> Subject: Re: <no subject> aka Replication Stall
>>>>
>>>> Next time a hang happens, could you get a thread dump of the master also?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>>>>> I apologize for the delay.  The replication stalling out doesn't happen
>>>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>>>> server that is currently locked on replication for a few hours.  Any more
>>>>> information please let me know. There are no errors in the logs either so
>>>>> very strange.
>>>>>
>>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>>> Thread Count: current=73, peak=125, daemon=70
>>>>> Full Thread Dump:
>>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>>> user time=23940.0000ms
>>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at
>>>>>
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
> v
>>>>> a:182)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>>> at
>>>>>
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
> m
>>>>> .java:108)
>>>>> at
>>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>>> at
>>>>>
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
> 1
>>>>> 22)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
> :
>>>>> 808)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
> 4
>>>>> )
>>>>> at
>>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>>> at
>>>>>
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
> v
>>>>> a:224)
>>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
> $
>>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
> i
>>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
> h
>>>>> eduledThreadPoolExecutor.java:205)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>>> time=1800610.0000ms user time=1743610.0000ms
>>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
> 3
>>>>> 74)
>>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:290)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
> .
>>>>> java:630)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
> s
>>>>> patcher.java:436)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
> h
>>>>> er.java:374)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
> r
>>>>> .java:302)
>>>>> at
>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
> :
>>>>> 273)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:235)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
> a
>>>>> va:233)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
> a
>>>>> va:175)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
> 8
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
> 2
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
> a
>>>>> :109)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
> :
>>>>> 879)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
> H
>>>>> ttp11NioProtocol.java:719)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
> :
>>>>> 2080)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>>> user time=0.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at
>>>>>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
> u
>>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>>> time=31410.0000ms user time=24650.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>>> user time=250.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>>> time=764410.0000ms user time=394670.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>>> time=128740.0000ms user time=95550.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
> t
>>>>> ainerBase.java:1579)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>>> time=17960.0000ms user time=11850.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>>> time=261720.0000ms user time=120120.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
> r
>>>>> SocketFactory.java:34)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>>> user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> 'Finalizer' Id=3, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>>> time=16460.0000ms
>>>>> user time=15430.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 'Reference Handler' Id=2, WAITING on
>>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>>> time=3390.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>>> time=1570.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
> 9
>>>>> )
>>>>> at
>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
> l
>>>>> .java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> [hidden email] - 702-943-7562
>>>>>
>>>>>
>>>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>>>>> Reply-To: <[hidden email]>
>>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>>> To: <[hidden email]>
>>>>>> Subject: Re: <no subject>
>>>>>>
>>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>>> snappuller.
>>>>>>
>>>>>> I guess it should be set to some very high value say 1hr for
>>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>>> configurable .
>>>>>>
>>>>>> --Noble
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>>> <[hidden email]> wrote:
>>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>>> we
>>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>>> <[hidden email]>wrote:
>>>>>>>
>>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Yonik
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>>> nightly
>>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>>>> will
>>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>>> to
>>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>>> always
>>>>>>>>> the same server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>>
>>>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>>>> Poll Interval     00:05:00
>>>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>>>     Size: 1.29 GB
>>>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>>> 163
>>>>>>>>> bytes/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> [hidden email] - 702-943-7562
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>
>>>>>
>>>
>>>
>
>



--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्-2
In reply to this post by Jeff Newburn
hi Jeff,
Did you observe a hang even after applying SOLR-1096
(https://issues.apache.org/jira/browse/SOLR-1096) ? This change gets
applied only on the slave . So you would get the benefit only if the
slave is updated with a new build


On Tue, May 5, 2009 at 9:12 PM, Jeff Newburn <[hidden email]> wrote:

> It does still stall.  This time I have included the master server thread
> dump.
>
> SOLR
> Solr Amin ()
> Thread Dump
> 6pm (6pm-main) Java HotSpot(TM) 64-Bit Server VM10.0-b19
> Thread Count: current=27, peak=29, daemon=23
> Full Thread Dump:
> 'http-8080-exec-13' Id=36, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueueSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1156740.0000ms user time=1020290.0000ms
> a sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolxecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-12' Id=35, RUNNABLE on lock=, total cpu time=1150940.0000ms
> user time=1010960.0000ms
> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
> at sun.mnagement.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
> at org.apache.jsp.admin.threaddump_jsp._jpService(threaddump_jsp.java:261)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.jaser.servlet.JspServletWrapper.service(JspServletWrapper.java:3
> 74)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> t
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.AplicationDispatcher.invoke(ApplicationDispatcher.
> java:630)
> at
> org.apache.catalina.core.ApplicatonDispatcher.processRequest(ApplicationDis
> patcher.java:436)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatch
> er.java:374)
> at
> org.apache.catalina.core.ApplicatioDispatcher.forward(ApplicationDispatcher
> .java:302)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 273)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
> FilterChain.java:235)
> at
> org.apache.catalina.core.ApplicationFilterCain.doFilter(ApplicationFilterCh
> ain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
> va:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
> va:175)
> at
> org.pache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
> )
> at
> org.apache.catalin.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
> )
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
> :109)
> at
> org.apache.catalina.connector.CoyoteAdapte.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
> 879)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
> ttp11NioProtocol.java:719)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.ru(NioEndpoint.java:
> 2080)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-exec-11' Id=34, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedynchronizer$ConditionObject@4
> d602cfb, total cpu time=1155200.0000ms user time=999270.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupprt.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ava:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-10' Id=33, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronzer$ConditionObject@4
> d602cfb, total cpu time=1223520.0000ms user time=1044630.0000ms
> at sun.mis.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbsractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.ang.Thread.run(Thread.java:619)
> 'http-8080-exec-9' Id=32, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1191400.0000ms user time=1036590.0000ms
> at sun.misc.Unsaf.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-8' Id=29, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchrnizer$ConditionObject@4
> d602cfb, total cpu time=1278320.0000ms user time=1083490.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-7' Id=28, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1268250.0000ms user time=1095880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-6' Id=26, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1255140.0000ms user time=1068550.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-5' Id=24, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1331170.0000ms user time=1152940.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(ockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-4' Id=23, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1405400.0000ms user time=1234250.0000ms
> at sun.miscUnsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-3' Id=22, TIMED_WAITING on
> lock=java.util.concurren.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1430550.0000ms user time=1231000.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPooExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-2' Id=21, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1395590.0000ms user time=121336.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTak(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-exec-1' Id=19, TIMED_WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4
> d602cfb, total cpu time=1439010.0000ms user time=1245880.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
> at
> java.til.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
> tim=6480.0000ms user time=2900.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8443-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu time=0.0000ms
> user time=0.0000ms
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'http-8080-ClientPoller' Id=16, RUNNABLE on lock=, total cpu
> time=83050.0000ms user time=36360.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.jav:619)
> 'http-8080-Acceptor-0' Id=15, RUNNABLE on lock=, total cpu time=27840.0000ms
> user time=11940.0000s
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=14,
> TIMED_WAITING on lock=, total cpu time=20320.0000ms user time=7580.0000ms
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Cont
> ainerBase.java:1579)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-5-thread-1' Id=13, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6
> 5293a6c, total cpu time=43550.0000ms user time=39790.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-3-thread-1' Id=12, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7
> 5babc56, total cpu time=2196520.0000ms user time=2041120.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'pool-1-thread-1' Id=11, WAITING on
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$onditionObject@3
> 6a2ee7c, total cpu time=2093540.0000ms user time=1941900.0000ms
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
> at java.lang.Thread.run(Thread.java:619)
> 'NioBlockingSelector.BlockPoller-2' Id=10, RUNNABLE on lock=, total cpu
> time=5580.0000ms user time=2120.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'NioBlockingSelector.BlockPoller-1' Id=9, RUNNABLE on lock=, total cpu
> time=333280.0000ms user time=107520.0000ms
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollrrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSe
> lector.java:305)
> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
> time=0.0000ms
> 'Finalizer' Id=3, WAITING on lock=java.lang.ref.ReferenceQueue$Lock@2d72aea,
> total cpu time=4900.0000ms user time=940.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 'Reference Handler' Id=2, WAITING on
> lock=java.lang.ref.Reference$Lock@7c426a1a, total cpu time=1380.0000ms user
> time=440.0000ms
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> 'main' Id=1, RUNNABLE on lock=, total cpu time=12080.0000ms user
> time=11180.0000ms
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Return to Admin Page
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Yonik Seeley <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Mon, 4 May 2009 16:07:50 -0400
>> To: <[hidden email]>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> On Thu, Apr 2, 2009 at 4:30 PM, Jeff Newburn <[hidden email]> wrote:
>>> Absolutely.  I have been monitoring but it seems to only happen when a full
>>> reindex occurs and even then intermittently.
>>
>> Any update on this?  Have you seen any additional hangs and been able
>> to get a thread dump of the master?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>> Jeff Newburn
>>> Software Engineer, Zappos.com
>>> [hidden email] - 702-943-7562
>>>
>>>
>>>> From: Yonik Seeley <[hidden email]>
>>>> Reply-To: <[hidden email]>
>>>> Date: Thu, 2 Apr 2009 15:00:28 -0400
>>>> To: <[hidden email]>
>>>> Subject: Re: <no subject> aka Replication Stall
>>>>
>>>> Next time a hang happens, could you get a thread dump of the master also?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>> On Mon, Mar 30, 2009 at 10:19 AM, Jeff Newburn <[hidden email]> wrote:
>>>>> I apologize for the delay.  The replication stalling out doesn't happen
>>>>> daily.  I will paste the thread dump below to try to help.  This is on a
>>>>> server that is currently locked on replication for a few hours.  Any more
>>>>> information please let me know. There are no errors in the logs either so
>>>>> very strange.
>>>>>
>>>>> Zappos Zeta (zeta-main) Java HotSpot(TM) 64-Bit Server VM10.0-b23
>>>>> Thread Count: current=73, peak=125, daemon=70
>>>>> Full Thread Dump:
>>>>> 'pool-18-thread-1' Id=238, RUNNABLE on lock=, total cpu time=59740.0000ms
>>>>> user time=23940.0000ms
>>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>>>> at
>>>>>
> org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.ja>>>>
> v
>>>>> a:182)
>>>>> at java.io.FilterInputStream.read(FilterInputStream.java:116)
>>>>> at
>>>>>
> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStrea>>>>
> m
>>>>> .java:108)
>>>>> at
>>>>> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:91)
>>>>> at
>>>>>
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:>>>>
> 1
>>>>> 22)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchPackets(SnapPuller.java>>>>
> :
>>>>> 808)
>>>>> at
>>>>>
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:76>>>>
> 4
>>>>> )
>>>>> at
>>>>> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:419)
>>>>> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:255)
>>>>> at
>>>>>
> org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.ja>>>>
> v
>>>>> a:224)
>>>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:139)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access>>>>
> $
>>>>> 101(ScheduledThreadPoolExecutor.java:98)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPer>>>>
> i
>>>>> odic(ScheduledThreadPoolExecutor.java:181)
>>>>> at
>>>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sc>>>>
> h
>>>>> eduledThreadPoolExecutor.java:205)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'pool-16-thread-1' Id=237, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> ff3438a, total cpu time=14952560.0000ms user time=14177150.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-190' Id=232, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1035270.0000ms user time=1001520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-189' Id=231, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1483590.0000ms user time=1437890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-188' Id=230, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486880.0000ms user time=1440520.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-187' Id=229, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1480360.0000ms user time=1432980.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-186' Id=228, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1486080.0000ms user time=1438720.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-182' Id=224, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1490900.0000ms user time=1443170.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-181' Id=223, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1478140.0000ms user time=1432910.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-180' Id=222, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799690.0000ms user time=1742020.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-179' Id=221, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799030.0000ms user time=1742220.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-178' Id=220, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817820.0000ms user time=1761810.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-177' Id=219, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803700.0000ms user time=1746890.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-175' Id=217, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1799850.0000ms user time=1743330.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-173' Id=215, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816300.0000ms user time=1759300.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-171' Id=213, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800290.0000ms user time=1743860.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-170' Id=212, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813310.0000ms user time=1757000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-169' Id=211, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808820.0000ms user time=1752140.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-168' Id=210, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1803510.0000ms user time=1746610.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-167' Id=209, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798590.0000ms user time=1742670.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-166' Id=208, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797140.0000ms user time=1740450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-165' Id=207, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810700.0000ms user time=1754510.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-164' Id=206, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1797940.0000ms user time=1740380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-161' Id=203, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806530.0000ms user time=1749990.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-160' Id=202, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1796680.0000ms user time=1739460.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-159' Id=201, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1813020.0000ms user time=1755560.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-158' Id=200, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1814620.0000ms user time=1756600.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-157' Id=199, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1794320.0000ms user time=1737130.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-156' Id=198, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1798210.0000ms user time=1741090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-154' Id=196, RUNNABLE on lock=, total cpu
>>>>> time=1800610.0000ms user time=1743610.0000ms
>>>>> at sun.management.ThreadImpl.getThreadInfo0(Native Method)
>>>>> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
>>>>> at org.apache.jsp.admin.threaddump_jsp._jspService(threaddump_jsp.java:261)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:>>>>
> 3
>>>>> 74)
>>>>> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:290)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher>>>>
> .
>>>>> java:630)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDi>>>>
> s
>>>>> patcher.java:436)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatc>>>>
> h
>>>>> er.java:374)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatche>>>>
> r
>>>>> .java:302)
>>>>> at
>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java>>>>
> :
>>>>> 273)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio>>>>
> n
>>>>> FilterChain.java:235)
>>>>> at
>>>>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC>>>>
> h
>>>>> ain.java:206)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j>>>>
> a
>>>>> va:233)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j>>>>
> a
>>>>> va:175)
>>>>> at
>>>>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12>>>>
> 8
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10>>>>
> 2
>>>>> )
>>>>> at
>>>>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav>>>>
> a
>>>>> :109)
>>>>> at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java>>>>
> :
>>>>> 879)
>>>>> at
>>>>>
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(>>>>
> H
>>>>> ttp11NioProtocol.java:719)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java>>>>
> :
>>>>> 2080)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j>>>>
> a
>>>>> va:885)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 07)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-153' Id=195, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1817100.0000ms user time=1759650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-152' Id=194, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811010.0000ms user time=1754450.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-150' Id=192, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1810480.0000ms user time=1753650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-149' Id=191, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1818660.0000ms user time=1761230.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-147' Id=189, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1815180.0000ms user time=1758160.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-146' Id=188, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1808550.0000ms user time=1751850.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-145' Id=187, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807340.0000ms user time=1751380.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-144' Id=186, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1795230.0000ms user time=1740000.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-143' Id=185, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1788870.0000ms user time=1733030.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-140' Id=182, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1800590.0000ms user time=1744650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-137' Id=179, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1807880.0000ms user time=1749660.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-136' Id=178, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1816230.0000ms user time=1759630.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-133' Id=175, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806840.0000ms user time=1751190.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-132' Id=174, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1811350.0000ms user time=1755120.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-131' Id=173, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1802280.0000ms user time=1746550.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-128' Id=170, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1793930.0000ms user time=1738310.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-127' Id=169, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=1806640.0000ms user time=1750650.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-126' Id=165, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3102840.0000ms user time=3008570.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-125' Id=164, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3140270.0000ms user time=3042210.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-5' Id=152, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-4' Id=151, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-3' Id=150, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=120.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-2' Id=149, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=110.0000ms user time=80.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-exec-1' Id=148, WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 1
>>>>> 34ad83f, total cpu time=140.0000ms user time=110.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> (
>>>>> AbstractQueuedSynchronizer.java:1925)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-107' Id=129, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3702320.0000ms user time=3587430.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-88' Id=110, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3684860.0000ms user time=3571250.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-65' Id=87, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=3682930.0000ms user time=3568270.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-exec-6' Id=28, TIMED_WAITING on
>>>>>
> lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@>>>>
> 5
>>>>> bd74a3c, total cpu time=4058340.0000ms user time=3932090.0000ms
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>>>> at
>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await>>>>
> N
>>>>> anos(AbstractQueuedSynchronizer.java:1963)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944>>>>
> )
>>>>> at
>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:>>>>
> 9
>>>>> 06)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'MultiThreadedHttpConnectionManager cleanup' Id=27, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@16bb19fe, total cpu time=0.0000ms
>>>>> user time=0.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at
>>>>>
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQ>>>>
> u
>>>>> eueThread.run(MultiThreadedHttpConnectionManager.java:1122)
>>>>> 'http-8443-ClientPoller' Id=24, RUNNABLE on lock=, total cpu
>>>>> time=31410.0000ms user time=24650.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8443-Acceptor-0' Id=23, RUNNABLE on lock=, total cpu time=350.0000ms
>>>>> user time=250.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-ClientPoller' Id=18, RUNNABLE on lock=, total cpu
>>>>> time=764410.0000ms user time=394670.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'http-8080-Acceptor-0' Id=17, RUNNABLE on lock=, total cpu
>>>>> time=128740.0000ms user time=95550.0000ms
>>>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>> at
>>>>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>>>>> at
>>>>> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'ContainerBackgroundProcessor[StandardEngine[Catalina]]' Id=16,
>>>>> TIMED_WAITING on lock=, total cpu time=26790.0000ms user time=18560.0000ms
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(Con>>>>
> t
>>>>> ainerBase.java:1579)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'NioBlockingSelector.BlockPoller-2' Id=13, RUNNABLE on lock=, total cpu
>>>>> time=17960.0000ms user time=11850.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'NioBlockingSelector.BlockPoller-1' Id=12, RUNNABLE on lock=, total cpu
>>>>> time=261720.0000ms user time=120120.0000ms
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>> at
>>>>>
> org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingS>>>>
> e
>>>>> lector.java:305)
>>>>> 'RMI TCP Accept-0' Id=11, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServe>>>>
> r
>>>>> SocketFactory.java:34)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-7009' Id=10, RUNNABLE on lock=, total cpu time=0.0000ms
>>>>> user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'RMI TCP Accept-0' Id=9, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at
>>>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspor>>>>
> t
>>>>> .java:369)
>>>>> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 'Signal Dispatcher' Id=4, RUNNABLE on lock=, total cpu time=0.0000ms user
>>>>> time=0.0000ms
>>>>> 'Finalizer' Id=3, WAITING on
>>>>> lock=java.lang.ref.ReferenceQueue$Lock@55a77e5a, total cpu
>>>>> time=16460.0000ms
>>>>> user time=15430.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>> 'Reference Handler' Id=2, WAITING on
>>>>> lock=java.lang.ref.Reference$Lock@675872a4, total cpu time=4630.0000ms user
>>>>> time=3390.0000ms
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>>> 'main' Id=1, RUNNABLE on lock=, total cpu time=1700.0000ms user
>>>>> time=1570.0000ms
>>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>> at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>> at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
>>>>> at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
>>>>> at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3>>>>
> 9
>>>>> )
>>>>> at
>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp>>>>
> l
>>>>> .java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
>>>>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
>>>>>
>>>>> --
>>>>> Jeff Newburn
>>>>> Software Engineer, Zappos.com
>>>>> [hidden email] - 702-943-7562
>>>>>
>>>>>
>>>>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>>>>> Reply-To: <[hidden email]>
>>>>>> Date: Tue, 24 Mar 2009 14:33:50 +0530
>>>>>> To: <[hidden email]>
>>>>>> Subject: Re: <no subject>
>>>>>>
>>>>>> We do not set an conn_timeoout,read_timeout for the httpclient in
>>>>>> snappuller.
>>>>>>
>>>>>> I guess it should be set to some very high value say 1hr for
>>>>>> read-timeout and say 1 minute for conn_timeout and we can make it
>>>>>> configurable .
>>>>>>
>>>>>> --Noble
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 2:13 PM, Shalin Shekhar Mangar
>>>>>> <[hidden email]> wrote:
>>>>>>> We should obviously get to the bottom of this. But I was thinking, should
>>>>>>> we
>>>>>>> have some sort of timeouts on the SnapPuller in the slave to avoid such
>>>>>>> scenarios? Locking out snap pulls forever is not a good idea.
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 8:57 PM, Yonik Seeley
>>>>>>> <[hidden email]>wrote:
>>>>>>>
>>>>>>>> So this is only one slave that hangs up and not the master?
>>>>>>>> Can you get thread dumps on both the master and the slave during a hang?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Yonik
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 10:44 AM, Jeff Newburn <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>> We are having an intermittent problem with replication. We reindex
>>>>>>>> nightly
>>>>>>>>> which usually means there are 2 commits during replication then a final
>>>>>>>>> commit/optimize at the end.  For some reason the replication will hang
>>>>>>>>> occasionally with the following screenshot.  This is frustrating as it
>>>>>>>> will
>>>>>>>>> completely stall out any further replications. Additionally, it seems
>>>>>>>>> to
>>>>>>>>> only happen on reindex and it will strike 1 server randomly but not
>>>>>>>> always
>>>>>>>>> the same server.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In case the screen shot doesn’t come through:
>>>>>>>>>
>>>>>>>>> Master        http://10.66.209.38:8080/solr/zeta-main/replication
>>>>>>>>>     Latest Index Version:1233423827699, Generation: 6237
>>>>>>>>>     Replicatable Index Version:0, Generation: 0
>>>>>>>>> Poll Interval     00:05:00
>>>>>>>>> Local Index     Index Version: 1233423827684, Generation: 6222
>>>>>>>>>     Location: /opt/solr-data/zeta-main/index
>>>>>>>>>     Size: 1.29 GB
>>>>>>>>>     Times Replicated Since Startup: 3591
>>>>>>>>>     Previous Replication Done At: Mon Mar 23 00:18:03 PDT 2009
>>>>>>>>>     Config Files Replicated At: Wed Mar 18 06:07:53 PDT 2009
>>>>>>>>>     Config Files Replicated: [synonyms.txt]
>>>>>>>>>     Times Config Files Replicated Since Startup: 4
>>>>>>>>>     Next Replication Cycle At: Mon Mar 23 00:27:55 PDT 2009
>>>>>>>>> Current Replication Status     Start Time: Mon Mar 23 00:22:55 PDT 2009
>>>>>>>>>     Files Downloaded: 12 / 163
>>>>>>>>>     Downloaded: 4.12 MB / 1.41 GB [0.0%]
>>>>>>>>>     Downloading File: _5no.tis, Downloaded: 0 bytes / 629.57 KB [0.0%]
>>>>>>>>>     Time Elapsed: 26371s, Estimated Time Remaining: 9216278s, Speed:
>>>>>>>>> 163
>>>>>>>>> bytes/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeff Newburn
>>>>>>>>> Software Engineer, Zappos.com
>>>>>>>>> [hidden email] - 702-943-7562
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>
>>>>>
>>>
>>>
>
>



--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Jeff Newburn
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्
We have not pushed the fix into production yet.  However, I am wondering two
things. 1. If the download takes more than 10 seconds (our replication can
take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
2 line changes 1 has a large amount. Do we need the latest 2 or just the
latest 1?

--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562


> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
> Reply-To: <[hidden email]>
> Date: Wed, 6 May 2009 10:05:49 +0530
> To: <[hidden email]>
> Subject: Re: <no subject> aka Replication Stall
>
> SOLR-1096

Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्-2
the patches have gone into the trunk. The latest patch should be the
one if you wish to run a patched Solr.

10 secs readTimeout means that if there is no data coming from the
other end for 10 secs, then the waiting thread returns throwing an
exception. It is not the total time taken to read the entire data. At
least that is what I observed while testing.

BTW, if the timeout occurs it resumes from the point where the failure
happened. It retries 5 times before giving up.

On Thu, May 7, 2009 at 7:32 PM, Jeff Newburn <[hidden email]> wrote:

> We have not pushed the fix into production yet.  However, I am wondering two
> things. 1. If the download takes more than 10 seconds (our replication can
> take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
> 2 line changes 1 has a large amount. Do we need the latest 2 or just the
> latest 1?
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> [hidden email] - 702-943-7562
>
>
>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>> Reply-To: <[hidden email]>
>> Date: Wed, 6 May 2009 10:05:49 +0530
>> To: <[hidden email]>
>> Subject: Re: <no subject> aka Replication Stall
>>
>> SOLR-1096
>
>



--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Jeff Newburn
Excellent! Thank you I am going to start testing that.
--
Jeff Newburn
Software Engineer, Zappos.com
[hidden email] - 702-943-7562


> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
> Reply-To: <[hidden email]>
> Date: Thu, 7 May 2009 20:26:02 +0530
> To: <[hidden email]>
> Subject: Re: <no subject> aka Replication Stall
>
> the patches have gone into the trunk. The latest patch should be the
> one if you wish to run a patched Solr.
>
> 10 secs readTimeout means that if there is no data coming from the
> other end for 10 secs, then the waiting thread returns throwing an
> exception. It is not the total time taken to read the entire data. At
> least that is what I observed while testing.
>
> BTW, if the timeout occurs it resumes from the point where the failure
> happened. It retries 5 times before giving up.
>
> On Thu, May 7, 2009 at 7:32 PM, Jeff Newburn <[hidden email]> wrote:
>> We have not pushed the fix into production yet.  However, I am wondering two
>> things. 1. If the download takes more than 10 seconds (our replication can
>> take up to 90 seconds) will that be an issue 2. There are 3 patches, 2 have
>> 2 line changes 1 has a large amount. Do we need the latest 2 or just the
>> latest 1?
>>
>> --
>> Jeff Newburn
>> Software Engineer, Zappos.com
>> [hidden email] - 702-943-7562
>>
>>
>>> From: Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>
>>> Reply-To: <[hidden email]>
>>> Date: Wed, 6 May 2009 10:05:49 +0530
>>> To: <[hidden email]>
>>> Subject: Re: <no subject> aka Replication Stall
>>>
>>> SOLR-1096
>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com

Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Yonik Seeley-2-2
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्-2
2009/5/7 Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>:
> BTW, if the timeout occurs it resumes from the point where the failure
> happened. It retries 5 times before giving up.

Sweet!  I was just going to ask what happens on a timeout.
Have you tested this out (say kill the master in the middle of
replicating a big file)?

-Yonik
http://www.lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: <no subject> aka Replication Stall

Noble Paul നോബിള്‍  नोब्ळ्-2
On Sat, May 9, 2009 at 2:23 AM, Yonik Seeley <[hidden email]> wrote:
> 2009/5/7 Noble Paul നോബിള്‍  नोब्ळ् <[hidden email]>:
>> BTW, if the timeout occurs it resumes from the point where the failure
>> happened. It retries 5 times before giving up.
>
> Sweet!  I was just going to ask what happens on a timeout.
> Have you tested this out (say kill the master in the middle of
> replicating a big file)?

Actually yes,

not by killing the master (then the replication will abort). I
modified the master code to close the connection after transferring x
MB of data. The slave retried an completed the replication .
>
> -Yonik
> http://www.lucidimagination.com
>



--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com