DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Jayson Minard
With autocommit, I am seeing threads pile up behind the CommitTracker scheduleCommitWithin method while commit and background merging is underway (blocked by CommitTracker.run). Is this intended? Or should the commit be scheduled and the threads move along working on the next "transaction" ? I see that scheduleCommitWithin is synchronized and it appears that Here are the stacks during the worst of it from YourKit:



Stacks at 2:05:46


1040425761@qtp-2083119959-87 [TIME_WAITING] CPU time: 0:09
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1061187254@qtp-1461351842-1 - Acceptor1 SelectChannelConnector@0.0.0.0:8083 [RUNNABLE, IN_NATIVE] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



110494822@qtp-2083119959-116 [TIME_WAITING] CPU time: 1:07
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



114208441@qtp-739893596-14 [BLOCKED] CPU time: 0:05
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1156712490@qtp-2083119959-84 [TIME_WAITING] CPU time: 0:05
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1207619095@qtp-2083119959-105 [RUNNABLE] CPU time: 0:45
sun.nio.ch.FileChannelImpl.read(ByteBuffer, long)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(byte[], int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.SegmentTermDocs.next()
org.apache.lucene.index.SegmentTermDocs.skipTo(int)
org.apache.lucene.search.TermScorer.skipTo(int)
org.apache.lucene.search.ConjunctionScorer.doNext()
org.apache.lucene.search.ConjunctionScorer.next()
org.apache.lucene.search.BooleanScorer2.next()
org.apache.lucene.search.ConjunctionScorer.next()
org.apache.lucene.search.BooleanScorer2.next()
org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop()
org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent()
org.apache.lucene.search.DisjunctionSumScorer.next()
org.apache.lucene.search.BooleanScorer2.next()
org.apache.lucene.search.ConjunctionScorer.next()
org.apache.lucene.search.ReqExclScorer.next()
org.apache.lucene.search.BooleanScorer2.score(Collector)
org.apache.lucene.search.IndexSearcher.doSearch(IndexReader, Weight, Filter, Collector)
org.apache.lucene.search.IndexSearcher.search(Weight, Filter, Collector)
org.apache.lucene.search.IndexSearcher.search(Weight, Filter, HitCollector)
org.apache.lucene.search.Searcher.search(Query, Filter, HitCollector)
org.apache.lucene.search.Searcher.search(Query, HitCollector)
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.handler.component.QueryComponent.process(ResponseBuilder)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1216648185@qtp-2083119959-99 [TIME_WAITING] CPU time: 0:19
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1224058200@qtp-739893596-30 [BLOCKED] CPU time: 0:05
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1257515479@qtp-739893596-1 - Acceptor1 SelectChannelConnector@0.0.0.0:8082 [RUNNABLE, IN_NATIVE] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1265289734@qtp-1461351842-2 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1270978201@qtp-739893596-49 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1363969808@qtp-2083119959-101 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



136669545@qtp-739893596-17 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1384971503@qtp-739893596-10 [BLOCKED] CPU time: 0:02
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1400534790@qtp-1461351842-0 - Acceptor0 SelectChannelConnector@0.0.0.0:8083 [RUNNABLE, IN_NATIVE] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1415754121@qtp-739893596-29 [BLOCKED] CPU time: 0:03
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1468735454@qtp-739893596-24 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



14765819@qtp-739893596-36 [BLOCKED] CPU time: 0:02
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1547216033@qtp-739893596-37 [BLOCKED] CPU time: 0:01
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1580482860@qtp-739893596-44 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1591907813@qtp-739893596-51 [RUNNABLE] CPU time: 0:00
java.lang.String.intern()
org.apache.lucene.document.Field.(String, boolean, String, Field$Store, Field$Index, Field$TermVector)
org.apache.lucene.document.Field.(String, String, Field$Store, Field$Index, Field$TermVector)
org.apache.solr.schema.FieldType.createField(SchemaField, String, float)
org.apache.solr.schema.SchemaField.createField(String, float)
org.apache.solr.update.DocumentBuilder.toDocument(SolrInputDocument, IndexSchema)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1646708079@qtp-739893596-40 [BLOCKED] CPU time: 0:01
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



164967086@qtp-739893596-3 [BLOCKED] CPU time: 0:26
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1694511298@qtp-739893596-48 [BLOCKED] CPU time: 0:01
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1737144087@qtp-739893596-15 [BLOCKED] CPU time: 0:02
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1753108243@qtp-2083119959-89 [TIME_WAITING] CPU time: 1:39
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



176271867@qtp-198733779-1 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1785326922@qtp-2083119959-7 - Acceptor1 SelectChannelConnector@0.0.0.0:8081 [RUNNABLE, IN_NATIVE] CPU time: 0:18
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1841596208@qtp-2083119959-107 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1915486209@qtp-2083119959-97 [TIME_WAITING] CPU time: 0:14
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



1944875884@qtp-739893596-46 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



2007882585@qtp-198733779-0 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



2013241583@qtp-739893596-39 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



2088100245@qtp-739893596-50 [BLOCKED] CPU time: 0:01
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



211007363@qtp-739893596-45 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



235895896@qtp-2083119959-82 [TIME_WAITING] CPU time: 2:26
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



239632879@qtp-2083119959-98 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



266424179@qtp-739893596-41 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



313491682@qtp-739893596-20 [BLOCKED] CPU time: 0:16
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



401643153@qtp-739893596-38 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



504715431@qtp-2083119959-106 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



51200920@qtp-739893596-47 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



553291430@qtp-739893596-23 [BLOCKED] CPU time: 0:32
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



57393293@qtp-739893596-43 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



604897611@qtp-2083119959-113 [TIME_WAITING] CPU time: 0:58
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



768673839@qtp-739893596-0 - Acceptor0 SelectChannelConnector@0.0.0.0:8082 [RUNNABLE, IN_NATIVE] CPU time: 0:01
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



76934931@qtp-2083119959-95 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



834470436@qtp-2083119959-108 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



870776559@qtp-739893596-42 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



904188954@qtp-2083119959-94 [TIME_WAITING] CPU time: 0:00
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



928415340@qtp-2083119959-8 - Acceptor0 SelectChannelConnector@0.0.0.0:8081 [RUNNABLE, IN_NATIVE] CPU time: 0:32
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



940941978@qtp-2083119959-85 [TIME_WAITING] CPU time: 0:13
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



947539423@qtp-739893596-34 [BLOCKED] CPU time: 0:00
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.scheduleCommitWithin(long)
org.apache.solr.update.DirectUpdateHandler2.delete(DeleteUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processDelete(DeleteUpdateCommand)
org.apache.solr.handler.XMLLoader.processDelete(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.processUpdate(UpdateRequestProcessor, XMLStreamReader)
org.apache.solr.handler.XMLLoader.load(SolrQueryRequest, SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
org.mortbay.thread.QueuedThreadPool$PoolThread.run()



ContinuationBayeux-0 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



pool-1-thread-1 [RUNNABLE] CPU time: 53:22
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(int)
org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next()
org.apache.lucene.search.FieldCacheImpl$7.createValue(IndexReader, Object)
org.apache.lucene.search.FieldCacheImpl$Cache.get(IndexReader, Object)
org.apache.lucene.search.FieldCacheImpl.getInts(IndexReader, String, FieldCache$IntParser)
org.apache.lucene.search.FieldCacheImpl.getInts(IndexReader, String)
org.apache.lucene.search.FieldSortedHitQueue.comparatorInt(IndexReader, String, FieldCache$IntParser)
org.apache.lucene.search.FieldSortedHitQueue$1.createValue(IndexReader, Object)
org.apache.lucene.search.FieldCacheImpl$Cache.get(IndexReader, Object)
org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(IndexReader, String, int, FieldCache$Parser, Locale, SortComparatorSource)
org.apache.lucene.search.FieldSortedHitQueue.(IndexReader, SortField[], int)
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher$QueryResult, SolrIndexSearcher$QueryCommand)
org.apache.solr.handler.component.QueryComponent.process(ResponseBuilder)
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.core.QuerySenderListener.newSearcher(SolrIndexSearcher, SolrIndexSearcher)
org.apache.solr.core.SolrCore$5.call()
java.lang.Thread.run()



pool-2-thread-1 [WAITING] CPU time: 0:34
java.util.concurrent.FutureTask.get()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.DirectUpdateHandler2$CommitTracker.run()
java.lang.Thread.run()



Timer-0 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-1 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-2 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-3 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-4 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-5 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()



Timer-6 [TIME_WAITING] CPU time: 0:00
java.util.TimerThread.run()


Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Jayson Minard
Does every thread really need to notify the update handler of the commit interval/threshold being reached, or really just the first thread that notices should send the signal, or better yet a background commit watching thread so that no "foreground" thread has to pay attention at all.  That is assuming they wouldn't need to block like they are now for a reason I'm likely unaware of...

--j
Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Mike Klaas
On 7-May-09, at 10:36 AM, jayson.minard wrote:

>
> Does every thread really need to notify the update handler of the  
> commit
> interval/threshold being reached, or really just the first thread that
> notices should send the signal, or better yet a background commit  
> watching
> thread so that no "foreground" thread has to pay attention at all.  
> That is
> assuming they wouldn't need to block like they are now for a reason  
> I'm
> likely unaware of...

This is due to the way Lucene was designed (although recent  
improvements in Lucene mean we can do better here).  See the recent  
thread "Autocommit blocking adds?" on solr-user for a related  
discussion.

As the person who first wrote the multi-threaded-ness of DUH2, I'd be  
very happy to promptly review any improvements made to it.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Jayson Minard
Mike,

I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in SOLR-1155, probably ready enough for your review to see if locking makes sense for current Lucene behavior.  

https://issues.apache.org/jira/browse/SOLR-1155

--j

Mike Klaas wrote
On 7-May-09, at 10:36 AM, jayson.minard wrote:

>
> Does every thread really need to notify the update handler of the  
> commit
> interval/threshold being reached, or really just the first thread that
> notices should send the signal, or better yet a background commit  
> watching
> thread so that no "foreground" thread has to pay attention at all.  
> That is
> assuming they wouldn't need to block like they are now for a reason  
> I'm
> likely unaware of...

This is due to the way Lucene was designed (although recent  
improvements in Lucene mean we can do better here).  See the recent  
thread "Autocommit blocking adds?" on solr-user for a related  
discussion.

As the person who first wrote the multi-threaded-ness of DUH2, I'd be  
very happy to promptly review any improvements made to it.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Mike Klaas
Hi Jayson,

Thanks, I'll take a look in the next few days.  The current patch  
doesn't guarantee index consistency during post-commit callback hooks,  
right?  This could be a problem for index replication.  (Incidentally,  
I'm rather unfamiliar with the new java-based replication design.  
Anyone care to comment on the implications?)

cheers,
-MIke

On 10-May-09, at 10:54 AM, jayson.minard wrote:

>
> Mike,
>
> I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in  
> SOLR-1155,
> probably ready enough for your review to see if locking makes sense  
> for
> current Lucene behavior.
>
> https://issues.apache.org/jira/browse/SOLR-1155
>
> --j
>
>
> Mike Klaas wrote:
>>
>> On 7-May-09, at 10:36 AM, jayson.minard wrote:
>>
>>>
>>> Does every thread really need to notify the update handler of the
>>> commit
>>> interval/threshold being reached, or really just the first thread  
>>> that
>>> notices should send the signal, or better yet a background commit
>>> watching
>>> thread so that no "foreground" thread has to pay attention at all.
>>> That is
>>> assuming they wouldn't need to block like they are now for a reason
>>> I'm
>>> likely unaware of...
>>
>> This is due to the way Lucene was designed (although recent
>> improvements in Lucene mean we can do better here).  See the recent
>> thread "Autocommit blocking adds?" on solr-user for a related
>> discussion.
>>
>> As the person who first wrote the multi-threaded-ness of DUH2, I'd be
>> very happy to promptly review any improvements made to it.
>>
>> -Mike
>>
>>
>
> --
> View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23472391.html
> Sent from the Solr - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Jayson Minard
The change to make commit consistent is possible during the call-back, I was trying to avoid that level of lock if possible but if it is the contract of that callback then we have to maintain it.  

Would be nice to be able to do the snapshot first, then release the lock and call the callback on the snapshot instead of holding a lock.  But it really isn't built that way currently with only one callback event.

Mike Klaas wrote
Hi Jayson,

Thanks, I'll take a look in the next few days.  The current patch  
doesn't guarantee index consistency during post-commit callback hooks,  
right?  This could be a problem for index replication.  (Incidentally,  
I'm rather unfamiliar with the new java-based replication design.  
Anyone care to comment on the implications?)

cheers,
-MIke

On 10-May-09, at 10:54 AM, jayson.minard wrote:

>
> Mike,
>
> I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in  
> SOLR-1155,
> probably ready enough for your review to see if locking makes sense  
> for
> current Lucene behavior.
>
> https://issues.apache.org/jira/browse/SOLR-1155
>
> --j
>
>
> Mike Klaas wrote:
>>
>> On 7-May-09, at 10:36 AM, jayson.minard wrote:
>>
>>>
>>> Does every thread really need to notify the update handler of the
>>> commit
>>> interval/threshold being reached, or really just the first thread  
>>> that
>>> notices should send the signal, or better yet a background commit
>>> watching
>>> thread so that no "foreground" thread has to pay attention at all.
>>> That is
>>> assuming they wouldn't need to block like they are now for a reason
>>> I'm
>>> likely unaware of...
>>
>> This is due to the way Lucene was designed (although recent
>> improvements in Lucene mean we can do better here).  See the recent
>> thread "Autocommit blocking adds?" on solr-user for a related
>> discussion.
>>
>> As the person who first wrote the multi-threaded-ness of DUH2, I'd be
>> very happy to promptly review any improvements made to it.
>>
>> -Mike
>>
>>
>
> --
> View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23472391.html
> Sent from the Solr - Dev mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin

Shalin Shekhar Mangar
In reply to this post by Mike Klaas
On Tue, May 12, 2009 at 5:49 AM, Mike Klaas <[hidden email]> wrote:

>
> Thanks, I'll take a look in the next few days.  The current patch doesn't
> guarantee index consistency during post-commit callback hooks, right?  This
> could be a problem for index replication.  (Incidentally, I'm rather
> unfamiliar with the new java-based replication design.  Anyone care to
> comment on the implications?)
>

I have not looked at the patch yet. If adds/deletes are allowed to go
through during post-commit callbacks, then it is not a problem for the java
based replication.

The java based replication uses IndexDeletionPolicy and gets a call back
after commits from the IW. Then the last commit point is replicated to the
slaves. Even if a backround merge is triggered, the deletion policy ensures
that the files of the commit point being replicated are preserved until the
replication finishes.

--
Regards,
Shalin Shekhar Mangar.