Solr 4.0 Alpha taking lot of CPU

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 4.0 Alpha taking lot of CPU

Pavitar Singh
Hi,

We upgraded to Solr 4.0 Alpha and our CPU usage shot off to 400%.In
profiling we are getting following trace.


   - *100.0%*  -
*java.lang*.Thread.run<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
      - *42 Collapsed methods
(show)<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
      *
         - *98.0%*  - *org.apache.lucene.index*
         .DocumentsWriter.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
            - *77.0%*  - *org.apache.lucene.index*
            .DocumentsWriterPerThread.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
               - *76.0%*  - *org.apache.lucene.index*
               .DocFieldProcessor.processDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                  - *76.0%*  - *org.apache.lucene.index*

.DocInverterPerField.processFields<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                     - *36.0%*  - *org.apache.lucene.analysis.miscellaneous*

.TrimFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                        - *35.0%*  - *org.apache.lucene.analysis.core*

.LowerCaseFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                           - *17.0%*  - *org.apache.lucene.analysis.ngram*

.NGramTokenFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                              - *9.2%*  - *org.apache.lucene.util*

.AttributeSource.clearAttributes<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                           - *34.0%*  - *org.apache.lucene.index*

.TermsHashPerField.add<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                        - *12.0%*  - *org.apache.lucene.index*

.FreqProxTermsWriterPerField.addTerm<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                           - *11.0%*  - *org.apache.lucene.index*

.FreqProxTermsWriterPerField.writeProx<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                              - *6.4%*  - *org.apache.lucene.index*

.TermsHashPerField.writeVInt<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                           - *11.0%*  - *
                        org.apache.lucene.analysis.tokenattributes*

.CharTermAttributeImpl.fillBytesRef<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
                     - *15.0%*  - *org.apache.lucene.index*
            .DocumentsWriterFlushControl.obtainAndLock<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4.0 Alpha taking lot of CPU

Pavitar Singh
In text format

100.0%  - java.lang.Thread.run
42 Collapsed methods (show)
98.0%  - org.apache.lucene.index.DocumentsWriter.updateDocument
77.0%  - org.apache.lucene.index.DocumentsWriterPerThread.updateDocument
76.0%  - org.apache.lucene.index.DocFieldProcessor.processDocument
76.0%  - org.apache.lucene.index.DocInverterPerField.processFields
36.0%  - org.apache.lucene.analysis.miscellaneous.TrimFilter.incrementToken
35.0%  - org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken
17.0%  - org.apache.lucene.analysis.ngram.NGramTokenFilter.incrementToken
9.2%  - org.apache.lucene.util.AttributeSource.clearAttributes
34.0%  - org.apache.lucene.index.TermsHashPerField.add
12.0%  - org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm
11.0%  - org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx
6.4%  - org.apache.lucene.index.TermsHashPerField.writeVInt
11.0%  -
org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl.fillBytesRef
15.0%  - org.apache.lucene.index.DocumentsWriterFlushControl.obtainAndLock

On Thu, Jul 12, 2012 at 5:41 AM, Pavitar Singh <[hidden email]> wrote:

> Hi,
>
> We upgraded to Solr 4.0 Alpha and our CPU usage shot off to 400%.In
> profiling we are getting following trace.
>
>
>    - *100.0%*  - *java.lang*.Thread.run<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>       - *42 Collapsed methods (show)<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>       *
>          - *98.0%*  - *org.apache.lucene.index*
>          .DocumentsWriter.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>             - *77.0%*  - *org.apache.lucene.index*
>             .DocumentsWriterPerThread.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                - *76.0%*  - *org.apache.lucene.index*
>                .DocFieldProcessor.processDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                   - *76.0%*  - *org.apache.lucene.index*
>                   .DocInverterPerField.processFields<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                      - *36.0%*  - *
>                      org.apache.lucene.analysis.miscellaneous*
>                      .TrimFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                         - *35.0%*  - *org.apache.lucene.analysis.core*
>                         .LowerCaseFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *17.0%*  - *org.apache.lucene.analysis.ngram*
>                            .NGramTokenFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                               - *9.2%*  - *org.apache.lucene.util*
>                               .AttributeSource.clearAttributes<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *34.0%*  - *org.apache.lucene.index*
>                      .TermsHashPerField.add<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                         - *12.0%*  - *org.apache.lucene.index*
>                         .FreqProxTermsWriterPerField.addTerm<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *11.0%*  - *org.apache.lucene.index*
>                            .FreqProxTermsWriterPerField.writeProx<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                               - *6.4%*  - *org.apache.lucene.index*
>                               .TermsHashPerField.writeVInt<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *11.0%*  - *
>                         org.apache.lucene.analysis.tokenattributes*
>                         .CharTermAttributeImpl.fillBytesRef<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                      - *15.0%*  - *org.apache.lucene.index*
>             .DocumentsWriterFlushControl.obtainAndLock<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4.0 Alpha taking lot of CPU

Yonik Seeley-2-2
In reply to this post by Pavitar Singh
On Wed, Jul 11, 2012 at 8:11 PM, Pavitar Singh <[hidden email]> wrote:
> We upgraded to Solr 4.0 Alpha and our CPU usage shot off to 400%.In
> profiling we are getting following trace.

That could either be good or bad.  Higher CPU can mean higher
concurrency.  Have you benchmarked your indexing performance?

Example: going from 60 minutes for indexing and 200% average CPU usage
to 30 minutes at 400% CPU would generally be considered a good thing.

-Yonik
http://lucidimagination.com

>
>    - *100.0%*  -
> *java.lang*.Thread.run<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>       - *42 Collapsed methods
> (show)<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>       *
>          - *98.0%*  - *org.apache.lucene.index*
>          .DocumentsWriter.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>             - *77.0%*  - *org.apache.lucene.index*
>             .DocumentsWriterPerThread.updateDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                - *76.0%*  - *org.apache.lucene.index*
>                .DocFieldProcessor.processDocument<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                   - *76.0%*  - *org.apache.lucene.index*
>
> .DocInverterPerField.processFields<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                      - *36.0%*  - *org.apache.lucene.analysis.miscellaneous*
>
> .TrimFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                         - *35.0%*  - *org.apache.lucene.analysis.core*
>
> .LowerCaseFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *17.0%*  - *org.apache.lucene.analysis.ngram*
>
> .NGramTokenFilter.incrementToken<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                               - *9.2%*  - *org.apache.lucene.util*
>
> .AttributeSource.clearAttributes<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *34.0%*  - *org.apache.lucene.index*
>
> .TermsHashPerField.add<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                         - *12.0%*  - *org.apache.lucene.index*
>
> .FreqProxTermsWriterPerField.addTerm<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *11.0%*  - *org.apache.lucene.index*
>
> .FreqProxTermsWriterPerField.writeProx<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                               - *6.4%*  - *org.apache.lucene.index*
>
> .TermsHashPerField.writeVInt<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                            - *11.0%*  - *
>                         org.apache.lucene.analysis.tokenattributes*
>
> .CharTermAttributeImpl.fillBytesRef<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>
>                      - *15.0%*  - *org.apache.lucene.index*
>             .DocumentsWriterFlushControl.obtainAndLock<https://rpm.newrelic.com/accounts/132291/applications/834717/profiles/1266#>