Solr 8.0.0 - CPU usage 100% when indexed documents

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 8.0.0 - CPU usage 100% when indexed documents

vishal patel-2
Hi

I have configured 2 shards and 3 zoo keeper. When i indexed document in collection, my CPU usage becomes a full.
I have attached thread dump.
Is there Any changes needed in solrconfig.xml?

Sent from Outlook
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

Jörn Franke
Can you please describe your scenario in detail ?

How does your load process look like (custom module? How many threads?)?

How many files do you try to index ? What is their format?
How does your solr config look like?

How many cores do you have? What else is installed on the Solr server?

Which Operation System?

What do the log files tell your from Solr and Zookeeper?

What is the Schema looking like?

> Am 08.04.2019 um 12:01 schrieb vishal patel <[hidden email]>:
>
> Hi
>
> I have configured 2 shards and 3 zoo keeper. When i indexed document in collection, my CPU usage becomes a full.
> I have attached thread dump.
> Is there Any changes needed in solrconfig.xml?
>
> Sent from Outlook
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

vishal patel-2
I have created two solr shards with 3 zoo keeper. First do upconfig in zoo keeper then start the both solr with different port then create a "actionscomments" collection using API call.

When I indexed one document in actionscomments, my CPU utilization go high.

Note :
upconfig command ::  zkcli.bat -zkhost 192.168.100.145:3181,192.168.100.145:3182,192.168.100.145:3183 -cmd upconfig -confdir E:/SolrCloud-8-0-0/solr1/server/solr/configsets/actionscomments/conf -confname actionscomments. [E:\SolrCloud-8-0-0\solr1\server\scripts\cloud-scripts]
Solr start command ::  solr start -p 7991 and solr start -p 7992 [E:\SolrCloud-8-0-0\solr1\bin and E:\SolrCloud-8-0-0\solr2\bin]
Create a collection :: http://192.168.102.150:7991/solr/admin/collections?_=1554285992377&action=CREATE&autoAddReplicas=false&collection.configName=actionscomments&maxShardsPerNode=1&name=actionscomments&numShards=2&replicationFactor=1&router.name=compositeId&wt=json
Operating system :: windows server 2008 R2 standard

When I indexed document, CPU goes high and in thread dump noticed commitScheduler-25-thread-2,commitScheduler-48-thread-2 ,commitScheduler-21-thread-2. After sometimes  it automatically removed and CPU goes down.

In log file I can not find any error and I indexed document using AsiteSolrCloudManager.

I have attached my solrconfig.xml and schema.xml and also thread dump which got from solr admin GUI.

Sent from Outlook

From: Jörn Franke <[hidden email]>
Sent: Monday, April 8, 2019 4:16 PM
To: [hidden email]
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents
 
Can you please describe your scenario in detail ?

How does your load process look like (custom module? How many threads?)?

How many files do you try to index ? What is their format?
How does your solr config look like?

How many cores do you have? What else is installed on the Solr server?

Which Operation System?

What do the log files tell your from Solr and Zookeeper?

What is the Schema looking like?

> Am 08.04.2019 um 12:01 schrieb vishal patel <[hidden email]>:
>
> Hi
>
> I have configured 2 shards and 3 zoo keeper. When i indexed document in collection, my CPU usage becomes a full.
> I have attached thread dump.
> Is there Any changes needed in solrconfig.xml?
>
> Sent from Outlook

schema.xml (26K) Download Attachment
solrconfig.xml (56K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

Shawn Heisey-2
On 4/8/2019 7:22 AM, vishal patel wrote:
> I have created two solr shards with 3 zoo keeper. First do upconfig in
> zoo keeper then start the both solr with different port then create a
> "actionscomments" collection using API call.
>
> When I indexed one document in actionscomments, my CPU utilization go high.

You were asked how you are doing the indexing.  You still haven't
provided that information. You said "AsiteSolrCloudManager" ... but when
I google for that, the only thing that comes up is this email thread.  I
have no idea what AsiteSolrCloudManager is.  One thing I *can* say is
that it is not part of Solr.

> I have attached my solrconfig.xml and schema.xml and also thread dump
> which got from solr admin GUI.

The schema and solrconfig came through.  The thread dump did not.  I'm
surprised that ANY attachments made it to the list ... normally they
don't.  The thread dump also did not come through on your first message.
  If you need to share files, you'll need to find a mechanism other than
attachments to do it.  File sharing websites work well.

So we don't have the thread dump.  But for a problem like this, a thread
dump is not going to be helpful.  I've never seen anything in a Java
thread dump to indicate which threads are using the most CPU.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

vishal patel-2
Sorry my mistake there is no class of that.

I have add the data using below code.
CloudSolrServer cloudServer = new CloudSolrServer(zkHost);
cloudServer.setDefaultCollection("actionscomments");
cloudServer.setParallelUpdates(true);
List<SolrInputDocument> docs = new ArrayList<>();
SolrInputDocument solrDocument = new SolrInputDocument();
solrDocument.addField("id", "123");
docs.add(solrDocument);
cloudServer.add(docs, 1000);

I have indexed only one document.

I attached thread dump snap shot but may be you didn't get. I have attached again one document which has both solr thread dump snap shot.

After the commitScheduler-25-thread-2,commitScheduler-48-thread-2 ,commitScheduler-21-thread-2 thread gone , CPU came normal.

Sent from Outlook

From: Shawn Heisey <[hidden email]>
Sent: Monday, April 8, 2019 9:21 PM
To: [hidden email]
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents
 
On 4/8/2019 7:22 AM, vishal patel wrote:
> I have created two solr shards with 3 zoo keeper. First do upconfig in
> zoo keeper then start the both solr with different port then create a
> "actionscomments" collection using API call.
>
> When I indexed one document in actionscomments, my CPU utilization go high.

You were asked how you are doing the indexing.  You still haven't
provided that information. You said "AsiteSolrCloudManager" ... but when
I google for that, the only thing that comes up is this email thread.  I
have no idea what AsiteSolrCloudManager is.  One thing I *can* say is
that it is not part of Solr.

> I have attached my solrconfig.xml and schema.xml and also thread dump
> which got from solr admin GUI.

The schema and solrconfig came through.  The thread dump did not.  I'm
surprised that ANY attachments made it to the list ... normally they
don't.  The thread dump also did not come through on your first message.
  If you need to share files, you'll need to find a mechanism other than
attachments to do it.  File sharing websites work well.

So we don't have the thread dump.  But for a problem like this, a thread
dump is not going to be helpful.  I've never seen anything in a Java
thread dump to indicate which threads are using the most CPU.

Thanks,
Shawn

threaddumpSolr.docx (549K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

Shawn Heisey-2
On 4/8/2019 11:00 PM, vishal patel wrote:

> Sorry my mistake there is no class of that.
>
> I have add the data using below code.
> CloudSolrServer cloudServer = new CloudSolrServer(zkHost);
> cloudServer.setDefaultCollection("actionscomments");
> cloudServer.setParallelUpdates(true);
> List<SolrInputDocument> docs = new ArrayList<>();
> SolrInputDocument solrDocument = new SolrInputDocument();
> solrDocument.addField("id", "123");
> docs.add(solrDocument);
> cloudServer.add(docs, 1000);

Side note:  This code is not using SolrJ 8.0.0.  CloudSolrServer was
deprecated in version 5.0.0 and completely removed in version 6.0.0.
I'm surprised this code even works at all with Solr 8.0.0 -- you need to
upgrade to SolrJ 8 and use CloudSolrClient.

How long does the system remain at 100 percent CPU when you index that
single document that only has one field?  If it's longer than a very
small fraction of a second, then my guess is that it's cache warming
queries using the CPU, not the indexing itself.

How many CPU cores are at 100 percent?  Is it just one, or multiple?  It
would be odd for it to be multiple, unless there is other activity going
on at the same time.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

vishal patel-2
Hi,

After your suggestion i changed code
String SOLR_URL="http://localhost:7991/solr/actionscomments";
SolrClient solrClient = new HttpSolrClient.Builder(SOLR_URL).build();
SolrInputDocument document = new SolrInputDocument();
document.addField("id","ACTC6401895");
solrClient.add(document);
solrClient.commit();

Still my CPU usage went high and my CPU has 4 core and no other application running in my machine.

After the lots of try, I found out the below issue.
Before solrconfig.xml (6.1.0)
<autoCommit>
       <maxTime>600000</maxTime>
       <maxDocs>20000</maxDocs>
       <openSearcher>false</openSearcher>
 </autoCommit>

After the below change in solrconfig.xml
<autoCommit> (8.0.0)
       <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
       <maxDocs>20000</maxDocs>
       <openSearcher>false</openSearcher>
 </autoCommit>

Actually I am upgrading solr 6.1.0 to 8.0.0. In 6.1.0 it is working fine with autocommit maxtime 600000.
But in 8.0.0, CPU usage goes high.[commitScheduler thread running long time]

Please give me more details why is it happening in solr 8.0.0.
Is any my mistake? In previous mail, I attached solrconfig.xml so please verify it.


Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: Shawn Heisey <[hidden email]>
Sent: Tuesday, April 9, 2019 1:38 PM
To: [hidden email]
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents

On 4/8/2019 11:00 PM, vishal patel wrote:

> Sorry my mistake there is no class of that.
>
> I have add the data using below code.
> CloudSolrServer cloudServer = new CloudSolrServer(zkHost);
> cloudServer.setDefaultCollection("actionscomments");
> cloudServer.setParallelUpdates(true);
> List<SolrInputDocument> docs = new ArrayList<>();
> SolrInputDocument solrDocument = new SolrInputDocument();
> solrDocument.addField("id", "123");
> docs.add(solrDocument);
> cloudServer.add(docs, 1000);

Side note:  This code is not using SolrJ 8.0.0.  CloudSolrServer was
deprecated in version 5.0.0 and completely removed in version 6.0.0.
I'm surprised this code even works at all with Solr 8.0.0 -- you need to
upgrade to SolrJ 8 and use CloudSolrClient.

How long does the system remain at 100 percent CPU when you index that
single document that only has one field?  If it's longer than a very
small fraction of a second, then my guess is that it's cache warming
queries using the CPU, not the indexing itself.

How many CPU cores are at 100 percent?  Is it just one, or multiple?  It
would be odd for it to be multiple, unless there is other activity going
on at the same time.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

Shawn Heisey-2
On 4/9/2019 10:53 PM, vishal patel wrote:
> Still my CPU usage went high and my CPU has 4 core and no other application running in my machine.

I was asking how many CPUs went to 100 percent, not how many CPUs you
have.  And I also asked how long CPU usage remains at 100 percent after
indexing a single document.

What Java version are you running?  We do have a possible bug that could
be affecting you.

https://issues.apache.org/jira/browse/SOLR-13349

If this is the problem you're experiencing, the solution would be to
either upgrade to Java 11 or wait for Solr 8.1 to be released.

Note that Oracle requires payment if you use their Java 11 in
production.  You're likely going to want to use OpenJDK.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.0.0 - CPU usage 100% when indexed documents

vishal patel-2
Thanks for your reply..

All 4 CPU core got high by 12 to 15 seconds. we used java 8.

I got your point. We will wait solr 8.1 rather upgrade OpenJDK 11.

Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: Shawn Heisey <[hidden email]>
Sent: Wednesday, April 10, 2019 9:07 PM
To: [hidden email]
Subject: Re: Solr 8.0.0 - CPU usage 100% when indexed documents

On 4/9/2019 10:53 PM, vishal patel wrote:
> Still my CPU usage went high and my CPU has 4 core and no other application running in my machine.

I was asking how many CPUs went to 100 percent, not how many CPUs you
have.  And I also asked how long CPU usage remains at 100 percent after
indexing a single document.

What Java version are you running?  We do have a possible bug that could
be affecting you.

https://issues.apache.org/jira/browse/SOLR-13349

If this is the problem you're experiencing, the solution would be to
either upgrade to Java 11 or wait for Solr 8.1 to be released.

Note that Oracle requires payment if you use their Java 11 in
production.  You're likely going to want to use OpenJDK.

Thanks,
Shawn