[collection create & delete] collection It is not created after several hundred times when it is repeatedly deleted and created. Resolved after restarting the service.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[collection create & delete] collection It is not created after several hundred times when it is repeatedly deleted and created. Resolved after restarting the service.

유정인
Hi

I am using solr 7.7.1.

I would like to ask about index issues.

Use the solr rest api to do the indexing process.

There is a problem here.

My indexing process creates a collection every time I perform a batch
index. When the index is completed, it alias, and the previous collection
is backed up and deleted. The problematic collection is batch indexed every
10 minutes. A message will be displayed that the collection can not be
created from about two days.

"Could not fully create collection"

The message is as above.

 

I am running solr using tomcat.

The above message will be resolved by restarting tomcat.

Deleting a collection does not seem to be complete. Removing collection
seems to reside in memory.

 

Why is this happening? And what is the solution?

I need your help. ㅠ.ㅠ

 

Reply | Threaded
Open this post in threaded view
|

Re: [collection create & delete] collection It is not created after several hundred times when it is repeatedly deleted and created. Resolved after restarting the service.

Shawn Heisey-2
On 4/29/2019 6:36 PM, 유정인 wrote:

> I am using solr 7.7.1.
>
> I would like to ask about index issues.
>
> Use the solr rest api to do the indexing process.
>
> There is a problem here.
>
> My indexing process creates a collection every time I perform a batch
> index. When the index is completed, it alias, and the previous collection
> is backed up and deleted. The problematic collection is batch indexed every
> 10 minutes. A message will be displayed that the collection can not be
> created from about two days.
>
> "Could not fully create collection"
>
> The message is as above.

We will need to see actual logfiles from your servers.  Error messages
are almost always significantly larger than the few words you have
quoted above, and there can be clues in surrounding log entries as well.
  The file will be named solr.log, and with Tomcat, I have no idea where
it's going to end up.

Message attachments won't work.  You'll need to use a file sharing website.

> I am running solr using tomcat.

Solr is not officially supported on Tomcat.  That's probably not the
cause of the problem you're having, but you should know that it's much
more difficult for us to provide support when Solr is running on a
platform that's different than default.

https://wiki.apache.org/solr/WhyNoWar

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

RE: [collection create & delete] collection It is not created after several hundred times when it is repeatedly deleted and created. Resolved after restarting the service.

유정인
The error log.
The important thing here is that the other collection was normal.
The following logs were generated when only collection with backup-delete-
create was created hundreds of times.
The collection where the issue occurred indexed batches every 10 minutes.

[Repeat(Repeat the process below) every 10 minutes]
collection create(issue) curl -v
"http://x.x.x.x:8080/admin/collections?action=CREATE&name=theme1&collection.
configName=theme1_conf&numShards=3&maxShardsPerNode=20&nrtReplicas=1&tlogRep
licas=1&pullReplicas=1"
collection config reload curl -v
http://x.x.x.x:8080/admin/collections?action=RELOAD&name=theme1
collection index curl -v http://x.x.x.x:8080/theme1/update?commit=true --
data-binary @theme.json -H "Content-type:text/json"
collection optimize curl -v
"http://x.x.x.x:8080/theme1/update?optimize=true"
collection alias moodify curl -v
"http://x.x.x.x:8080/admin/collections?action=CREATEALIAS&name=theme&collect
ions=theme1"
collection backup curl -v
"http://x.x.x.x:8080/admin/collections?action=BACKUP&location=/data/_INDEX_B
ACKUP&name=theme_09&collection=theme2"
collection del curl -v
"http://x.x.x.x:8080/admin/collections?action=DELETE&name=theme2"
============================================================================
======================================================
2019-04-27 21:50:32.043 ERROR (OverseerThreadFactory-1184-thread-4-
processing-n:211.60.221.94:9080_) [   ]
o.a.s.c.a.c.OverseerCollectionMessageHandler [processResponse:880] Error
from shard: http://x.x.x.x:8080
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://x.x.x.x:8080
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClien
t.java:657) ~[solr-solrj-7.7.1.jar:7.7.1
5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java
:255) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58
- ishan - 2019-02-23 02:39:09]
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java
:244) ~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58
- ishan - 2019-02-23 02:39:09]
        at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
~[solr-solrj-7.7.1.jar:7.7.1 5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 -
ishan - 2019-02-23 02:39:09]
        at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShard
Handler.java:173) ~[solr-core-7.7.1.jar:7.7.1
5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:07]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.
0_131]
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.
0_131]
        at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(In
strumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$e
xecute$0(ExecutorUtil.java:209) ~[solr-solrj-7.7.1.jar:7.7.1
5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42) [?:1.8.0_131]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
~[?:1.8.0_131]
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_131]
        at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBuffer
Impl.java:139) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBuffer
Impl.java:155) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferIm
pl.java:284) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRes
ponseParser.java:140) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRes
ponseParser.java:57) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.ja
va:261) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Defa
ultBHttpClientConnection.java:165) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:1
67) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestEx
ecutor.java:272) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.jav
a:124) ~[httpcore-4.4.4.jar:4.4.4]
        at
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(Instrumen
tedHttpRequestExecutor.java:120) ~[solr-core-7.7.1.jar:7.7.1
5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:07]
        at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:27
1) ~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.
java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
~[httpclient-4.5.2.jar:4.5.2]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.
java:110) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.
java:184) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.
java:82) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.
java:55) ~[httpclient-4.5.2.jar:4.5.2]
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClien
t.java:542) ~[solr-solrj-7.7.1.jar:7.7.1
5bf96d32f88eb8a2f5e775339885cd6ba84a3b58 - ishan - 2019-02-23 02:39:09]
        ... 12 more
2019-04-27 21:50:38.239 ERROR (OverseerThreadFactory-1184-thread-4-
processing-n:211.60.221.94:9080_) [   ]
o.a.s.c.a.c.OverseerCollectionMessageHandler [log:159] Collection: theme1
operation: delete failed:org.apache.solr.common.SolrException: Invalid
status request for requestId: theme2_del5297443937917431notfoundretried
6times
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.waitF
orCoreAdminAsyncCallToComplete(OverseerCollectionMessageHandler.java:966)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.waitF
orAsyncCallsToComplete(OverseerCollectionMessageHandler.java:906)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.proce
ssResponses(OverseerCollectionMessageHandler.java:776)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.colle
ctionCmd(OverseerCollectionMessageHandler.java:835)
        at
org.apache.solr.cloud.api.collections.DeleteCollectionCmd.call(DeleteCollect
ionCmd.java:117)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.proce
ssMessage(OverseerCollectionMessageHandler.java:292)
        at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor
.java:496)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$e
xecute$0(ExecutorUtil.java:209)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
        at java.lang.Thread.run(Thread.java:748)
2019-04-27 22:00:30.794 ERROR (OverseerThreadFactory-1184-thread-4-
processing-n:211.60.221.94:9080_) [   ]
o.a.s.c.a.c.OverseerCollectionMessageHandler [log:159] Collection: theme1
operation: create failed:org.apache.solr.common.SolrException: Could not
fully create collection: theme1
        at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollect
ionCmd.java:170)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.proce
ssMessage(OverseerCollectionMessageHandler.java:292)
        at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor
.java:496)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$e
xecute$0(ExecutorUtil.java:209)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
        at java.lang.Thread.run(Thread.java:748)
2019-04-27 22:10:30.717 ERROR (OverseerThreadFactory-1184-thread-4-
processing-n:211.60.221.94:9080_) [   ]
o.a.s.c.a.c.OverseerCollectionMessageHandler [log:159] Collection: theme1
operation: create failed:org.apache.solr.common.SolrException: Could not
fully create collection: theme1
        at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollect
ionCmd.java:170)
        at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.proce
ssMessage(OverseerCollectionMessageHandler.java:292)
        at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor
.java:496)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$e
xecute$0(ExecutorUtil.java:209)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
        at java.lang.Thread.run(Thread.java:748)
============================================================================
======================================================-----Original Message-
----
From: Shawn Heisey <[hidden email]>
Sent: Tuesday, April 30, 2019 1:12 PM
To: [hidden email]
Subject: Re: [collection create & delete] collection It is not created
after several hundred times when it is repeatedly deleted and created.
Resolved after restarting the service.

On 4/29/2019 6:36 PM, 유정인 wrote:

> I am using solr 7.7.1.
>
> I would like to ask about index issues.
>
> Use the solr rest api to do the indexing process.
>
> There is a problem here.
>
> My indexing process creates a collection every time I perform a batch
> index. When the index is completed, it alias, and the previous
> collection is backed up and deleted. The problematic collection is
> batch indexed every
> 10 minutes. A message will be displayed that the collection can not be
> created from about two days.
>
> "Could not fully create collection"
>
> The message is as above.

We will need to see actual logfiles from your servers.  Error messages are
almost always significantly larger than the few words you have quoted
above, and there can be clues in surrounding log entries as well.
  The file will be named solr.log, and with Tomcat, I have no idea where
it's going to end up.

Message attachments won't work.  You'll need to use a file sharing website.

> I am running solr using tomcat.

Solr is not officially supported on Tomcat.  That's probably not the cause
of the problem you're having, but you should know that it's much more
difficult for us to provide support when Solr is running on a platform
that's different than default.

https://wiki.apache.org/solr/WhyNoWar

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: [collection create & delete] collection It is not created after several hundred times when it is repeatedly deleted and created. Resolved after restarting the service.

Shawn Heisey-2
On 4/30/2019 1:38 AM, 유정인 wrote:
> 2019-04-27 21:50:32.043 ERROR (OverseerThreadFactory-1184-thread-4-
> processing-n:211.60.221.94:9080_) [   ]
> o.a.s.c.a.c.OverseerCollectionMessageHandler [processResponse:880] Error
> from shard: http://x.x.x.x:8080
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://x.x.x.x:8080

<snip>

> Caused by: java.net.SocketException: Connection reset

Java is saying that the TCP connection was reset.

This would most likely mean that at the TCP layer, a RST packet was
received.  To my knowledge, Solr doesn't reset connections.  Solr
doesn't contain ANY networking code.  On your install, all the
networking is done in Tomcat code, which utilizes Java APIs, and the
heavy lifting of TCP itself is handled by the operating system.

So when I said that I didn't think Tomcat could be responsible for your
problem, I might have been wrong.  This seems to be a problem at the
transport layer, and that is something that Tomcat probably can influence.

This could also be a lower level problem ... with the OS, or with your
networking equipment.  If the Solr connections are traversing the
Internet or a WAN, it could also be a networking problem you cannot
control at all -- somewhere in somebody else's routing infrastructure.

Why did you choose the more difficult path of running under Tomcat?
Have you tried running on the fully tested container that comes with Solr?

Thanks,
Shawn