Solr process takes several minutes before accepting commands after restart

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr process takes several minutes before accepting commands after restart

Koen De Groote
Hello

I was testing some backup/restore scenarios.

1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
SolrCloud, with zookeeper.

The steps are as follows:

1. Manually delete the data folder.
2. Restart the container. The process is now in error mode, complaining
that it cannot find the cores.
3. Fix the install, meaning create new data folders, which are empty at
this point.
4. Restart the container again, to pick up the empty folders and not be in
error anymore.
5. Perform the restore
6. Check if everything is available again

The problem is between step 4 and 5. After step 4, it takes several minutes
before solr actually responds to curl commands.

Once responsive, the restore happened just fine. But it's very stressful in
a situation where you have to restore a production environment and the
process just doesn't respond for 5-10 minutes.

We're talking about 20GB of data here, so not very much, but not little
either.

Is it normal that it takes so long before solr responds? If not, what
should I look at in order to find the cause?

I have asked this before recently, though the wording was confusing. This
should be clearer.

Kind regards,
Koen De Groote
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Jörn Franke
Have you checked the log files of Solr?


Do you have a service mesh in-between? Could it be something at the network layer/container orchestration  that is blocking requests for some minutes?

> Am 20.11.2019 um 10:32 schrieb Koen De Groote <[hidden email]>:
>
> Hello
>
> I was testing some backup/restore scenarios.
>
> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> SolrCloud, with zookeeper.
>
> The steps are as follows:
>
> 1. Manually delete the data folder.
> 2. Restart the container. The process is now in error mode, complaining
> that it cannot find the cores.
> 3. Fix the install, meaning create new data folders, which are empty at
> this point.
> 4. Restart the container again, to pick up the empty folders and not be in
> error anymore.
> 5. Perform the restore
> 6. Check if everything is available again
>
> The problem is between step 4 and 5. After step 4, it takes several minutes
> before solr actually responds to curl commands.
>
> Once responsive, the restore happened just fine. But it's very stressful in
> a situation where you have to restore a production environment and the
> process just doesn't respond for 5-10 minutes.
>
> We're talking about 20GB of data here, so not very much, but not little
> either.
>
> Is it normal that it takes so long before solr responds? If not, what
> should I look at in order to find the cause?
>
> I have asked this before recently, though the wording was confusing. This
> should be clearer.
>
> Kind regards,
> Koen De Groote
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Koen De Groote
The logs files showed a startup, printing of all the config options that
had been set, 1 or 2 commands that got executed and then nothing.

Sending the curl did not get shown in the logs files until after that
period where Solr became unresponsive.

Service mesh, I don't think so? It's in a docker container, but that
shouldn't be a problem, it usually never is.


On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]> wrote:

> Have you checked the log files of Solr?
>
>
> Do you have a service mesh in-between? Could it be something at the
> network layer/container orchestration  that is blocking requests for some
> minutes?
>
> > Am 20.11.2019 um 10:32 schrieb Koen De Groote <
> [hidden email]>:
> >
> > Hello
> >
> > I was testing some backup/restore scenarios.
> >
> > 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> > SolrCloud, with zookeeper.
> >
> > The steps are as follows:
> >
> > 1. Manually delete the data folder.
> > 2. Restart the container. The process is now in error mode, complaining
> > that it cannot find the cores.
> > 3. Fix the install, meaning create new data folders, which are empty at
> > this point.
> > 4. Restart the container again, to pick up the empty folders and not be
> in
> > error anymore.
> > 5. Perform the restore
> > 6. Check if everything is available again
> >
> > The problem is between step 4 and 5. After step 4, it takes several
> minutes
> > before solr actually responds to curl commands.
> >
> > Once responsive, the restore happened just fine. But it's very stressful
> in
> > a situation where you have to restore a production environment and the
> > process just doesn't respond for 5-10 minutes.
> >
> > We're talking about 20GB of data here, so not very much, but not little
> > either.
> >
> > Is it normal that it takes so long before solr responds? If not, what
> > should I look at in order to find the cause?
> >
> > I have asked this before recently, though the wording was confusing. This
> > should be clearer.
> >
> > Kind regards,
> > Koen De Groote
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Erick Erickson
Koen:

Do you have any spellcheckers or suggesters defined with buildOnCommit or buildOnStartup set to “true”? Depending on the implementation, this may have to read the stored data for the field used in the suggester/spellchecker from _every_ document in your collection, which can take many minutes. Even if your implementation in your config is file-based it can still take a while.

Shot in the dark….

Erick

> On Nov 21, 2019, at 4:03 AM, Koen De Groote <[hidden email]> wrote:
>
> The logs files showed a startup, printing of all the config options that
> had been set, 1 or 2 commands that got executed and then nothing.
>
> Sending the curl did not get shown in the logs files until after that
> period where Solr became unresponsive.
>
> Service mesh, I don't think so? It's in a docker container, but that
> shouldn't be a problem, it usually never is.
>
>
> On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]> wrote:
>
>> Have you checked the log files of Solr?
>>
>>
>> Do you have a service mesh in-between? Could it be something at the
>> network layer/container orchestration  that is blocking requests for some
>> minutes?
>>
>>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
>> [hidden email]>:
>>>
>>> Hello
>>>
>>> I was testing some backup/restore scenarios.
>>>
>>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
>>> SolrCloud, with zookeeper.
>>>
>>> The steps are as follows:
>>>
>>> 1. Manually delete the data folder.
>>> 2. Restart the container. The process is now in error mode, complaining
>>> that it cannot find the cores.
>>> 3. Fix the install, meaning create new data folders, which are empty at
>>> this point.
>>> 4. Restart the container again, to pick up the empty folders and not be
>> in
>>> error anymore.
>>> 5. Perform the restore
>>> 6. Check if everything is available again
>>>
>>> The problem is between step 4 and 5. After step 4, it takes several
>> minutes
>>> before solr actually responds to curl commands.
>>>
>>> Once responsive, the restore happened just fine. But it's very stressful
>> in
>>> a situation where you have to restore a production environment and the
>>> process just doesn't respond for 5-10 minutes.
>>>
>>> We're talking about 20GB of data here, so not very much, but not little
>>> either.
>>>
>>> Is it normal that it takes so long before solr responds? If not, what
>>> should I look at in order to find the cause?
>>>
>>> I have asked this before recently, though the wording was confusing. This
>>> should be clearer.
>>>
>>> Kind regards,
>>> Koen De Groote
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Koen De Groote
Erick:

No suggesters. There is 1 spellchecker for

<str name="queryAnalyzerFieldType">text_general</str>

But no buildOnCommit or buildOnStartup setting mentioned anywhere.

That being said, the point in time at which this occurs, the database is
guaranteed to be empty, as the data folders had previously been deleted and
recreated empty. Then the docker container is restarted and this behavior
is observed.

Long shot, but even if Solr is getting data from zookeeper telling of file
locations and checking for the existence of these files... that should be
pretty fast, I'd think.

This is really disturbing. I know what to expect when recovering now, but
someone doing this on a live environment that has to be up again ASAP is
probably going to be sweating bullets.


On Thu, Nov 21, 2019 at 2:45 PM Erick Erickson <[hidden email]>
wrote:

> Koen:
>
> Do you have any spellcheckers or suggesters defined with buildOnCommit or
> buildOnStartup set to “true”? Depending on the implementation, this may
> have to read the stored data for the field used in the
> suggester/spellchecker from _every_ document in your collection, which can
> take many minutes. Even if your implementation in your config is file-based
> it can still take a while.
>
> Shot in the dark….
>
> Erick
>
> > On Nov 21, 2019, at 4:03 AM, Koen De Groote <[hidden email]>
> wrote:
> >
> > The logs files showed a startup, printing of all the config options that
> > had been set, 1 or 2 commands that got executed and then nothing.
> >
> > Sending the curl did not get shown in the logs files until after that
> > period where Solr became unresponsive.
> >
> > Service mesh, I don't think so? It's in a docker container, but that
> > shouldn't be a problem, it usually never is.
> >
> >
> > On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]>
> wrote:
> >
> >> Have you checked the log files of Solr?
> >>
> >>
> >> Do you have a service mesh in-between? Could it be something at the
> >> network layer/container orchestration  that is blocking requests for
> some
> >> minutes?
> >>
> >>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
> >> [hidden email]>:
> >>>
> >>> Hello
> >>>
> >>> I was testing some backup/restore scenarios.
> >>>
> >>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> >>> SolrCloud, with zookeeper.
> >>>
> >>> The steps are as follows:
> >>>
> >>> 1. Manually delete the data folder.
> >>> 2. Restart the container. The process is now in error mode, complaining
> >>> that it cannot find the cores.
> >>> 3. Fix the install, meaning create new data folders, which are empty at
> >>> this point.
> >>> 4. Restart the container again, to pick up the empty folders and not be
> >> in
> >>> error anymore.
> >>> 5. Perform the restore
> >>> 6. Check if everything is available again
> >>>
> >>> The problem is between step 4 and 5. After step 4, it takes several
> >> minutes
> >>> before solr actually responds to curl commands.
> >>>
> >>> Once responsive, the restore happened just fine. But it's very
> stressful
> >> in
> >>> a situation where you have to restore a production environment and the
> >>> process just doesn't respond for 5-10 minutes.
> >>>
> >>> We're talking about 20GB of data here, so not very much, but not little
> >>> either.
> >>>
> >>> Is it normal that it takes so long before solr responds? If not, what
> >>> should I look at in order to find the cause?
> >>>
> >>> I have asked this before recently, though the wording was confusing.
> This
> >>> should be clearer.
> >>>
> >>> Kind regards,
> >>> Koen De Groote
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

David Hastings
https://lucidworks.com/post/solr-suggester/

You must set buildonstartup to false, the default is true. Try it

> On Nov 21, 2019, at 3:21 PM, Koen De Groote <[hidden email]> wrote:
>
> Erick:
>
> No suggesters. There is 1 spellchecker for
>
> <str name="queryAnalyzerFieldType">text_general</str>
>
> But no buildOnCommit or buildOnStartup setting mentioned anywhere.
>
> That being said, the point in time at which this occurs, the database is
> guaranteed to be empty, as the data folders had previously been deleted and
> recreated empty. Then the docker container is restarted and this behavior
> is observed.
>
> Long shot, but even if Solr is getting data from zookeeper telling of file
> locations and checking for the existence of these files... that should be
> pretty fast, I'd think.
>
> This is really disturbing. I know what to expect when recovering now, but
> someone doing this on a live environment that has to be up again ASAP is
> probably going to be sweating bullets.
>
>
> On Thu, Nov 21, 2019 at 2:45 PM Erick Erickson <[hidden email]>
> wrote:
>
>> Koen:
>>
>> Do you have any spellcheckers or suggesters defined with buildOnCommit or
>> buildOnStartup set to “true”? Depending on the implementation, this may
>> have to read the stored data for the field used in the
>> suggester/spellchecker from _every_ document in your collection, which can
>> take many minutes. Even if your implementation in your config is file-based
>> it can still take a while.
>>
>> Shot in the dark….
>>
>> Erick
>>
>>> On Nov 21, 2019, at 4:03 AM, Koen De Groote <[hidden email]>
>> wrote:
>>>
>>> The logs files showed a startup, printing of all the config options that
>>> had been set, 1 or 2 commands that got executed and then nothing.
>>>
>>> Sending the curl did not get shown in the logs files until after that
>>> period where Solr became unresponsive.
>>>
>>> Service mesh, I don't think so? It's in a docker container, but that
>>> shouldn't be a problem, it usually never is.
>>>
>>>
>>> On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]>
>> wrote:
>>>
>>>> Have you checked the log files of Solr?
>>>>
>>>>
>>>> Do you have a service mesh in-between? Could it be something at the
>>>> network layer/container orchestration  that is blocking requests for
>> some
>>>> minutes?
>>>>
>>>>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
>>>> [hidden email]>:
>>>>>
>>>>> Hello
>>>>>
>>>>> I was testing some backup/restore scenarios.
>>>>>
>>>>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
>>>>> SolrCloud, with zookeeper.
>>>>>
>>>>> The steps are as follows:
>>>>>
>>>>> 1. Manually delete the data folder.
>>>>> 2. Restart the container. The process is now in error mode, complaining
>>>>> that it cannot find the cores.
>>>>> 3. Fix the install, meaning create new data folders, which are empty at
>>>>> this point.
>>>>> 4. Restart the container again, to pick up the empty folders and not be
>>>> in
>>>>> error anymore.
>>>>> 5. Perform the restore
>>>>> 6. Check if everything is available again
>>>>>
>>>>> The problem is between step 4 and 5. After step 4, it takes several
>>>> minutes
>>>>> before solr actually responds to curl commands.
>>>>>
>>>>> Once responsive, the restore happened just fine. But it's very
>> stressful
>>>> in
>>>>> a situation where you have to restore a production environment and the
>>>>> process just doesn't respond for 5-10 minutes.
>>>>>
>>>>> We're talking about 20GB of data here, so not very much, but not little
>>>>> either.
>>>>>
>>>>> Is it normal that it takes so long before solr responds? If not, what
>>>>> should I look at in order to find the cause?
>>>>>
>>>>> I have asked this before recently, though the wording was confusing.
>> This
>>>>> should be clearer.
>>>>>
>>>>> Kind regards,
>>>>> Koen De Groote
>>>>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

David Hastings
https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/using-solr-auto-suggest.html


If you need more references. Set all parameters yourself, don’t rely on defaults.

> On Nov 21, 2019, at 3:41 PM, Dave <[hidden email]> wrote:
>
> https://lucidworks.com/post/solr-suggester/
>
> You must set buildonstartup to false, the default is true. Try it
>
>> On Nov 21, 2019, at 3:21 PM, Koen De Groote <[hidden email]> wrote:
>>
>> Erick:
>>
>> No suggesters. There is 1 spellchecker for
>>
>> <str name="queryAnalyzerFieldType">text_general</str>
>>
>> But no buildOnCommit or buildOnStartup setting mentioned anywhere.
>>
>> That being said, the point in time at which this occurs, the database is
>> guaranteed to be empty, as the data folders had previously been deleted and
>> recreated empty. Then the docker container is restarted and this behavior
>> is observed.
>>
>> Long shot, but even if Solr is getting data from zookeeper telling of file
>> locations and checking for the existence of these files... that should be
>> pretty fast, I'd think.
>>
>> This is really disturbing. I know what to expect when recovering now, but
>> someone doing this on a live environment that has to be up again ASAP is
>> probably going to be sweating bullets.
>>
>>
>> On Thu, Nov 21, 2019 at 2:45 PM Erick Erickson <[hidden email]>
>> wrote:
>>
>>> Koen:
>>>
>>> Do you have any spellcheckers or suggesters defined with buildOnCommit or
>>> buildOnStartup set to “true”? Depending on the implementation, this may
>>> have to read the stored data for the field used in the
>>> suggester/spellchecker from _every_ document in your collection, which can
>>> take many minutes. Even if your implementation in your config is file-based
>>> it can still take a while.
>>>
>>> Shot in the dark….
>>>
>>> Erick
>>>
>>>> On Nov 21, 2019, at 4:03 AM, Koen De Groote <[hidden email]>
>>> wrote:
>>>>
>>>> The logs files showed a startup, printing of all the config options that
>>>> had been set, 1 or 2 commands that got executed and then nothing.
>>>>
>>>> Sending the curl did not get shown in the logs files until after that
>>>> period where Solr became unresponsive.
>>>>
>>>> Service mesh, I don't think so? It's in a docker container, but that
>>>> shouldn't be a problem, it usually never is.
>>>>
>>>>
>>>> On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]>
>>> wrote:
>>>>
>>>>> Have you checked the log files of Solr?
>>>>>
>>>>>
>>>>> Do you have a service mesh in-between? Could it be something at the
>>>>> network layer/container orchestration  that is blocking requests for
>>> some
>>>>> minutes?
>>>>>
>>>>>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
>>>>> [hidden email]>:
>>>>>>
>>>>>> Hello
>>>>>>
>>>>>> I was testing some backup/restore scenarios.
>>>>>>
>>>>>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
>>>>>> SolrCloud, with zookeeper.
>>>>>>
>>>>>> The steps are as follows:
>>>>>>
>>>>>> 1. Manually delete the data folder.
>>>>>> 2. Restart the container. The process is now in error mode, complaining
>>>>>> that it cannot find the cores.
>>>>>> 3. Fix the install, meaning create new data folders, which are empty at
>>>>>> this point.
>>>>>> 4. Restart the container again, to pick up the empty folders and not be
>>>>> in
>>>>>> error anymore.
>>>>>> 5. Perform the restore
>>>>>> 6. Check if everything is available again
>>>>>>
>>>>>> The problem is between step 4 and 5. After step 4, it takes several
>>>>> minutes
>>>>>> before solr actually responds to curl commands.
>>>>>>
>>>>>> Once responsive, the restore happened just fine. But it's very
>>> stressful
>>>>> in
>>>>>> a situation where you have to restore a production environment and the
>>>>>> process just doesn't respond for 5-10 minutes.
>>>>>>
>>>>>> We're talking about 20GB of data here, so not very much, but not little
>>>>>> either.
>>>>>>
>>>>>> Is it normal that it takes so long before solr responds? If not, what
>>>>>> should I look at in order to find the cause?
>>>>>>
>>>>>> I have asked this before recently, though the wording was confusing.
>>> This
>>>>>> should be clearer.
>>>>>>
>>>>>> Kind regards,
>>>>>> Koen De Groote
>>>>>
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Koen De Groote
In reply to this post by David Hastings
Thanks for the link, that's some nice documentation and guidelines.

I'll probably have time to test again next week, but in the meantime, it
does make me scratch my head.

I deleted the root folder and re-instrumented the environment.

So, there's nothing there. Nothing.

The text says:

"Do you really want to re-read, decompress and add the field from *every*
document to the suggester *every time you start Solr!* Likely not, but you
can if you want to."

However, there shouldn't be any documents, since it all got deleted and an
empty database got set up to facilitate the restore. The slowness happens
before the restore. With the fresh install.

But yeah, I'll try it out, thanks for the links.




On Thu, Nov 21, 2019 at 9:41 PM Dave <[hidden email]> wrote:

> https://lucidworks.com/post/solr-suggester/
>
> You must set buildonstartup to false, the default is true. Try it
>
> > On Nov 21, 2019, at 3:21 PM, Koen De Groote <[hidden email]>
> wrote:
> >
> > Erick:
> >
> > No suggesters. There is 1 spellchecker for
> >
> > <str name="queryAnalyzerFieldType">text_general</str>
> >
> > But no buildOnCommit or buildOnStartup setting mentioned anywhere.
> >
> > That being said, the point in time at which this occurs, the database is
> > guaranteed to be empty, as the data folders had previously been deleted
> and
> > recreated empty. Then the docker container is restarted and this behavior
> > is observed.
> >
> > Long shot, but even if Solr is getting data from zookeeper telling of
> file
> > locations and checking for the existence of these files... that should be
> > pretty fast, I'd think.
> >
> > This is really disturbing. I know what to expect when recovering now, but
> > someone doing this on a live environment that has to be up again ASAP is
> > probably going to be sweating bullets.
> >
> >
> > On Thu, Nov 21, 2019 at 2:45 PM Erick Erickson <[hidden email]>
> > wrote:
> >
> >> Koen:
> >>
> >> Do you have any spellcheckers or suggesters defined with buildOnCommit
> or
> >> buildOnStartup set to “true”? Depending on the implementation, this may
> >> have to read the stored data for the field used in the
> >> suggester/spellchecker from _every_ document in your collection, which
> can
> >> take many minutes. Even if your implementation in your config is
> file-based
> >> it can still take a while.
> >>
> >> Shot in the dark….
> >>
> >> Erick
> >>
> >>> On Nov 21, 2019, at 4:03 AM, Koen De Groote <
> [hidden email]>
> >> wrote:
> >>>
> >>> The logs files showed a startup, printing of all the config options
> that
> >>> had been set, 1 or 2 commands that got executed and then nothing.
> >>>
> >>> Sending the curl did not get shown in the logs files until after that
> >>> period where Solr became unresponsive.
> >>>
> >>> Service mesh, I don't think so? It's in a docker container, but that
> >>> shouldn't be a problem, it usually never is.
> >>>
> >>>
> >>> On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]>
> >> wrote:
> >>>
> >>>> Have you checked the log files of Solr?
> >>>>
> >>>>
> >>>> Do you have a service mesh in-between? Could it be something at the
> >>>> network layer/container orchestration  that is blocking requests for
> >> some
> >>>> minutes?
> >>>>
> >>>>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
> >>>> [hidden email]>:
> >>>>>
> >>>>> Hello
> >>>>>
> >>>>> I was testing some backup/restore scenarios.
> >>>>>
> >>>>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> >>>>> SolrCloud, with zookeeper.
> >>>>>
> >>>>> The steps are as follows:
> >>>>>
> >>>>> 1. Manually delete the data folder.
> >>>>> 2. Restart the container. The process is now in error mode,
> complaining
> >>>>> that it cannot find the cores.
> >>>>> 3. Fix the install, meaning create new data folders, which are empty
> at
> >>>>> this point.
> >>>>> 4. Restart the container again, to pick up the empty folders and not
> be
> >>>> in
> >>>>> error anymore.
> >>>>> 5. Perform the restore
> >>>>> 6. Check if everything is available again
> >>>>>
> >>>>> The problem is between step 4 and 5. After step 4, it takes several
> >>>> minutes
> >>>>> before solr actually responds to curl commands.
> >>>>>
> >>>>> Once responsive, the restore happened just fine. But it's very
> >> stressful
> >>>> in
> >>>>> a situation where you have to restore a production environment and
> the
> >>>>> process just doesn't respond for 5-10 minutes.
> >>>>>
> >>>>> We're talking about 20GB of data here, so not very much, but not
> little
> >>>>> either.
> >>>>>
> >>>>> Is it normal that it takes so long before solr responds? If not, what
> >>>>> should I look at in order to find the cause?
> >>>>>
> >>>>> I have asked this before recently, though the wording was confusing.
> >> This
> >>>>> should be clearer.
> >>>>>
> >>>>> Kind regards,
> >>>>> Koen De Groote
> >>>>
> >>
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr process takes several minutes before accepting commands after restart

Erick Erickson
right, doesn’t sound likely that it’s rebuilding suggesters/spellcheckers. Throwing a profiler at it to see where it’s spending the time might be easiest.

> On Nov 22, 2019, at 3:45 AM, Koen De Groote <[hidden email]> wrote:
>
> Thanks for the link, that's some nice documentation and guidelines.
>
> I'll probably have time to test again next week, but in the meantime, it
> does make me scratch my head.
>
> I deleted the root folder and re-instrumented the environment.
>
> So, there's nothing there. Nothing.
>
> The text says:
>
> "Do you really want to re-read, decompress and add the field from *every*
> document to the suggester *every time you start Solr!* Likely not, but you
> can if you want to."
>
> However, there shouldn't be any documents, since it all got deleted and an
> empty database got set up to facilitate the restore. The slowness happens
> before the restore. With the fresh install.
>
> But yeah, I'll try it out, thanks for the links.
>
>
>
>
> On Thu, Nov 21, 2019 at 9:41 PM Dave <[hidden email]> wrote:
>
>> https://lucidworks.com/post/solr-suggester/
>>
>> You must set buildonstartup to false, the default is true. Try it
>>
>>> On Nov 21, 2019, at 3:21 PM, Koen De Groote <[hidden email]>
>> wrote:
>>>
>>> Erick:
>>>
>>> No suggesters. There is 1 spellchecker for
>>>
>>> <str name="queryAnalyzerFieldType">text_general</str>
>>>
>>> But no buildOnCommit or buildOnStartup setting mentioned anywhere.
>>>
>>> That being said, the point in time at which this occurs, the database is
>>> guaranteed to be empty, as the data folders had previously been deleted
>> and
>>> recreated empty. Then the docker container is restarted and this behavior
>>> is observed.
>>>
>>> Long shot, but even if Solr is getting data from zookeeper telling of
>> file
>>> locations and checking for the existence of these files... that should be
>>> pretty fast, I'd think.
>>>
>>> This is really disturbing. I know what to expect when recovering now, but
>>> someone doing this on a live environment that has to be up again ASAP is
>>> probably going to be sweating bullets.
>>>
>>>
>>> On Thu, Nov 21, 2019 at 2:45 PM Erick Erickson <[hidden email]>
>>> wrote:
>>>
>>>> Koen:
>>>>
>>>> Do you have any spellcheckers or suggesters defined with buildOnCommit
>> or
>>>> buildOnStartup set to “true”? Depending on the implementation, this may
>>>> have to read the stored data for the field used in the
>>>> suggester/spellchecker from _every_ document in your collection, which
>> can
>>>> take many minutes. Even if your implementation in your config is
>> file-based
>>>> it can still take a while.
>>>>
>>>> Shot in the dark….
>>>>
>>>> Erick
>>>>
>>>>> On Nov 21, 2019, at 4:03 AM, Koen De Groote <
>> [hidden email]>
>>>> wrote:
>>>>>
>>>>> The logs files showed a startup, printing of all the config options
>> that
>>>>> had been set, 1 or 2 commands that got executed and then nothing.
>>>>>
>>>>> Sending the curl did not get shown in the logs files until after that
>>>>> period where Solr became unresponsive.
>>>>>
>>>>> Service mesh, I don't think so? It's in a docker container, but that
>>>>> shouldn't be a problem, it usually never is.
>>>>>
>>>>>
>>>>> On Wed, Nov 20, 2019 at 10:42 AM Jörn Franke <[hidden email]>
>>>> wrote:
>>>>>
>>>>>> Have you checked the log files of Solr?
>>>>>>
>>>>>>
>>>>>> Do you have a service mesh in-between? Could it be something at the
>>>>>> network layer/container orchestration  that is blocking requests for
>>>> some
>>>>>> minutes?
>>>>>>
>>>>>>> Am 20.11.2019 um 10:32 schrieb Koen De Groote <
>>>>>> [hidden email]>:
>>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> I was testing some backup/restore scenarios.
>>>>>>>
>>>>>>> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
>>>>>>> SolrCloud, with zookeeper.
>>>>>>>
>>>>>>> The steps are as follows:
>>>>>>>
>>>>>>> 1. Manually delete the data folder.
>>>>>>> 2. Restart the container. The process is now in error mode,
>> complaining
>>>>>>> that it cannot find the cores.
>>>>>>> 3. Fix the install, meaning create new data folders, which are empty
>> at
>>>>>>> this point.
>>>>>>> 4. Restart the container again, to pick up the empty folders and not
>> be
>>>>>> in
>>>>>>> error anymore.
>>>>>>> 5. Perform the restore
>>>>>>> 6. Check if everything is available again
>>>>>>>
>>>>>>> The problem is between step 4 and 5. After step 4, it takes several
>>>>>> minutes
>>>>>>> before solr actually responds to curl commands.
>>>>>>>
>>>>>>> Once responsive, the restore happened just fine. But it's very
>>>> stressful
>>>>>> in
>>>>>>> a situation where you have to restore a production environment and
>> the
>>>>>>> process just doesn't respond for 5-10 minutes.
>>>>>>>
>>>>>>> We're talking about 20GB of data here, so not very much, but not
>> little
>>>>>>> either.
>>>>>>>
>>>>>>> Is it normal that it takes so long before solr responds? If not, what
>>>>>>> should I look at in order to find the cause?
>>>>>>>
>>>>>>> I have asked this before recently, though the wording was confusing.
>>>> This
>>>>>>> should be clearer.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Koen De Groote
>>>>>>
>>>>
>>>>
>>