Async RELOADCOLLECTION never completes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Async RELOADCOLLECTION never completes

Karl Stoney
Hi,
We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

❯ curl -s http://solr.search-solr.prod.k8.atcloud.io/solr/admin/collections\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
{
  "responseHeader": {
    "status": 0,
    "QTime": 2
  },
  "status": {
    "state": "running",
    "msg": "found [1581585716] in running tasks"
  }
}

The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

The only way I’ve found to clear this up is to rolling restart solr.

Solr 8.4.1

Any ideas?
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Reply | Threaded
Open this post in threaded view
|

Re: Async RELOADCOLLECTION never completes

Karl Stoney
When performing a rolling restart we see:

09:43:31.890 [OverseerThreadFactory-42-thread-5-processing-n:solr-5.search-solr.prod.k8.atcloud.io:80_solr] ERROR org.apache.solr.cloud.OverseerTaskProcessor - :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-map-failure

Which I find interesting, everything (resources wise) is very healthy.

On 13/02/2020, 09:34, "Karl Stoney" <[hidden email]> wrote:

    Hi,
    We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

    ❯ curl -s https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr.search-solr.prod.k8.atcloud.io%2Fsolr%2Fadmin%2Fcollections&amp;data=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C3a627213825a4b56415008d7b067eb73%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637171832681589729&amp;sdata=Kx3OY%2BMkXw%2Bob0M0ZRmnehfAxffoSdGvJyV%2FlfdwdxY%3D&amp;reserved=0\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
    {
      "responseHeader": {
        "status": 0,
        "QTime": 2
      },
      "status": {
        "state": "running",
        "msg": "found [1581585716] in running tasks"
      }
    }

    The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

    The only way I’ve found to clear this up is to rolling restart solr.

    Solr 8.4.1

    Any ideas?
    This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.


This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Reply | Threaded
Open this post in threaded view
|

Re: Async RELOADCOLLECTION never completes

Karl Stoney
I actually narrowed this down to changing the schema version from 1.5 to 1.6 and then doing a RELOADCOLLECTION, it consistently hangs.  Several of our nodes go briefly into a recovering state too.
________________________________
From: Karl Stoney <[hidden email]>
Sent: 13 February 2020 09:49
To: [hidden email] <[hidden email]>
Subject: Re: Async RELOADCOLLECTION never completes

When performing a rolling restart we see:

09:43:31.890 [OverseerThreadFactory-42-thread-5-processing-n:solr-5.search-solr.prod.k8.atcloud.io:80_solr] ERROR org.apache.solr.cloud.OverseerTaskProcessor - :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-map-failure

Which I find interesting, everything (resources wise) is very healthy.

On 13/02/2020, 09:34, "Karl Stoney" <[hidden email]> wrote:

    Hi,
    We’re periodically seeing an ASYNC task to RELOADCOLLECTION never complete, it’s just permanently “running”:

    ❯ curl -s https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr.search-solr.prod.k8.atcloud.io%2Fsolr%2Fadmin%2Fcollections&amp;data=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C7a5d7c308dac49b5ab3408d7b06a0ff3%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637171841881478917&amp;sdata=fjl0QkOCCanmgUHAXX%2F5HvzBcexCmugLItWNIzG1D%2Fc%3D&amp;reserved=0\?action\=REQUESTSTATUS\&requestid\=1581585716 | jq .
    {
      "responseHeader": {
        "status": 0,
        "QTime": 2
      },
      "status": {
        "state": "running",
        "msg": "found [1581585716] in running tasks"
      }
    }

    The collection appears to have been reloaded fine (from the gui, it’s using the right config), so we’re a bit baffled.

    The only way I’ve found to clear this up is to rolling restart solr.

    Solr 8.4.1

    Any ideas?
    This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.


This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
This e-mail is sent on behalf of Auto Trader Group Plc, Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 9439967). This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.