Could not load collection from ZK:

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Could not load collection from ZK:

Aman Deep Singh
I'm facing a issue in solr sometimes zookeeper failes to load the solr
collection stating

org.apache.solr.common.SolrException: Could not load collection from ZK:

My current setup details is

   1. 5 Nodes  with 4 cores  ,7.6 GB RAM each which contains solr node and
   zookeeper
   2. No sharding is used
   3. index size is around 2.5 GB
   4. solrnode RAM-4GB
   5. Average load -10k RPM
   6. Indexing maximum 1000 docs/Minute  (around 100 batch of 5 to 10 docs
   each)

The GC logs analysis is also normal
Node 1-
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4yLnppcC0tMTEtNTctMw==
Node 2-
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4zLmN1cnJlbnQuemlwLS03LTQ2LTU2
Node 3-
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4zLmN1cnJlbnQuemlwLS04LTIzLTM5
Node 4-
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4yLnppcC0tOC0yMC01NQ==
Node 5-
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy41LmN1cnJlbnQuemlwLS04LTE5LTE0


Admin UI image https://pasteboard.co/1Nd7ArAf0.png

Any idea what causing this problem and how to overcame from this.

Thanks,
Aman Deep Singh
Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Shawn Heisey-2
On 6/20/2017 6:08 AM, Aman Deep Singh wrote:
> I'm facing a issue in solr sometimes zookeeper failes to load the solr
> collection stating
>
> org.apache.solr.common.SolrException: Could not load collection from ZK:

This is not the full error message.  It will be dozens of lines long,
and may contain one or more "Caused by" sections with separate
stacktraces.  We need the whole thing, with complete text of all lines
included.  If you are looking at the admin UI to see these messages, you
need to look in the solr.log file instead.  It looks like you've used
the service installer script, so you should find that in /var/solr/logs.

> My current setup details is
>
>    1. 5 Nodes  with 4 cores  ,7.6 GB RAM each which contains solr node and
>    zookeeper
>    2. No sharding is used
>    3. index size is around 2.5 GB
>    4. solrnode RAM-4GB
>    5. Average load -10k RPM
>    6. Indexing maximum 1000 docs/Minute  (around 100 batch of 5 to 10 docs
>    each)
>
> The GC logs analysis is also normal
> Node 1-
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4yLnppcC0tMTEtNTctMw==
> Node 2-
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4zLmN1cnJlbnQuemlwLS03LTQ2LTU2
> Node 3-
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4zLmN1cnJlbnQuemlwLS04LTIzLTM5
> Node 4-
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy4yLnppcC0tOC0yMC01NQ==
> Node 5-
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMjAvLS1zb2xyX2djLmxvZy41LmN1cnJlbnQuemlwLS04LTE5LTE0

Overall the GC stats look good.  There are a few long GCs, but the "stop
the world" stats are pretty good.

> Admin UI image https://pasteboard.co/1Nd7ArAf0.png

This screenshot indicates that this system is using swap.  This is
usually an indication that there's an overall memory shortage on the
system ... but if there is no *active* swapping, there's a *small*
chance that this is not a problem.  What I would like to see is a
screenshot with the "top" program running (not htop or any other
variant, I'm after "top" itself), with shift-M pressed to sort the list
by memory usage.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Aman Deep Singh
This error is coming in the application which is using solrj to communicate
to the solr
full stacktrace is

Request processing failed; nested exception is com.gdn.solr620.org.apache.
solr.common.SolrException: Could not load collection from ZK:
productCollection
org.springframework.web.util.NestedServletException: Request processing
failed; nested exception is com.gdn.solr620.org.apache.solr.common.
SolrException: Could not load collection from ZK: productCollection at org.
springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet
.java:973) ~[spring-webmvc-4.1.0.RELEASE.jar:4.1.0.RELEASE] at org.
springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:
863) ~[spring-webmvc-4.1.0.RELEASE.jar:4.1.0.RELEASE] at javax.servlet.http.
HttpServlet.service(HttpServlet.java:648) ~[servlet-api.jar:na] at org.
springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:
837) ~[spring-webmvc-4.1.0.RELEASE.jar:4.1.0.RELEASE] at javax.servlet.http.
HttpServlet.service(HttpServlet.java:729) ~[servlet-api.jar:na] at org.
apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:230) [catalina.jar:8.5.4] at org.apache.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [
catalina.jar:8.5.4] at org.apache.tomcat.websocket.server.WsFilter.doFilter(
WsFilter.java:52) ~[tomcat-websocket.jar:8.5.4] at org.apache.catalina.core.
ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [
catalina.jar:8.5.4] at org.apache.catalina.core.ApplicationFilterChain.
doFilter(ApplicationFilterChain.java:165) [catalina.jar:8.5.4] at org.
springframework.web.filter.CharacterEncodingFilter.doFilterInternal(
CharacterEncodingFilter.java:88) ~[spring-web-4.1.0.RELEASE.jar:4.1.0.
RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(
OncePerRequestFilter.java:107) [spring-web-4.1.0.RELEASE.jar:4.1.0.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:192) [catalina.jar:8.5.4] at org.apache.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [
catalina.jar:8.5.4] at com.gdn.x.seoul.web.ui.filter.
ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:64) ~[
classes/:na] at org.apache.catalina.core.ApplicationFilterChain.
internalDoFilter(ApplicationFilterChain.java:192) [catalina.jar:8.5.4] at
org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:165) [catalina.jar:8.5.4] at com.gdn.x.seoul.web
.ui.filter.RedirectionFilter.doFilter(RedirectionFilter.java:61) [classes/:
na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:192) [catalina.jar:8.5.4] at org.apache.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [
catalina.jar:8.5.4] at com.gdn.x.seoul.web.util.AccessLogFilter.doFilter(
AccessLogFilter.java:83) [seoul-common-web-4.11.3.jar:na] at org.apache.
catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain
.java:192) [catalina.jar:8.5.4] at org.apache.catalina.core.
ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [catalina.
jar:8.5.4] at org.springframework.security.web.FilterChainProxy$
VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-
3.2.5.RELEASE.jar:3.2.5.RELEASE] at org.springframework.security.web.access.
intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:
118) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE] at org.
springframework.security.web.access.intercept.FilterSecurityInterceptor.
doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.5.
RELEASE.jar:3.2.5.RELEASE] at org.springframework.security.web.
FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [
spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE] at org.springframework.
security.web.access.ExceptionTranslationFilter.doFilter(
ExceptionTranslationFilter.java:113) [spring-security-web-3.2.5.RELEASE.jar:
3.2.5.RELEASE] at org.springframework.security.web.FilterChainProxy$
VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-
3.2.5.RELEASE.jar:3.2.5.RELEASE] at org.springframework.security.web.
authentication.AnonymousAuthenticationFilter.doFilter(
AnonymousAuthenticationFilter.java:113) [spring-security-web-3.2.5.RELEASE.
jar:3.2.5.RELEASE] at org.springframework.security.web.FilterChainProxy$
VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-
3.2.5.RELEASE.jar:3.2.5.RELEASE] at org.springframework.security.web.
servletapi.SecurityContextHolderAwareRequestFilter.doFilter(
SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.5
.RELEASE.jar:3.2.5.RELEASE]

Top command images are at
https://www.dropbox.com/sh/vxorykk8tmb6amb/AABYIcFuRyfSnlkS6I-Tr5HNa?dl=0
Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Aman Deep Singh
Sorry Shawn,
It didn't copy entire stacktrace I put the stacktrace at
https://www.dropbox.com/s/zf8b87m24ei2ils/solr%20exception2?dl=0

Note: I have shaded the solr library under com.gdn.solr620  so all solr
class will be appear as com.gdn.solr620.org.apache.solr.*



On Tue, Jun 20, 2017 at 8:09 PM Shawn Heisey <[hidden email]> wrote:

> On 6/20/2017 8:25 AM, Aman Deep Singh wrote:
> > This error is coming in the application which is using solrj to
> communicate
> > to the solr
> > full stacktrace is
> >
> > Request processing failed; nested exception is
> com.gdn.solr620.org.apache.
> > solr.common.SolrException: Could not load collection from ZK:
> > productCollection
> <snip>
> > Top command images are at
> >
> https://www.dropbox.com/sh/vxorykk8tmb6amb/AABYIcFuRyfSnlkS6I-Tr5HNa?dl=0
>
> Are there any "Caused by" clauses that come after that stacktrace?  It
> doesn't seem to be complete.  There are no Solr classes shown, so I
> can't see where in Solr code the exception occurred.  If it happened in
> Solr code, then there should be more to the error message.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Alessandro Benedetti
hi Aman,
I had similar issues in the past and the reason was attributed to :

SOLR-8868 <https://issues.apache.org/jira/browse/SOLR-8868>  

Which unfortunately is not solved yet.

Did you manage to find a different cause in your case?

hope that helps.

Regards



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Shawn Heisey-2
In reply to this post by Aman Deep Singh
On 6/20/2017 9:46 AM, Aman Deep Singh wrote:
> Sorry Shawn,
> It didn't copy entire stacktrace I put the stacktrace at
> https://www.dropbox.com/s/zf8b87m24ei2ils/solr%20exception2?dl=0
>
> Note: I have shaded the solr library under com.gdn.solr620  so all solr
> class will be appear as com.gdn.solr620.org.apache.solr.*

Wow, I really dropped the ball here.  The thread is nearly a year old. 
I somehow missed the reply.  I am sorry about that!  Thank Alessandro
for reviving the thread and making it clear that I never replied.

This is the innermost cause:

Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for
/collections/productCollection/state.json

Either there are network issues talking to ZooKeeper, or something else
caused a timeout.  Solr's default ZK client timeout when it is not
configured is 15 seconds.  In recent versions, the example
configurations have an explicit setting of 30 seconds.  Solr's
zkClientTimeout is used to set ZooKeeper's sessionTimeout, and that's
what is exceeded when a session expires.

When this kind of error happens, it means something has gone VERY wrong
-- 15 seconds is a REALLY long time when programs are trying to talk to
each other.

One common cause of problems like this is extreme GC pauses.  Typically
a pause problem capable of causing a ZK timeout would be due to the heap
being too small, but it's always possible that it could happen because
the heap is VERY large.

Errors on the client side may not be as informative as corresponding
errors in the solr.log file on the server(s).  It would be a good idea
to check solr.log for errors as well.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Aman Deep Singh
Hi Shawn & Alessandro,
We have tried to increase the heap also but we were facing the same issue but after removing the ZK from the solr server to their dedicated server this problem goes away, Yes when we are facing  this issue the GC activity was high around 60-70% out of 400%.
Regards,
Aman Deep singh

On 25/05/18, 5:08 AM, "Shawn Heisey" <[hidden email]> wrote:

    On 6/20/2017 9:46 AM, Aman Deep Singh wrote:
    > Sorry Shawn,
    > It didn't copy entire stacktrace I put the stacktrace at
    > https://www.dropbox.com/s/zf8b87m24ei2ils/solr%20exception2?dl=0
    >
    > Note: I have shaded the solr library under com.gdn.solr620  so all solr
    > class will be appear as com.gdn.solr620.org.apache.solr.*
   
    Wow, I really dropped the ball here.  The thread is nearly a year old.
    I somehow missed the reply.  I am sorry about that!  Thank Alessandro
    for reviving the thread and making it clear that I never replied.
   
    This is the innermost cause:
   
    Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
    KeeperErrorCode = Session expired for
    /collections/productCollection/state.json
   
    Either there are network issues talking to ZooKeeper, or something else
    caused a timeout.  Solr's default ZK client timeout when it is not
    configured is 15 seconds.  In recent versions, the example
    configurations have an explicit setting of 30 seconds.  Solr's
    zkClientTimeout is used to set ZooKeeper's sessionTimeout, and that's
    what is exceeded when a session expires.
   
    When this kind of error happens, it means something has gone VERY wrong
    -- 15 seconds is a REALLY long time when programs are trying to talk to
    each other.
   
    One common cause of problems like this is extreme GC pauses.  Typically
    a pause problem capable of causing a ZK timeout would be due to the heap
    being too small, but it's always possible that it could happen because
    the heap is VERY large.
   
    Errors on the client side may not be as informative as corresponding
    errors in the solr.log file on the server(s).  It would be a good idea
    to check solr.log for errors as well.
   
    Thanks,
    Shawn
   
   


Reply | Threaded
Open this post in threaded view
|

Re: Could not load collection from ZK:

Matteo Grolla
Hi everybody,
    I'm facing the same problem on solr 7.3.
Probably requesting a longer session to zk (the default 10s seems too
short) will solve the problem but I'm puzzled by the fact that this error
is reported by solrj as a SolrException with status code 400 (BAD_REQUEST).
in ZkStateReader

   public static DocCollection getCollectionLive(ZkStateReader zkStateReader,
String coll) {

  try {
    return zkStateReader.fetchCollectionState(coll, null);
  } catch (KeeperException e) {
    throw new SolrException(ErrorCode.BAD_REQUEST, "Could not load
collection from ZK: " + coll, e);
  } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
    throw new SolrException(ErrorCode.BAD_REQUEST, "Could not load
collection from ZK: " + coll, e);
  }
}


Retrying the reques could solve the problem, but a client should't retry a
BAD_REQUEST.
Why isn't this reported as a 503 (SERVICE_UNAVAILABLE) ?
I think solrj should distinguish the cases:
A: communication problem with zk -> 503
B: user asked a non existing collection ->400

Thanks


Il giorno ven 25 mag 2018 alle ore 05:02 Aman Singh <
[hidden email]> ha scritto:

> Hi Shawn & Alessandro,
> We have tried to increase the heap also but we were facing the same issue
> but after removing the ZK from the solr server to their dedicated server
> this problem goes away, Yes when we are facing  this issue the GC activity
> was high around 60-70% out of 400%.
> Regards,
> Aman Deep singh
>
> On 25/05/18, 5:08 AM, "Shawn Heisey" <[hidden email]> wrote:
>
>     On 6/20/2017 9:46 AM, Aman Deep Singh wrote:
>     > Sorry Shawn,
>     > It didn't copy entire stacktrace I put the stacktrace at
>     > https://www.dropbox.com/s/zf8b87m24ei2ils/solr%20exception2?dl=0
>     >
>     > Note: I have shaded the solr library under com.gdn.solr620  so all
> solr
>     > class will be appear as com.gdn.solr620.org.apache.solr.*
>
>     Wow, I really dropped the ball here.  The thread is nearly a year old.
>     I somehow missed the reply.  I am sorry about that!  Thank Alessandro
>     for reviving the thread and making it clear that I never replied.
>
>     This is the innermost cause:
>
>     Caused by:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
>     KeeperErrorCode = Session expired for
>     /collections/productCollection/state.json
>
>     Either there are network issues talking to ZooKeeper, or something else
>     caused a timeout.  Solr's default ZK client timeout when it is not
>     configured is 15 seconds.  In recent versions, the example
>     configurations have an explicit setting of 30 seconds.  Solr's
>     zkClientTimeout is used to set ZooKeeper's sessionTimeout, and that's
>     what is exceeded when a session expires.
>
>     When this kind of error happens, it means something has gone VERY wrong
>     -- 15 seconds is a REALLY long time when programs are trying to talk to
>     each other.
>
>     One common cause of problems like this is extreme GC pauses.  Typically
>     a pause problem capable of causing a ZK timeout would be due to the
> heap
>     being too small, but it's always possible that it could happen because
>     the heap is VERY large.
>
>     Errors on the client side may not be as informative as corresponding
>     errors in the solr.log file on the server(s).  It would be a good idea
>     to check solr.log for errors as well.
>
>     Thanks,
>     Shawn
>
>
>
>
>