Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

harjags
2019-07-09 17:37:30.053 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore [promotion_rules-v1_shard1_replica_n1]
**CLOSING SolrCore org.apache.solr.core.SolrCore**@44de0f89
2019-07-09 17:37:30.054 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.promotion_rules-v1.shard1.replica_n1, tag=44de0f89
2019-07-09 17:37:30.056 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.m.r.SolrJmxReporter Closing reporter
[org.apache.solr.metrics.reporters.SolrJmxReporter@36a60356: rootName =
null, domain = solr.core.promotion_rules-v1.shard1.replica_n1, service url =
null, agent id = null] for registry
solr.core.promotion_rules-v1.shard1.replica_n1 /
com.codahale.metrics.MetricRegistry@79be463d
2019-07-09 17:37:30.148 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.promotion_rules-v1.shard1.leader, tag=44de0f89
2019-07-09 17:37:30.159 INFO
(facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.u.p.DocExpirationUpdateProcessorFactory
Triggering Graceful close of DocExpiration Executor
2019-07-09 17:38:03.062 **ERROR
(facetExecutor-67-thread-26-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-4] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]**
2019-07-09 17:38:03.067 ERROR
(facetExecutor-67-thread-28-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] **o.a.s.c.SolrCore Too many close [count:-5] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]**
2019-07-09 17:38:06.176 ERROR
(facetExecutor-67-thread-25-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-7] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:38:07.602 ERROR
(facetExecutor-67-thread-24-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-6] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:38:11.868 ERROR
(facetExecutor-67-thread-23-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-8] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:38:32.522 ERROR
(facetExecutor-67-thread-4-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] **o.a.s.c.SolrCore Too many close [count:-9] on**
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:08.862 ERROR
(facetExecutor-67-thread-3-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-10] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:08.988 ERROR
(facetExecutor-67-thread-6-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-12] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:09.048 ERROR
(facetExecutor-67-thread-5-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-13] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:10.474 ERROR
(facetExecutor-67-thread-2-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-11] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:10.572 ERROR
(facetExecutor-67-thread-7-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-14] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]
2019-07-09 17:39:10.646 ERROR
(facetExecutor-67-thread-9-processing-n:127.0.0.1:8983_solr
x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1
r:core_node22) [   ] o.a.s.c.SolrCore Too many close [count:-15] on
org.apache.solr.core.SolrCore@44de0f89. Please report this exception to
[hidden email]




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

harjags
SolrQuery used:
q=*:*&defType=edismax&fl=comboid,skuid&start=0&rows=15&facet.limit=100&variant=A&debugQuery=true&debug.explain.structured=true&fq=!status_facet:inactive&fq=!mode:AF_ONLY&fq=**{!join+from=skuid+to=skuid+fromIndex=promotion_rules}id:promo-opera-test-1000_promo-opera-test-1000**&enableManagedResourceRange=true&facet=true&facet.mincount=1&**facet.threads=32**&facet.field=dynamiccategory_facet&facet.recommend.maxfacets=30&facet.recommend.debug=true&sort=ratingandreviewsort+desc&qt=/promo&timeAllowed=5000


We are migrating to 7.6. We have been using joins with multithreaded facets
for years now.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

harjags
Few deatils added.

public void close() {
    int count = refCount.decrementAndGet();
    if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
    if (count < 0) {
      log.error("Too many close [count:{}] on {}. Please report this
exception to [hidden email]", count, this );
      assert false : "Too many closes on SolrCore";
      return;
    }
    log.info("{} CLOSING SolrCore {}", logid, this);

    // stop reporting metrics
    try {
      coreMetricManager.close();
    } catch (Throwable e) {
      SolrException.log(log, e);
      if (e instanceof  Error) {
        throw (Error) e;
      }
    }}

The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.

Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.

Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.

if (fromRef != null) {
          final RefCounted<SolrIndexSearcher> ref = fromRef;
          info.addCloseHook(new Closeable() {
            @Override
            public void close() {
              ref.decref();
            }
          });
        }

        info.addCloseHook(new Closeable() {
          @Override
          public void close() {
            fromCore.close();
          }
        });



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

harjags
In reply to this post by harjags
Few deatils added.

public void close() {
    int count = refCount.decrementAndGet();
    if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
    if (count < 0) {
      log.error("Too many close [count:{}] on {}. Please report this
exception to [hidden email]", count, this );
      assert false : "Too many closes on SolrCore";
      return;
    }
    log.info("{} CLOSING SolrCore {}", logid, this);

    // stop reporting metrics
    try {
      coreMetricManager.close();
    } catch (Throwable e) {
      SolrException.log(log, e);
      if (e instanceof  Error) {
        throw (Error) e;
      }
    }}

The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.

Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.

Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.

if (fromRef != null) {
          final RefCounted<SolrIndexSearcher> ref = fromRef;
          info.addCloseHook(new Closeable() {
            @Override
            public void close() {
              ref.decref();
            }
          });
        }

        info.addCloseHook(new Closeable() {
          @Override
          public void close() {
            fromCore.close();
          }
        });



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

Mikhail Khludnev-2
This case is worth to be covered with test. Beside of that, if join is
executed in multiple threads it might make bigger footprint that gain from
employing threads. Does it make and impact for consequent searches?

On Thu, Aug 22, 2019 at 8:42 AM harjagsbby <[hidden email]>
wrote:

> Few deatils added.
>
> public void close() {
>     int count = refCount.decrementAndGet();
>     if (count > 0) return; // close is called often, and only actually
> closes if nothing is using it.
>     if (count < 0) {
>       log.error("Too many close [count:{}] on {}. Please report this
> exception to [hidden email]", count, this );
>       assert false : "Too many closes on SolrCore";
>       return;
>     }
>     log.info("{} CLOSING SolrCore {}", logid, this);
>
>     // stop reporting metrics
>     try {
>       coreMetricManager.close();
>     } catch (Throwable e) {
>       SolrException.log(log, e);
>       if (e instanceof  Error) {
>         throw (Error) e;
>       }
>     }}
>
> The above code has refCount in SolrCore.java this atomicRef is in negative
> when we do multi threaded facets and joins.
>
> Can anyone explain the semaphore implemenattion here. What would be
> scenario
> when this can go below 0. I can clearly see each thread of multithreaded
> facet trying to close the core.
>
> Every Thread of facet tries to call createWeight of JoinQParserPlugin and
> attaches closeHooks which getscalled from
> SolrRequestInfo.clearRequestInfo()
> .
>
> if (fromRef != null) {
>           final RefCounted<SolrIndexSearcher> ref = fromRef;
>           info.addCloseHook(new Closeable() {
>             @Override
>             public void close() {
>               ref.decref();
>             }
>           });
>         }
>
>         info.addCloseHook(new Closeable() {
>           @Override
>           public void close() {
>             fromCore.close();
>           }
>         });
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

harjags
This post was updated on .
I dint quite get your question.Our Solr collection under contention is not
shareded. Data being joint is collocated.INfact multithreaded facet help our
performance.

Afer debugging more solr 7 calls JoinQParserPlugin createWeight is called by
each thread of multithreaded facets but not in solr 6. Is this something to
do with changes to underlying Threadpool and threadlocal variables?



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions

Mikhail Khludnev-2
Raised  https://issues.apache.org/jira/browse/SOLR-13715

On Fri, Aug 23, 2019 at 12:20 PM harjagsbby <[hidden email]>
wrote:

> I dint quite get your question.Our Solr collection under contention is not
> shareded. Data being joint is collocated.INfact multithreaded facet help
> our
> performance.
>
> Afer debugging more solr 7 calls JoinQParserPlugin createWeight is called
> by
> each thread of multithreaded facets but not in solr 6. Is this something to
> do with changes to ExecutorUtils awaitTermination?
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


--
Sincerely yours
Mikhail Khludnev