SolrCloud 5.3.1 "IndexWriter is closed"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrCloud 5.3.1 "IndexWriter is closed"

Kelly, Frank
Just wondering if anyone has seen this before and might understand why this is happening

When we deploy a new schema.xml adding some new search terms we get the dreaded “IndexWriter is closed” exception and the only solution we have found to date is a Solr restart :-(

Environment:
  • Solr 5.3.1 in Solr Cloud (3 shards each with 3 replicas across 3 EC2 Vms)
  • 300m+ documents (60+ GB index)
  • Relevant settings?
    • SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=1000
    • SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=300000"

Is there anything “unsafe” about reload on a collection that is handling live traffic?

Cheers!

-Frank

 

Frank Kelly

Principal Software Engineer

Identity Profile Team (SCBE, Traces, CDA) 


HERE 

5 Wayside Rd, Burlington, MA 01803, USA

42° 29' 7" N 71° 11' 32" W

 

               
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud 5.3.1 "IndexWriter is closed"

Rick Leir-2
Frank,

I assume you re-index everything after changing schema.xml?

cheers -- Rick


On 2017-09-11 04:38 PM, Kelly, Frank wrote:

> Just wondering if anyone has seen this before and might understand why
> this is happening
>
> When we deploy a new schema.xml adding some new search terms we get
> the dreaded “IndexWriter is closed” exception and the only solution we
> have found to date is a Solr restart :-(
>
> Environment:
>
>   * Solr 5.3.1 in Solr Cloud (3 shards each with 3 replicas across 3
>     EC2 Vms)
>   * 300m+ documents (60+ GB index)
>   * Relevant settings?
>       o SOLR_OPTS="$SOLR_OPTS -Dsolr.autoSoftCommit.maxTime=1000”
>       o SOLR_OPTS="$SOLR_OPTS -Dsolr.autoCommit.maxTime=300000"
>
>
> Is there anything “unsafe” about reload on a collection that is
> handling live traffic?
> Could be related to https://issues.apache.org/jira/browse/SOLR-9830
> And https://issues.apache.org/jira/browse/SOLR-9699 in particular?
>
> Cheers!
>
> -Frank
>
> Description: Macintosh
> HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf
>
> *Frank Kelly*
>
> *Principal Software Engineer*
>
> Identity Profile Team (SCBE, Traces, CDA)
>
>
> HERE
>
> 5 Wayside Rd, Burlington, MA 01803, USA
>
> *42° 29' 7" N 71° 11' 32" W*
>
> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif
> <http://360.here.com/> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif
> <https://www.twitter.com/here> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif
> <https://www.facebook.com/here> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif
> <https://www.linkedin.com/company/heremaps> Description:
> /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif
> <https://www.instagram.com/here/>

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud 5.3.1 "IndexWriter is closed"

Kelly, Frank
No - these are new terms for new documents we will be adding later so no
need to reindex old documents.

Frank

 
Frank Kelly
Principal Software Engineer
Identity Profile Team (SCBE, Traces, CDA)

HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
 
 <http://360.here.com/>     <https://www.twitter.com/here>
<https://www.facebook.com/here>
<https://www.linkedin.com/company/heremaps>
<https://www.instagram.com/here/>



On 9/12/17, 6:27 AM, "Rick Leir" <[hidden email]> wrote:

>Frank,
>
>I assume you re-index everything after changing schema.xml?
>
>cheers -- Rick
>

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud 5.3.1 "IndexWriter is closed"

Susheel Kumar-3
Kelly -

If you do not make any change to schema and just reload your collection,
does it work fine? How much time it takes to reload the collection?

I am suspecting some conflict with commit frequency (5mins) and collection
reload.

Thnx

On Tue, Sep 12, 2017 at 6:59 AM, Kelly, Frank <[hidden email]> wrote:

> No - these are new terms for new documents we will be adding later so no
> need to reindex old documents.
>
> Frank
>
>
> Frank Kelly
> Principal Software Engineer
> Identity Profile Team (SCBE, Traces, CDA)
>
> HERE
> 5 Wayside Rd, Burlington, MA 01803, USA
> 42° 29' 7" N 71° 11' 32" W
>
>  <http://360.here.com/>     <https://www.twitter.com/here>
> <https://www.facebook.com/here>
> <https://www.linkedin.com/company/heremaps>
> <https://www.instagram.com/here/>
>
>
>
> On 9/12/17, 6:27 AM, "Rick Leir" <[hidden email]> wrote:
>
> >Frank,
> >
> >I assume you re-index everything after changing schema.xml?
> >
> >cheers -- Rick
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud 5.3.1 "IndexWriter is closed"

Kelly, Frank
The schema change doesn¹t seem to be making any difference - just the act
of a reload whilst handling live traffic.

The reload takes about 30 seconds and soon after (within a few seconds) we
start to see IndexWriter closed exceptions

-Frank

 
Frank Kelly
Principal Software Engineer
Identity Profile Team (SCBE, Traces, CDA)

HERE
5 Wayside Rd, Burlington, MA 01803, USA
42° 29' 7" N 71° 11' 32" W
 
 <http://360.here.com/>     <https://www.twitter.com/here>
<https://www.facebook.com/here>
<https://www.linkedin.com/company/heremaps>
<https://www.instagram.com/here/>



On 9/12/17, 9:13 AM, "Susheel Kumar" <[hidden email]> wrote:

>Kelly -
>
>If you do not make any change to schema and just reload your collection,
>does it work fine? How much time it takes to reload the collection?
>
>I am suspecting some conflict with commit frequency (5mins) and collection
>reload.
>
>Thnx

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud 5.3.1 "IndexWriter is closed"

Susheel Kumar-3
During Reload collection, it does close the IndexWriter and opens a new.
So looks like with live traffic (is it heavy traffic, how many qps?) and
collection reload, its running into a situation when IndexWriter is not
able to open up again.  Not sure if this issue might have been fixed in 5
or 6.x and may work fine in latest 6.6.1 if you could test anyway.

https://www.quora.com/What-happens-during-a-Solr-Core-Reload


On Tue, Sep 12, 2017 at 12:47 PM, Kelly, Frank <[hidden email]> wrote:

> The schema change doesn¹t seem to be making any difference - just the act
> of a reload whilst handling live traffic.
>
> The reload takes about 30 seconds and soon after (within a few seconds) we
> start to see IndexWriter closed exceptions
>
> -Frank
>
>
> Frank Kelly
> Principal Software Engineer
> Identity Profile Team (SCBE, Traces, CDA)
>
> HERE
> 5 Wayside Rd, Burlington, MA 01803, USA
> 42° 29' 7" N 71° 11' 32" W
>
>  <http://360.here.com/>     <https://www.twitter.com/here>
> <https://www.facebook.com/here>
> <https://www.linkedin.com/company/heremaps>
> <https://www.instagram.com/here/>
>
>
>
> On 9/12/17, 9:13 AM, "Susheel Kumar" <[hidden email]> wrote:
>
> >Kelly -
> >
> >If you do not make any change to schema and just reload your collection,
> >does it work fine? How much time it takes to reload the collection?
> >
> >I am suspecting some conflict with commit frequency (5mins) and collection
> >reload.
> >
> >Thnx
>
>