Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

Sebastian Riemer
Hi,

we've recently changed our Solr-Version from 6.2.1 to 7.5.0, and since then, whenever we execute a query on solr, a new thread is being created and never closed.

These threads are all labelled "Connection evictor" and the gather until a critical mass is reached and either the OS cannot create anymore OS threads, or an out of memory error is being produced.

First I thought, that this might have as cause we were using a higher SolrJ-Version than our Solr-Server (by mistakenly forgetting to uprade the server version too):

So we had for SolrJ: 7.4.0

<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-solrj</artifactId>
    <version>7.4.0</version>
</dependency>

And for Solr-Server:  6.2.1

But now I just installed the newest Solr-Server-Version 7.5.0 and still I see with each Solr-Search performed an additional Thread being created and never released.

When downgrading SolrJ to 6.2.1 I can verify, that no new threads are created when doing a solr search.

What do you think about this? Are there any known pitfalls? Maybe I missed some crucial changes necessary when upgrading to 7.5.0?

What about differing versions in SolrJ and Solr-Server? As far as I recall the docs, one major-version-difference up/down in both ways should be o.k.

Thanks for all your feedback,

Yours sincerely

Sebastian Riemer
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

Jason Gerlowski
Hey Sebastian,

As for how Solr/SolrJ compatibility is handled, the story for SolrJ
looks a lot like the story for Solr itself - major version changes can
introduce breaking changes, so it is best to avoid using SolrJ 6.x
with Solr 7.x.  In practice I think changes that break Solr/SolrJ
compatibility are relatively rare though, so it might be possible if
your hand is forced.

As for the behavior you described...I think I understand what you're
describing, but to make sure:  Are the "connection-evictor" threads
accumulating in your client application, on the Solr server itself, or
both?

I suspect you're seeing this in your client code.  If so, it'd really
help us to help you if you could provide some more details on how
you're using SolrJ.  Can you share a small snippet (JUnit test?) that
reproduces the problem?  How are you creating the SolrClient you're
using to send requests?  Which SolrClient implementation(s) are you
using?  Are you providing your own HttpClient, or letting SolrClient
create its own?  It'll be much easier for others to help with a little
more detail there.

Best,

Jason

On Fri, Nov 23, 2018 at 10:38 AM Sebastian Riemer <[hidden email]> wrote:

>
> Hi,
>
> we've recently changed our Solr-Version from 6.2.1 to 7.5.0, and since then, whenever we execute a query on solr, a new thread is being created and never closed.
>
> These threads are all labelled "Connection evictor" and the gather until a critical mass is reached and either the OS cannot create anymore OS threads, or an out of memory error is being produced.
>
> First I thought, that this might have as cause we were using a higher SolrJ-Version than our Solr-Server (by mistakenly forgetting to uprade the server version too):
>
> So we had for SolrJ: 7.4.0
>
> <dependency>
>     <groupId>org.apache.solr</groupId>
>     <artifactId>solr-solrj</artifactId>
>     <version>7.4.0</version>
> </dependency>
>
> And for Solr-Server:  6.2.1
>
> But now I just installed the newest Solr-Server-Version 7.5.0 and still I see with each Solr-Search performed an additional Thread being created and never released.
>
> When downgrading SolrJ to 6.2.1 I can verify, that no new threads are created when doing a solr search.
>
> What do you think about this? Are there any known pitfalls? Maybe I missed some crucial changes necessary when upgrading to 7.5.0?
>
> What about differing versions in SolrJ and Solr-Server? As far as I recall the docs, one major-version-difference up/down in both ways should be o.k.
>
> Thanks for all your feedback,
>
> Yours sincerely
>
> Sebastian Riemer
Reply | Threaded
Open this post in threaded view
|

AW: Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

Sebastian Riemer
Dear Jason,

Thank you for your response! I'm happy to tell you that we did resolve our issue (by currently downgrading to 6.5 on both solrj-client and server side).

We were formerly executing queries by creating our HttpSolrClient this way (for each Query):

SolrClient client = new HttpSolrClient(urlString);

I'm not sure but I guess this way of creating a client is deprecated now? Anyhow a colleague changed this to:

SolrClient client = new HttpSolrClient.Builder(urlString).build();

Again, this is done _for each query_ we execute.

As we discovered by now (like here http://lucene.472066.n3.nabble.com/6-6-gt-7-5-SolrJ-seeing-many-quot-Connection-evictor-quot-Threads-td4410488.html) this is not the correct way to do it. Instead we should only create one client per core and reuse it.

Interestingly enough, with 6.5 the HttpSolrClient.Builder-way of creating a new client for every query seems to not create new threads for each query (or it closes It automatically or reuses existing I don't know).

Maybe it was a bad idea in the first place, to create a new client for every query.

Anyways, thanks a lot for your answer - we'll definitely have to revisit the way we create the HttpSolrClient and check for the recommended way of doing so.

All the best,

Sebastian


-----Urspr√ľngliche Nachricht-----
Von: Jason Gerlowski [mailto:[hidden email]]
Gesendet: Montag, 26. November 2018 17:55
An: [hidden email]
Betreff: Re: Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

Hey Sebastian,

As for how Solr/SolrJ compatibility is handled, the story for SolrJ looks a lot like the story for Solr itself - major version changes can introduce breaking changes, so it is best to avoid using SolrJ 6.x with Solr 7.x.  In practice I think changes that break Solr/SolrJ compatibility are relatively rare though, so it might be possible if your hand is forced.

As for the behavior you described...I think I understand what you're describing, but to make sure:  Are the "connection-evictor" threads accumulating in your client application, on the Solr server itself, or both?

I suspect you're seeing this in your client code.  If so, it'd really help us to help you if you could provide some more details on how you're using SolrJ.  Can you share a small snippet (JUnit test?) that reproduces the problem?  How are you creating the SolrClient you're using to send requests?  Which SolrClient implementation(s) are you using?  Are you providing your own HttpClient, or letting SolrClient create its own?  It'll be much easier for others to help with a little more detail there.

Best,

Jason

On Fri, Nov 23, 2018 at 10:38 AM Sebastian Riemer <[hidden email]> wrote:

>
> Hi,
>
> we've recently changed our Solr-Version from 6.2.1 to 7.5.0, and since then, whenever we execute a query on solr, a new thread is being created and never closed.
>
> These threads are all labelled "Connection evictor" and the gather until a critical mass is reached and either the OS cannot create anymore OS threads, or an out of memory error is being produced.
>
> First I thought, that this might have as cause we were using a higher SolrJ-Version than our Solr-Server (by mistakenly forgetting to uprade the server version too):
>
> So we had for SolrJ: 7.4.0
>
> <dependency>
>     <groupId>org.apache.solr</groupId>
>     <artifactId>solr-solrj</artifactId>
>     <version>7.4.0</version>
> </dependency>
>
> And for Solr-Server:  6.2.1
>
> But now I just installed the newest Solr-Server-Version 7.5.0 and still I see with each Solr-Search performed an additional Thread being created and never released.
>
> When downgrading SolrJ to 6.2.1 I can verify, that no new threads are created when doing a solr search.
>
> What do you think about this? Are there any known pitfalls? Maybe I missed some crucial changes necessary when upgrading to 7.5.0?
>
> What about differing versions in SolrJ and Solr-Server? As far as I recall the docs, one major-version-difference up/down in both ways should be o.k.
>
> Thanks for all your feedback,
>
> Yours sincerely
>
> Sebastian Riemer