Connection mismanagement in Solrj?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Connection mismanagement in Solrj?

Walter Underwood, Netflix
We just switched to Solrj from a home-grown client and we have a huge
jump in the number of connections to the server, enough that our
load balancer was rejecting connections in production tonight.

Does that sound familiar? We're running 1.3.

I set the timeouts and connection pools to the same values I'd
used in my other code, also based on HTTPClient.

We can roll back to my code temporarily, but we want some of
the Solrj facet support for a new project.

wunder

Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Noble Paul നോബിള്‍  नोब्ळ्
you can set any connection parameters for the HttpClient and pass on
the instance to CommonsHttpSolrServer and that will be used for making
requests

make sure that you are not reusing instance of CommonsHttpSolrServer

On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
<[hidden email]> wrote:

> We just switched to Solrj from a home-grown client and we have a huge
> jump in the number of connections to the server, enough that our
> load balancer was rejecting connections in production tonight.
>
> Does that sound familiar? We're running 1.3.
>
> I set the timeouts and connection pools to the same values I'd
> used in my other code, also based on HTTPClient.
>
> We can roll back to my code temporarily, but we want some of
> the Solrj facet support for a new project.
>
> wunder
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Walter Underwood, Netflix
I did that. Unless CommonsHttpSolrServer changes some defaults, it should
be the same as my earlier code. I set max connections, max per host,
retries, and timeouts for connect and read.

We are seeing 20-40 open connections instead of a few.

wunder

On 1/26/09 10:28 PM, "Noble Paul നോബിള്‍  नोब्ळ्" <[hidden email]>
wrote:

> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <[hidden email]> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Noble Paul നോബിള്‍  नोब्ळ्
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्
correction

wrong: make sure that you are not reusing instance of CommonsHttpSolrServer
correct : make sure that you are reusing instance of CommonsHttpSolrServer


On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  नोब्ळ्
<[hidden email]> wrote:

> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <[hidden email]> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>
>
> --
> --Noble Paul
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Noble Paul നോബിള്‍  नोब्ळ्
In reply to this post by Noble Paul നോബിള്‍ नोब्ळ्
are you making requests in parallel ?
which ConnectionManager are you using for HttpClient?

On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  नोब्ळ्
<[hidden email]> wrote:

> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <[hidden email]> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>
>
> --
> --Noble Paul
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Walter Underwood, Netflix
Making requests in parallel, using the default connection manager,
which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
for all requests.

wunder

On 1/26/09 10:59 PM, "Noble Paul നോബിള്‍  नोब्ळ्" <[hidden email]>
wrote:

> are you making requests in parallel ?
> which ConnectionManager are you using for HttpClient?
>
> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  नोब्ळ्
> <[hidden email]> wrote:
>> you can set any connection parameters for the HttpClient and pass on
>> the instance to CommonsHttpSolrServer and that will be used for making
>> requests
>>
>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>
>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>> <[hidden email]> wrote:
>>> We just switched to Solrj from a home-grown client and we have a huge
>>> jump in the number of connections to the server, enough that our
>>> load balancer was rejecting connections in production tonight.
>>>
>>> Does that sound familiar? We're running 1.3.
>>>
>>> I set the timeouts and connection pools to the same values I'd
>>> used in my other code, also based on HTTPClient.
>>>
>>> We can roll back to my code temporarily, but we want some of
>>> the Solrj facet support for a new project.
>>>
>>> wunder
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Yonik Seeley-2-2
In reply to this post by Walter Underwood, Netflix
That's interesting.... SolrJ doesn't touch HTTPClient params if one is
provided in the constructor.

I guess I'd try to sniff the headers first and see if any difference
sticks out between the clients.
I normally just use netcat and pretend to be the solr server.

-Yonik


On Tue, Jan 27, 2009 at 12:29 AM, Walter Underwood
<[hidden email]> wrote:

> We just switched to Solrj from a home-grown client and we have a huge
> jump in the number of connections to the server, enough that our
> load balancer was rejecting connections in production tonight.
>
> Does that sound familiar? We're running 1.3.
>
> I set the timeouts and connection pools to the same values I'd
> used in my other code, also based on HTTPClient.
>
> We can roll back to my code temporarily, but we want some of
> the Solrj facet support for a new project.
>
> wunder
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Ryan McKinley
In reply to this post by Walter Underwood, Netflix
if you use this constructor:

   public CommonsHttpSolrServer(URL baseURL, HttpClient client)

then solrj never touches the HttpClient configuration.

I normally reuse a single CommonsHttpSolrServer as well.


On Jan 27, 2009, at 9:52 AM, Walter Underwood wrote:

> Making requests in parallel, using the default connection manager,
> which is multi-threaded, and we are reusing a single  
> CommonsHttpSolrServer
> for all requests.
>
> wunder
>
> On 1/26/09 10:59 PM, "Noble Paul നോബിള്‍  नो
> ब्ळ्" <[hidden email]>
> wrote:
>
>> are you making requests in parallel ?
>> which ConnectionManager are you using for HttpClient?
>>
>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  
>> नोब्ळ्
>> <[hidden email]> wrote:
>>> you can set any connection parameters for the HttpClient and pass on
>>> the instance to CommonsHttpSolrServer and that will be used for  
>>> making
>>> requests
>>>
>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>
>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>> <[hidden email]> wrote:
>>>> We just switched to Solrj from a home-grown client and we have a  
>>>> huge
>>>> jump in the number of connections to the server, enough that our
>>>> load balancer was rejecting connections in production tonight.
>>>>
>>>> Does that sound familiar? We're running 1.3.
>>>>
>>>> I set the timeouts and connection pools to the same values I'd
>>>> used in my other code, also based on HTTPClient.
>>>>
>>>> We can roll back to my code temporarily, but we want some of
>>>> the Solrj facet support for a new project.
>>>>
>>>> wunder
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Jon Baer
Could it be the framework you are using around it?  I know some IOC
containers will auto pool objects underneath as a service without you really
knowing it is being done or has to be explicitly turned off.  Just a
thought.  I use a single server for all requests behind a Hivemind setup ...
umm not by choice :-\

- Jon

On Tue, Jan 27, 2009 at 12:32 PM, Ryan McKinley <[hidden email]> wrote:

> if you use this constructor:
>
>  public CommonsHttpSolrServer(URL baseURL, HttpClient client)
>
> then solrj never touches the HttpClient configuration.
>
> I normally reuse a single CommonsHttpSolrServer as well.
>
>
>
> On Jan 27, 2009, at 9:52 AM, Walter Underwood wrote:
>
>  Making requests in parallel, using the default connection manager,
>> which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
>> for all requests.
>>
>> wunder
>>
>> On 1/26/09 10:59 PM, "Noble Paul നോബിള്‍  नोब्ळ्" <[hidden email]>
>> wrote:
>>
>>  are you making requests in parallel ?
>>> which ConnectionManager are you using for HttpClient?
>>>
>>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  नोब्ळ्
>>> <[hidden email]> wrote:
>>>
>>>> you can set any connection parameters for the HttpClient and pass on
>>>> the instance to CommonsHttpSolrServer and that will be used for making
>>>> requests
>>>>
>>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>>
>>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>>> <[hidden email]> wrote:
>>>>
>>>>> We just switched to Solrj from a home-grown client and we have a huge
>>>>> jump in the number of connections to the server, enough that our
>>>>> load balancer was rejecting connections in production tonight.
>>>>>
>>>>> Does that sound familiar? We're running 1.3.
>>>>>
>>>>> I set the timeouts and connection pools to the same values I'd
>>>>> used in my other code, also based on HTTPClient.
>>>>>
>>>>> We can roll back to my code temporarily, but we want some of
>>>>> the Solrj facet support for a new project.
>>>>>
>>>>> wunder
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection mismanagement in Solrj?

Noble Paul നോബിള്‍  नोब्ळ्
In reply to this post by Walter Underwood, Netflix
if you are making requests in parallel , then it is likely that you
see many connections open at a time. They will get cleaned up over
time . But if you wish to clean them up explicitly use
httpclient.getHttpConnectionManager()r#closeIdleConnections()

On Tue, Jan 27, 2009 at 8:22 PM, Walter Underwood
<[hidden email]> wrote:

> Making requests in parallel, using the default connection manager,
> which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
> for all requests.
>
> wunder
>
> On 1/26/09 10:59 PM, "Noble Paul നോബിള്‍  नोब्ळ्" <[hidden email]>
> wrote:
>
>> are you making requests in parallel ?
>> which ConnectionManager are you using for HttpClient?
>>
>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്‍  नोब्ळ्
>> <[hidden email]> wrote:
>>> you can set any connection parameters for the HttpClient and pass on
>>> the instance to CommonsHttpSolrServer and that will be used for making
>>> requests
>>>
>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>
>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>> <[hidden email]> wrote:
>>>> We just switched to Solrj from a home-grown client and we have a huge
>>>> jump in the number of connections to the server, enough that our
>>>> load balancer was rejecting connections in production tonight.
>>>>
>>>> Does that sound familiar? We're running 1.3.
>>>>
>>>> I set the timeouts and connection pools to the same values I'd
>>>> used in my other code, also based on HTTPClient.
>>>>
>>>> We can roll back to my code temporarily, but we want some of
>>>> the Solrj facet support for a new project.
>>>>
>>>> wunder
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>
>



--
--Noble Paul