httpSolrServer and exyternal load balancer

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

httpSolrServer and exyternal load balancer

lee carroll
Hi

We have the following solr http server

<bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
id="solrserver" >
<constructor-arg value="urlToSlaveLoadBalancer" />
<property name="soTimeout" value="1000" />
<property name="connectionTimeout" value="1000" />
<property name="defaultMaxConnectionsPerHost" value="5" />
<property name="maxTotalConnections" value="20" />
<property name="allowCompression" value="true" />
</bean>

The issue we face is the f5 balancer is returning a cookie which the client
is hanging onto. resulting in the same slave being hit for all requests.

one obvious solution is to config the load balancer to be non sticky
however politically a "non-standard" load balancer is timescale suicide.
(It is an out sourced corporate thing)

I'm not keen to use the LB http solr server as i don't want this to be a
concern of the software and have a list of servers etc. (although as a stop
gap may well have to)

My question is can I configure the solr server to ignore client state ? We
are on solr 3.4

Thanks in advance lee c
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

Erick Erickson
What client state? Solr servers are stateless, they don't
keep any information specific to particular clients so this
doesn't seem to be a problem.

What Solr _does_ do is cache things like fq clauses, but
these are not user-specific. Which actually argues for going
to the same slave on the theory that requests from a
user are more likely to have the same fq clauses. Consider
faceting on shoes. The user clicks "mens" and you add an
fq like &fq=gender:mens. Then the user wants dress shoes
so you submit another query &fq=gender:mens&fq=style:dress.
The first fq clause has already been calculated and cached so
doesn't have to be re-calculated for the second query...

But the stickiness is usually the way Solr is used, so this seems
like a red herring.

FWIW,
Erick

On Thu, Sep 27, 2012 at 7:06 AM, Lee Carroll
<[hidden email]> wrote:

> Hi
>
> We have the following solr http server
>
> <bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
> id="solrserver" >
> <constructor-arg value="urlToSlaveLoadBalancer" />
> <property name="soTimeout" value="1000" />
> <property name="connectionTimeout" value="1000" />
> <property name="defaultMaxConnectionsPerHost" value="5" />
> <property name="maxTotalConnections" value="20" />
> <property name="allowCompression" value="true" />
> </bean>
>
> The issue we face is the f5 balancer is returning a cookie which the client
> is hanging onto. resulting in the same slave being hit for all requests.
>
> one obvious solution is to config the load balancer to be non sticky
> however politically a "non-standard" load balancer is timescale suicide.
> (It is an out sourced corporate thing)
>
> I'm not keen to use the LB http solr server as i don't want this to be a
> concern of the software and have a list of servers etc. (although as a stop
> gap may well have to)
>
> My question is can I configure the solr server to ignore client state ? We
> are on solr 3.4
>
> Thanks in advance lee c
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

lee carroll
Hi Erick,

the load balancer in front of the solr servers is dropping the cookie not
the solr server themselves.

are you saying the clients http connection manager builds will ignore this
state ? it looks like they do not. It looks like the
client is passing the cookie back to the load balancer

I want to configure the clients not to pass cookies basically.

Does that make sense ?



On 27 September 2012 12:54, Erick Erickson <[hidden email]> wrote:

> What client state? Solr servers are stateless, they don't
> keep any information specific to particular clients so this
> doesn't seem to be a problem.
>
> What Solr _does_ do is cache things like fq clauses, but
> these are not user-specific. Which actually argues for going
> to the same slave on the theory that requests from a
> user are more likely to have the same fq clauses. Consider
> faceting on shoes. The user clicks "mens" and you add an
> fq like &fq=gender:mens. Then the user wants dress shoes
> so you submit another query &fq=gender:mens&fq=style:dress.
> The first fq clause has already been calculated and cached so
> doesn't have to be re-calculated for the second query...
>
> But the stickiness is usually the way Solr is used, so this seems
> like a red herring.
>
> FWIW,
> Erick
>
> On Thu, Sep 27, 2012 at 7:06 AM, Lee Carroll
> <[hidden email]> wrote:
> > Hi
> >
> > We have the following solr http server
> >
> > <bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
> > id="solrserver" >
> > <constructor-arg value="urlToSlaveLoadBalancer" />
> > <property name="soTimeout" value="1000" />
> > <property name="connectionTimeout" value="1000" />
> > <property name="defaultMaxConnectionsPerHost" value="5" />
> > <property name="maxTotalConnections" value="20" />
> > <property name="allowCompression" value="true" />
> > </bean>
> >
> > The issue we face is the f5 balancer is returning a cookie which the
> client
> > is hanging onto. resulting in the same slave being hit for all requests.
> >
> > one obvious solution is to config the load balancer to be non sticky
> > however politically a "non-standard" load balancer is timescale suicide.
> > (It is an out sourced corporate thing)
> >
> > I'm not keen to use the LB http solr server as i don't want this to be a
> > concern of the software and have a list of servers etc. (although as a
> stop
> > gap may well have to)
> >
> > My question is can I configure the solr server to ignore client state ?
> We
> > are on solr 3.4
> >
> > Thanks in advance lee c
>
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

Erick Erickson
But again, why do you want to do this? I really think you don't.

I'm assuming that when you say this:
"...resulting in the same slave being hit for all requests."

you mean "all requests _from the same client_". If that's
not what's happening, then disregard my maundering
because when it comes to setting up LBs, I'm clueless. But
I can say that many installations have LBs set up with
sticky sessions on a per-client basis......

Consider another scenario; replication. If you have 2 slaves,
each with a polling interval of 5 minutes note that they are
not coordinated. So slave 1 can poll at 14:00:00. Slave 2
at 14:01:00. Say there's been a commit at 14:00:30. Requests
to slave 2 will have a different view of the index than slave 1,
so if your user resends the exact same request, they may
see different results. I could submit the request 5 times in a
row and the results would not only be different each time, they
would flip-flop back and forth.

I wouldn't do this unless and until you have a demonstrated need.

Best
Erick

On Thu, Sep 27, 2012 at 8:07 AM, Lee Carroll
<[hidden email]> wrote:

> Hi Erick,
>
> the load balancer in front of the solr servers is dropping the cookie not
> the solr server themselves.
>
> are you saying the clients http connection manager builds will ignore this
> state ? it looks like they do not. It looks like the
> client is passing the cookie back to the load balancer
>
> I want to configure the clients not to pass cookies basically.
>
> Does that make sense ?
>
>
>
> On 27 September 2012 12:54, Erick Erickson <[hidden email]> wrote:
>
>> What client state? Solr servers are stateless, they don't
>> keep any information specific to particular clients so this
>> doesn't seem to be a problem.
>>
>> What Solr _does_ do is cache things like fq clauses, but
>> these are not user-specific. Which actually argues for going
>> to the same slave on the theory that requests from a
>> user are more likely to have the same fq clauses. Consider
>> faceting on shoes. The user clicks "mens" and you add an
>> fq like &fq=gender:mens. Then the user wants dress shoes
>> so you submit another query &fq=gender:mens&fq=style:dress.
>> The first fq clause has already been calculated and cached so
>> doesn't have to be re-calculated for the second query...
>>
>> But the stickiness is usually the way Solr is used, so this seems
>> like a red herring.
>>
>> FWIW,
>> Erick
>>
>> On Thu, Sep 27, 2012 at 7:06 AM, Lee Carroll
>> <[hidden email]> wrote:
>> > Hi
>> >
>> > We have the following solr http server
>> >
>> > <bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
>> > id="solrserver" >
>> > <constructor-arg value="urlToSlaveLoadBalancer" />
>> > <property name="soTimeout" value="1000" />
>> > <property name="connectionTimeout" value="1000" />
>> > <property name="defaultMaxConnectionsPerHost" value="5" />
>> > <property name="maxTotalConnections" value="20" />
>> > <property name="allowCompression" value="true" />
>> > </bean>
>> >
>> > The issue we face is the f5 balancer is returning a cookie which the
>> client
>> > is hanging onto. resulting in the same slave being hit for all requests.
>> >
>> > one obvious solution is to config the load balancer to be non sticky
>> > however politically a "non-standard" load balancer is timescale suicide.
>> > (It is an out sourced corporate thing)
>> >
>> > I'm not keen to use the LB http solr server as i don't want this to be a
>> > concern of the software and have a list of servers etc. (although as a
>> stop
>> > gap may well have to)
>> >
>> > My question is can I configure the solr server to ignore client state ?
>> We
>> > are on solr 3.4
>> >
>> > Thanks in advance lee c
>>
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

lee carroll
Hi Erick

Our application has one  CommonsHttpSolrServer for each solr core used by
our web app. Whilst we have many web app clients
solr only has 1 client, our application. Does that make sense. This is why
sticky load balancing is an issue for us.

I cannot see any where the state is being handled in the
CommonsHttpSolrServer  impl ? It looks like the state is not being passed
by the client or am i missing something?

Cheers Lee c

On 27 September 2012 14:00, Erick Erickson <[hidden email]> wrote:

> But again, why do you want to do this? I really think you don't.
>
> I'm assuming that when you say this:
> "...resulting in the same slave being hit for all requests."
>
> you mean "all requests _from the same client_". If that's
> not what's happening, then disregard my maundering
> because when it comes to setting up LBs, I'm clueless. But
> I can say that many installations have LBs set up with
> sticky sessions on a per-client basis......
>
> Consider another scenario; replication. If you have 2 slaves,
> each with a polling interval of 5 minutes note that they are
> not coordinated. So slave 1 can poll at 14:00:00. Slave 2
> at 14:01:00. Say there's been a commit at 14:00:30. Requests
> to slave 2 will have a different view of the index than slave 1,
> so if your user resends the exact same request, they may
> see different results. I could submit the request 5 times in a
> row and the results would not only be different each time, they
> would flip-flop back and forth.
>
> I wouldn't do this unless and until you have a demonstrated need.
>
> Best
> Erick
>
> On Thu, Sep 27, 2012 at 8:07 AM, Lee Carroll
> <[hidden email]> wrote:
> > Hi Erick,
> >
> > the load balancer in front of the solr servers is dropping the cookie not
> > the solr server themselves.
> >
> > are you saying the clients http connection manager builds will ignore
> this
> > state ? it looks like they do not. It looks like the
> > client is passing the cookie back to the load balancer
> >
> > I want to configure the clients not to pass cookies basically.
> >
> > Does that make sense ?
> >
> >
> >
> > On 27 September 2012 12:54, Erick Erickson <[hidden email]>
> wrote:
> >
> >> What client state? Solr servers are stateless, they don't
> >> keep any information specific to particular clients so this
> >> doesn't seem to be a problem.
> >>
> >> What Solr _does_ do is cache things like fq clauses, but
> >> these are not user-specific. Which actually argues for going
> >> to the same slave on the theory that requests from a
> >> user are more likely to have the same fq clauses. Consider
> >> faceting on shoes. The user clicks "mens" and you add an
> >> fq like &fq=gender:mens. Then the user wants dress shoes
> >> so you submit another query &fq=gender:mens&fq=style:dress.
> >> The first fq clause has already been calculated and cached so
> >> doesn't have to be re-calculated for the second query...
> >>
> >> But the stickiness is usually the way Solr is used, so this seems
> >> like a red herring.
> >>
> >> FWIW,
> >> Erick
> >>
> >> On Thu, Sep 27, 2012 at 7:06 AM, Lee Carroll
> >> <[hidden email]> wrote:
> >> > Hi
> >> >
> >> > We have the following solr http server
> >> >
> >> > <bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
> >> > id="solrserver" >
> >> > <constructor-arg value="urlToSlaveLoadBalancer" />
> >> > <property name="soTimeout" value="1000" />
> >> > <property name="connectionTimeout" value="1000" />
> >> > <property name="defaultMaxConnectionsPerHost" value="5" />
> >> > <property name="maxTotalConnections" value="20" />
> >> > <property name="allowCompression" value="true" />
> >> > </bean>
> >> >
> >> > The issue we face is the f5 balancer is returning a cookie which the
> >> client
> >> > is hanging onto. resulting in the same slave being hit for all
> requests.
> >> >
> >> > one obvious solution is to config the load balancer to be non sticky
> >> > however politically a "non-standard" load balancer is timescale
> suicide.
> >> > (It is an out sourced corporate thing)
> >> >
> >> > I'm not keen to use the LB http solr server as i don't want this to
> be a
> >> > concern of the software and have a list of servers etc. (although as a
> >> stop
> >> > gap may well have to)
> >> >
> >> > My question is can I configure the solr server to ignore client state
> ?
> >> We
> >> > are on solr 3.4
> >> >
> >> > Thanks in advance lee c
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

Erick Erickson
Ahh, I finally think I get it. I was missing the connection being
the CommonsHttpSolrServer. That's the thing that's locking
on to a particular slave....

I'm afraid I'm not up enough on the internals here to be much
help, so I'll have to defer....

Erick.

On Thu, Sep 27, 2012 at 10:20 AM, Lee Carroll
<[hidden email]> wrote:

> Hi Erick
>
> Our application has one  CommonsHttpSolrServer for each solr core used by
> our web app. Whilst we have many web app clients
> solr only has 1 client, our application. Does that make sense. This is why
> sticky load balancing is an issue for us.
>
> I cannot see any where the state is being handled in the
> CommonsHttpSolrServer  impl ? It looks like the state is not being passed
> by the client or am i missing something?
>
> Cheers Lee c
>
> On 27 September 2012 14:00, Erick Erickson <[hidden email]> wrote:
>
>> But again, why do you want to do this? I really think you don't.
>>
>> I'm assuming that when you say this:
>> "...resulting in the same slave being hit for all requests."
>>
>> you mean "all requests _from the same client_". If that's
>> not what's happening, then disregard my maundering
>> because when it comes to setting up LBs, I'm clueless. But
>> I can say that many installations have LBs set up with
>> sticky sessions on a per-client basis......
>>
>> Consider another scenario; replication. If you have 2 slaves,
>> each with a polling interval of 5 minutes note that they are
>> not coordinated. So slave 1 can poll at 14:00:00. Slave 2
>> at 14:01:00. Say there's been a commit at 14:00:30. Requests
>> to slave 2 will have a different view of the index than slave 1,
>> so if your user resends the exact same request, they may
>> see different results. I could submit the request 5 times in a
>> row and the results would not only be different each time, they
>> would flip-flop back and forth.
>>
>> I wouldn't do this unless and until you have a demonstrated need.
>>
>> Best
>> Erick
>>
>> On Thu, Sep 27, 2012 at 8:07 AM, Lee Carroll
>> <[hidden email]> wrote:
>> > Hi Erick,
>> >
>> > the load balancer in front of the solr servers is dropping the cookie not
>> > the solr server themselves.
>> >
>> > are you saying the clients http connection manager builds will ignore
>> this
>> > state ? it looks like they do not. It looks like the
>> > client is passing the cookie back to the load balancer
>> >
>> > I want to configure the clients not to pass cookies basically.
>> >
>> > Does that make sense ?
>> >
>> >
>> >
>> > On 27 September 2012 12:54, Erick Erickson <[hidden email]>
>> wrote:
>> >
>> >> What client state? Solr servers are stateless, they don't
>> >> keep any information specific to particular clients so this
>> >> doesn't seem to be a problem.
>> >>
>> >> What Solr _does_ do is cache things like fq clauses, but
>> >> these are not user-specific. Which actually argues for going
>> >> to the same slave on the theory that requests from a
>> >> user are more likely to have the same fq clauses. Consider
>> >> faceting on shoes. The user clicks "mens" and you add an
>> >> fq like &fq=gender:mens. Then the user wants dress shoes
>> >> so you submit another query &fq=gender:mens&fq=style:dress.
>> >> The first fq clause has already been calculated and cached so
>> >> doesn't have to be re-calculated for the second query...
>> >>
>> >> But the stickiness is usually the way Solr is used, so this seems
>> >> like a red herring.
>> >>
>> >> FWIW,
>> >> Erick
>> >>
>> >> On Thu, Sep 27, 2012 at 7:06 AM, Lee Carroll
>> >> <[hidden email]> wrote:
>> >> > Hi
>> >> >
>> >> > We have the following solr http server
>> >> >
>> >> > <bean class="org.apache.solr.client.solrj.impl.CommonsHttpSolrServer"
>> >> > id="solrserver" >
>> >> > <constructor-arg value="urlToSlaveLoadBalancer" />
>> >> > <property name="soTimeout" value="1000" />
>> >> > <property name="connectionTimeout" value="1000" />
>> >> > <property name="defaultMaxConnectionsPerHost" value="5" />
>> >> > <property name="maxTotalConnections" value="20" />
>> >> > <property name="allowCompression" value="true" />
>> >> > </bean>
>> >> >
>> >> > The issue we face is the f5 balancer is returning a cookie which the
>> >> client
>> >> > is hanging onto. resulting in the same slave being hit for all
>> requests.
>> >> >
>> >> > one obvious solution is to config the load balancer to be non sticky
>> >> > however politically a "non-standard" load balancer is timescale
>> suicide.
>> >> > (It is an out sourced corporate thing)
>> >> >
>> >> > I'm not keen to use the LB http solr server as i don't want this to
>> be a
>> >> > concern of the software and have a list of servers etc. (although as a
>> >> stop
>> >> > gap may well have to)
>> >> >
>> >> > My question is can I configure the solr server to ignore client state
>> ?
>> >> We
>> >> > are on solr 3.4
>> >> >
>> >> > Thanks in advance lee c
>> >>
>>
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

Chris Hostetter-3
In reply to this post by lee carroll

: The issue we face is the f5 balancer is returning a cookie which the client
: is hanging onto. resulting in the same slave being hit for all requests.
        ...
: My question is can I configure the solr server to ignore client state ? We
: are on solr 3.4

I'm not an expert on HTTP session affinity as implemented by various load
blanacers, but i can say with a high degree of confidence:

1) SolrJ doesn't care about cookies

2) if any part of the codepath you are using cares about cookies sent back
from your load-balancer, it would be the HttpClient objects used by
CommonsHttpSolrServer.

3) you have total control over the HttpClient objects used by
CommonsHttpSolrServer via an optional constructor arg.

4) https://hc.apache.org/httpcomponents-client-ga/tutorial/html/statemgmt.html#d5e799

-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: httpSolrServer and exyternal load balancer

lee carroll
Cheers, saved the day

Lee C

On 28 September 2012 23:27, Chris Hostetter <[hidden email]>wrote:

>
> : The issue we face is the f5 balancer is returning a cookie which the
> client
> : is hanging onto. resulting in the same slave being hit for all requests.
>         ...
> : My question is can I configure the solr server to ignore client state ?
> We
> : are on solr 3.4
>
> I'm not an expert on HTTP session affinity as implemented by various load
> blanacers, but i can say with a high degree of confidence:
>
> 1) SolrJ doesn't care about cookies
>
> 2) if any part of the codepath you are using cares about cookies sent back
> from your load-balancer, it would be the HttpClient objects used by
> CommonsHttpSolrServer.
>
> 3) you have total control over the HttpClient objects used by
> CommonsHttpSolrServer via an optional constructor arg.
>
> 4)
> https://hc.apache.org/httpcomponents-client-ga/tutorial/html/statemgmt.html#d5e799
>
> -Hoss
>