Solr Setup using NRT and PULL replicas

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr Setup using NRT and PULL replicas

Daniel Carrasco
Hello,

We've a cluster consisting in 7 to 10 NRT nodes serving data to a webpage
(products, categories,...), but every time a leader node fails importing
data (connection lost, broken pipe...), the entire cluster goes to
recovering mode and then is not working for about 15-30 minutes. That's a
lot for the webpage so we're trying to reduce that problem to minimal and
we're thinking about new PULL nodes.

We've for now Solr 7.2.1, but we're planning to migrate to Solr 7.5, and
I've read on Solr guide that recommended setups are:

   - All NRT
   - All TLOG
   - Some TLOG with PULL replicas

We're not fully convinced about TLOG replicas because we've read something
about index problems if a node goes down suddenly, or using kill -9 (just
what the init.d script does if takes long to stop/restart), and is the
leader, or about the increase in recovering times respecting to NRT, so
we're thinking about to keep some NRT instead TLOG.

To have two NRT nodes and the rest TLOG is a good setup, or better to think
in TOG nodes?

Thanks!

--
_________________________________________

      Daniel Carrasco Marín
      Ingeniería para la Innovación i2TIC, S.L.
      Tlf:  +34 911 12 32 84 Ext: 223
      www.i2tic.com
_________________________________________
Reply | Threaded
Open this post in threaded view
|

Re: Solr Setup using NRT and PULL replicas

Edward Ribeiro
To mix NRT and TLOG/PULL replicas is not recommended. It is all NRT nodes
or TLOG nodes mixed (or not) with PULL replicas. As you know, all PULL
replicas is not possible.

According to the talk below, one of the reasons is that if you have NRT
mixed with TLOG and PULL replicas then a leadership change could make all
tlog/pull replicas to download the entire index segments from the new
leader:

https://youtu.be/XIb8X3MwVKc at minute 23:50

Also see this Activate 2018 talk https://youtu.be/dkWy2ykzAv0 at minute
14:45.

And this on is nice too: https://youtu.be/XqfTjd9KDWU

Best,
Edward


Em sex, 30 de nov de 2018 08:56, Daniel Carrasco <[hidden email]
escreveu:

> Hello,
>
> We've a cluster consisting in 7 to 10 NRT nodes serving data to a webpage
> (products, categories,...), but every time a leader node fails importing
> data (connection lost, broken pipe...), the entire cluster goes to
> recovering mode and then is not working for about 15-30 minutes. That's a
> lot for the webpage so we're trying to reduce that problem to minimal and
> we're thinking about new PULL nodes.
>
> We've for now Solr 7.2.1, but we're planning to migrate to Solr 7.5, and
> I've read on Solr guide that recommended setups are:
>
>    - All NRT
>    - All TLOG
>    - Some TLOG with PULL replicas
>
> We're not fully convinced about TLOG replicas because we've read something
> about index problems if a node goes down suddenly, or using kill -9 (just
> what the init.d script does if takes long to stop/restart), and is the
> leader, or about the increase in recovering times respecting to NRT, so
> we're thinking about to keep some NRT instead TLOG.
>
> To have two NRT nodes and the rest TLOG is a good setup, or better to think
> in TOG nodes?
>
> Thanks!
>
> --
> _________________________________________
>
>       Daniel Carrasco Marín
>       Ingeniería para la Innovación i2TIC, S.L.
>       Tlf:  +34 911 12 32 84 Ext: 223
>       www.i2tic.com
> _________________________________________
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Setup using NRT and PULL replicas

Daniel Carrasco
Thanks for all the info,

We're trying some things to find the way to make it more stable, becuase
we're getting into troubles with the cluster. This weekend we've got three
downtimes in a few hours because some nodes of the cluster loose their
connection a bit during an import (we don't know why), and then the entire
cluster goes down (all replicas are NRT).

Looks like was related with hugepages of that nodes and for now we're
investigateing why.

Greetings!


El dom., 2 dic. 2018 a las 18:48, Edward Ribeiro (<[hidden email]>)
escribió:

> To mix NRT and TLOG/PULL replicas is not recommended. It is all NRT nodes
> or TLOG nodes mixed (or not) with PULL replicas. As you know, all PULL
> replicas is not possible.
>
> According to the talk below, one of the reasons is that if you have NRT
> mixed with TLOG and PULL replicas then a leadership change could make all
> tlog/pull replicas to download the entire index segments from the new
> leader:
>
> https://youtu.be/XIb8X3MwVKc at minute 23:50
>
> Also see this Activate 2018 talk https://youtu.be/dkWy2ykzAv0 at minute
> 14:45.
>
> And this on is nice too: https://youtu.be/XqfTjd9KDWU
>
> Best,
> Edward
>
>
> Em sex, 30 de nov de 2018 08:56, Daniel Carrasco <[hidden email]
> escreveu:
>
> > Hello,
> >
> > We've a cluster consisting in 7 to 10 NRT nodes serving data to a webpage
> > (products, categories,...), but every time a leader node fails importing
> > data (connection lost, broken pipe...), the entire cluster goes to
> > recovering mode and then is not working for about 15-30 minutes. That's a
> > lot for the webpage so we're trying to reduce that problem to minimal and
> > we're thinking about new PULL nodes.
> >
> > We've for now Solr 7.2.1, but we're planning to migrate to Solr 7.5, and
> > I've read on Solr guide that recommended setups are:
> >
> >    - All NRT
> >    - All TLOG
> >    - Some TLOG with PULL replicas
> >
> > We're not fully convinced about TLOG replicas because we've read
> something
> > about index problems if a node goes down suddenly, or using kill -9 (just
> > what the init.d script does if takes long to stop/restart), and is the
> > leader, or about the increase in recovering times respecting to NRT, so
> > we're thinking about to keep some NRT instead TLOG.
> >
> > To have two NRT nodes and the rest TLOG is a good setup, or better to
> think
> > in TOG nodes?
> >
> > Thanks!
> >
> > --
> > _________________________________________
> >
> >       Daniel Carrasco Marín
> >       Ingeniería para la Innovación i2TIC, S.L.
> >       Tlf:  +34 911 12 32 84 Ext: 223
> >       www.i2tic.com
> > _________________________________________
> >
>


--
_________________________________________

      Daniel Carrasco Marín
      Ingeniería para la Innovación i2TIC, S.L.
      Tlf:  +34 911 12 32 84 Ext: 223
      www.i2tic.com
_________________________________________