Cannot set pollInterval in SolrCloud for PULL or TLOG replica

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot set pollInterval in SolrCloud for PULL or TLOG replica

Dmitry Vorotilin
Hi everyone,

We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only and
all replicas are PULL.
We have bulk updates so like once a day we reindex all cores (no soft
commits, only hard commit every 15s), do commit with openSearcher=true and
all our indexes become available for search.

The issue is that for PULL replication when leader reindexing starts it
downloads index every
hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
interval of 7000ms) then puts index into proper directory and just reopens
searcher so that we see no changes on leader because there was no commit
with openSearcher=true yet and that index keeps growing on PULL replicas.

Judging by this page
<https://lucene.apache.org/solr/guide/7_7/index-replication.html#index-replication-in-solr>
there's no setting for pollInterval or when to start replication on slaves
in SolrCloud and the info is rather confusing because in cloud we still use
the same handlers which we cannot configure.

We changed replication from NRT to PULL because we don't need realtime and
burn CPU with bulk updates on every machine, but this constantly catching
up index on slaves isn't any better...

Do you know any way to fix it?
Reply | Threaded
Open this post in threaded view
|

RE: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

Vadim Ivanov
Hi, Dmitri
There was discussion here a while ago...
http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
May be it helps you somehow.

--
Vadim


> -----Original Message-----
> From: Dmitry Vorotilin [mailto:[hidden email]]
> Sent: Tuesday, April 16, 2019 9:41 AM
> To: [hidden email]
> Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
>
> Hi everyone,
>
> We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only and
> all replicas are PULL.
> We have bulk updates so like once a day we reindex all cores (no soft
> commits, only hard commit every 15s), do commit with openSearcher=true
> and
> all our indexes become available for search.
>
> The issue is that for PULL replication when leader reindexing starts it
> downloads index every
> hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> interval of 7000ms) then puts index into proper directory and just reopens
> searcher so that we see no changes on leader because there was no commit
> with openSearcher=true yet and that index keeps growing on PULL replicas.
>
> Judging by this page
> <https://lucene.apache.org/solr/guide/7_7/index-replication.html#index-
> replication-in-solr>
> there's no setting for pollInterval or when to start replication on slaves
> in SolrCloud and the info is rather confusing because in cloud we still use
> the same handlers which we cannot configure.
>
> We changed replication from NRT to PULL because we don't need realtime
> and
> burn CPU with bulk updates on every machine, but this constantly catching
> up index on slaves isn't any better...
>
> Do you know any way to fix it?

Reply | Threaded
Open this post in threaded view
|

Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

Dmitry Vorotilin
Hi Vadim, thank you seems like we both had similar questions.
So I think that all confirms that it's not configurable for now. That's in
fact a pity because it only makes sense to use PULL/TLOG replicas in order
to save CPU and not reindex docs on every node but current situation with
reopening searcher every time ruins it all at least for bulk updates. The
only solution I see now is to use manual replication and trigger it on
every node after leader optimized index and this configuration was
available on master-salve legacy...

On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
[hidden email]> wrote:

> Hi, Dmitri
> There was discussion here a while ago...
>
> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
> May be it helps you somehow.
>
> --
> Vadim
>
>
> > -----Original Message-----
> > From: Dmitry Vorotilin [mailto:[hidden email]]
> > Sent: Tuesday, April 16, 2019 9:41 AM
> > To: [hidden email]
> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> >
> > Hi everyone,
> >
> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
> and
> > all replicas are PULL.
> > We have bulk updates so like once a day we reindex all cores (no soft
> > commits, only hard commit every 15s), do commit with openSearcher=true
> > and
> > all our indexes become available for search.
> >
> > The issue is that for PULL replication when leader reindexing starts it
> > downloads index every
> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> > interval of 7000ms) then puts index into proper directory and just
> reopens
> > searcher so that we see no changes on leader because there was no commit
> > with openSearcher=true yet and that index keeps growing on PULL replicas.
> >
> > Judging by this page
> > <https://lucene.apache.org/solr/guide/7_7/index-replication.html#index-
> > replication-in-solr>
> > there's no setting for pollInterval or when to start replication on
> slaves
> > in SolrCloud and the info is rather confusing because in cloud we still
> use
> > the same handlers which we cannot configure.
> >
> > We changed replication from NRT to PULL because we don't need realtime
> > and
> > burn CPU with bulk updates on every machine, but this constantly catching
> > up index on slaves isn't any better...
> >
> > Do you know any way to fix it?
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

Dmitry Vorotilin
It looks like `/solr/<core>/replication?command=disablepoll` doesn't work
in cloud mode so there's no way to change settings for interval as well as
to say replicas to stop polling.
My own conclusion: if you have bulk updates and commit with
openSearcher=true only at the end PULL/TLOG replicas isn't your choice, the
only option you have is NRT which burns CPU on all machines slowing down
all select queries.

On Wed, Apr 17, 2019 at 3:25 PM Dmitry Vorotilin <[hidden email]>
wrote:

> Hi Vadim, thank you seems like we both had similar questions.
> So I think that all confirms that it's not configurable for now. That's in
> fact a pity because it only makes sense to use PULL/TLOG replicas in order
> to save CPU and not reindex docs on every node but current situation with
> reopening searcher every time ruins it all at least for bulk updates. The
> only solution I see now is to use manual replication and trigger it on
> every node after leader optimized index and this configuration was
> available on master-salve legacy...
>
> On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
> [hidden email]> wrote:
>
>> Hi, Dmitri
>> There was discussion here a while ago...
>>
>> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-td4417253.html
>> May be it helps you somehow.
>>
>> --
>> Vadim
>>
>>
>> > -----Original Message-----
>> > From: Dmitry Vorotilin [mailto:[hidden email]]
>> > Sent: Tuesday, April 16, 2019 9:41 AM
>> > To: [hidden email]
>> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
>> >
>> > Hi everyone,
>> >
>> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
>> and
>> > all replicas are PULL.
>> > We have bulk updates so like once a day we reindex all cores (no soft
>> > commits, only hard commit every 15s), do commit with openSearcher=true
>> > and
>> > all our indexes become available for search.
>> >
>> > The issue is that for PULL replication when leader reindexing starts it
>> > downloads index every
>> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
>> > interval of 7000ms) then puts index into proper directory and just
>> reopens
>> > searcher so that we see no changes on leader because there was no commit
>> > with openSearcher=true yet and that index keeps growing on PULL
>> replicas.
>> >
>> > Judging by this page
>> > <https://lucene.apache.org/solr/guide/7_7/index-replication.html#index-
>> > replication-in-solr>
>> > there's no setting for pollInterval or when to start replication on
>> slaves
>> > in SolrCloud and the info is rather confusing because in cloud we still
>> use
>> > the same handlers which we cannot configure.
>> >
>> > We changed replication from NRT to PULL because we don't need realtime
>> > and
>> > burn CPU with bulk updates on every machine, but this constantly
>> catching
>> > up index on slaves isn't any better...
>> >
>> > Do you know any way to fix it?
>>
>>
Reply | Threaded
Open this post in threaded view
|

RE: Cannot set pollInterval in SolrCloud for PULL or TLOG replica

Vadim Ivanov
In my use case, I don't have bulk updates.
I just have continuously heavy updates on most cores

But maybe you can try to set updateHandler.autoCommit.maxTime through config api before and after bulk updates.
updateHandler.autoCommit.maxTime
https://lucene.apache.org/solr/guide/7_6/config-api.html

--
Vadim


> -----Original Message-----
> From: Dmitry Vorotilin [mailto:[hidden email]]
> Sent: Wednesday, April 17, 2019 7:22 PM
> To: [hidden email]
> Subject: Re: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
>
> It looks like `/solr/<core>/replication?command=disablepoll` doesn't work
> in cloud mode so there's no way to change settings for interval as well as
> to say replicas to stop polling.
> My own conclusion: if you have bulk updates and commit with
> openSearcher=true only at the end PULL/TLOG replicas isn't your choice, the
> only option you have is NRT which burns CPU on all machines slowing down
> all select queries.
>
> On Wed, Apr 17, 2019 at 3:25 PM Dmitry Vorotilin <[hidden email]>
> wrote:
>
> > Hi Vadim, thank you seems like we both had similar questions.
> > So I think that all confirms that it's not configurable for now. That's in
> > fact a pity because it only makes sense to use PULL/TLOG replicas in order
> > to save CPU and not reindex docs on every node but current situation with
> > reopening searcher every time ruins it all at least for bulk updates. The
> > only solution I see now is to use manual replication and trigger it on
> > every node after leader optimized index and this configuration was
> > available on master-salve legacy...
> >
> > On Tue, Apr 16, 2019 at 6:30 PM Vadim Ivanov <
> > [hidden email]> wrote:
> >
> >> Hi, Dmitri
> >> There was discussion here a while ago...
> >>
> >> http://lucene.472066.n3.nabble.com/Soft-commit-and-new-replica-types-
> td4417253.html
> >> May be it helps you somehow.
> >>
> >> --
> >> Vadim
> >>
> >>
> >> > -----Original Message-----
> >> > From: Dmitry Vorotilin [mailto:[hidden email]]
> >> > Sent: Tuesday, April 16, 2019 9:41 AM
> >> > To: [hidden email]
> >> > Subject: Cannot set pollInterval in SolrCloud for PULL or TLOG replica
> >> >
> >> > Hi everyone,
> >> >
> >> > We have SolrCloud cluster with 3 zk and 3 solr nodes. It's 1 shard only
> >> and
> >> > all replicas are PULL.
> >> > We have bulk updates so like once a day we reindex all cores (no soft
> >> > commits, only hard commit every 15s), do commit with
> openSearcher=true
> >> > and
> >> > all our indexes become available for search.
> >> >
> >> > The issue is that for PULL replication when leader reindexing starts it
> >> > downloads index every
> >> > hard commit / 2 seconds (o.a.s.h.ReplicationHandler Poll scheduled at an
> >> > interval of 7000ms) then puts index into proper directory and just
> >> reopens
> >> > searcher so that we see no changes on leader because there was no
> commit
> >> > with openSearcher=true yet and that index keeps growing on PULL
> >> replicas.
> >> >
> >> > Judging by this page
> >> > <https://lucene.apache.org/solr/guide/7_7/index-replication.html#index-
> >> > replication-in-solr>
> >> > there's no setting for pollInterval or when to start replication on
> >> slaves
> >> > in SolrCloud and the info is rather confusing because in cloud we still
> >> use
> >> > the same handlers which we cannot configure.
> >> >
> >> > We changed replication from NRT to PULL because we don't need
> realtime
> >> > and
> >> > burn CPU with bulk updates on every machine, but this constantly
> >> catching
> >> > up index on slaves isn't any better...
> >> >
> >> > Do you know any way to fix it?
> >>
> >>