Tlogs are not purged with CDCR started and buffer disabled

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Tlogs are not purged with CDCR started and buffer disabled

Louis
This post was updated on .
Using Solr 7.7.3-snapshot, 1 shard + 3 replicas on source and target cluster

When unidirectional CDCR enabled and buffer disabled through CDCR API, my understanding is,
when data is successfully forwarded to target and committed, tlogs on both
source and target should be purged.

However, the source node doesn't purge tlogs no matter how I tried(manually
committed as well) while tlogs on target are purged. (if I turn off CDCR and
import data, tlogs is nicely cleaned)
 
So I tested with some queries.. and there are no errors. queue size is 0,
and the last processed version is not -1 either.

I also double-checked CDCR buffer disabled on both source and target, and CDCR is enabled on source and disabled on target and data replication to target is also working fine(except the fact that
tlogs on source node keep growing). Restarting source nodes doesn't work either.

What am I missing and what else should I check next?

curl -k https://localhost:8983/solr/tbh_manuals_uni/cdcr?action=STATUS (source cluster)
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "status":[
    "process","started",
    "buffer","disabled"]}

$ curl -k
https://localhost:8983/solr/tbh_manuals_uni_shard1_replica_n2/cdcr?action=QUEUES (source cluster)
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "queues":[
    "host1:8981,host2:8981,host3:8981/solr",[
      "tbh_manuals_uni",[
        "queueSize",0,
        "lastTimestamp","2020-01-08T23:16:26.899Z"]]],
  "tlogTotalSize":503573,
  "tlogTotalCount":278,
  "updateLogSynchronizer":"stopped"}

$ curl -k
https://localhost:8983/solr/tbh_manuals_uni_shard1_replica_n2/cdcr?action=ERRORS (source/target cluster)
{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "errors":[
    "host1:8981,host2:8981,host3:8981/solr",[
      "tbh_manuals_uni",[
        "consecutiveErrors",0,
        "bad_request",0,
        "internal",0,
        "last",[]]]]}

$ curl -k
https://localhost:8983/solr/tbh_manuals_uni_shard1_replica_n2/cdcr?action=LASTPROCESSEDVERSION (source cluster)
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "lastProcessedVersion":1655203836093005824}



I actually see some errors on zookeeper.out file only in target's leader
node as follows. However honestly, I don't know what they mean..



2020-01-08 15:11:42,740 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590008 type:create cxid:0xd2
zxid:0x3000000b4 txntype:-1 reqpath:n/a Error Path:/solr/collections
Error:KeeperErrorCode = NodeExists for /solr/collections
2020-01-08 15:11:42,742 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590008 type:create cxid:0xd3
zxid:0x3000000b5 txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni Error:KeeperErrorCode = NodeExists
for /solr/collections/tbh_manuals_uni
2020-01-08 15:11:42,744 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590008 type:create cxid:0xd4
zxid:0x3000000b6 txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni/terms Error:KeeperErrorCode =
NodeExists for /solr/collections/tbh_manuals_uni/terms
2020-01-08 15:11:42,745 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590008 type:create cxid:0xd5
zxid:0x3000000b7 txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni/terms/shard1 Error:KeeperErrorCode =
NodeExists for /solr/collections/tbh_manuals_uni/terms/shard1
2020-01-08 15:11:42,821 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:create cxid:0x23c
zxid:0x3000000ba txntype:-1 reqpath:n/a Error Path:/solr/collections
Error:KeeperErrorCode = NodeExists for /solr/collections
2020-01-08 15:11:42,823 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:create cxid:0x23d
zxid:0x3000000bb txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni Error:KeeperErrorCode = NodeExists
for /solr/collections/tbh_manuals_uni
2020-01-08 15:11:42,825 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:create cxid:0x23e
zxid:0x3000000bc txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni/terms Error:KeeperErrorCode =
NodeExists for /solr/collections/tbh_manuals_uni/terms
2020-01-08 15:11:42,827 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:create cxid:0x23f
zxid:0x3000000bd txntype:-1 reqpath:n/a Error
Path:/solr/collections/tbh_manuals_uni/terms/shard1 Error:KeeperErrorCode =
NodeExists for /solr/collections/tbh_manuals_uni/terms/shard1
2020-01-08 15:11:45,185 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:setData cxid:0x274
zxid:0x3000000ce txntype:-1 reqpath:n/a Error
Path:/solr/overseer/collection-queue-work/qnr-0000000016
Error:KeeperErrorCode = NoNode for
/solr/overseer/collection-queue-work/qnr-0000000016
2020-01-08 15:11:45,187 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when
processing sessionid:0x301d2ecaf590005 type:delete cxid:0x275
zxid:0x3000000cf txntype:-1 reqpath:n/a Error
Path:/solr/overseer/collection-queue-work/qn-0000000016
Error:KeeperErrorCode = NoNode for
/solr/overseer/collection-queue-work/qn-0000000016




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Tlogs are not purged when CDCR is enabled

Louis
This post was updated on .
Another finding is, no matter how I tried to disable buffer with the
following setup on both nodes, it is always started with enabled first time.

<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
 
  <lst name="buffer">
    <str name="defaultState">disabled</str>
  </lst>
</requestHandler>

Once I call CDCR API to disable buffer, it turns to be disabled. I wonder if
https://issues.apache.org/jira/browse/SOLR-11652 is related to this issue..

How can I make the default state of buffer disabled if this setup doesn't
work?



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html