Use of TLog

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Use of TLog

Sripra deep
Hi Guys,

 I observed a scenario with the tlog creation and usage and couldn't find
any usage for the tlog.

Solr version: 7.1.0
Number of shards = 3
Number of replica = 1
I indexed the about 10k docs into the collection.

 Scenario 1:
  Using add replica collection API, I created one more replica (tried with
both nrt and tlog) neither of the replicas doesn't pull the tlog files.
Only the index files are pulled from master.
  * If the tlog is not present in a replica then during ungraceful shutdown
of the solr server how the replicas will regain the index without tlog
files.
  * To verify the above scenario, I killed the newly added replica server
with kill -9 <pid> command and started back
  also stopped the leader node.

 Questions:
  1) TLog files are not used even in the case of ungraceful shutdown, where
else it will be used?
  2) Tlog files doesn't get copied to the newly added replica so adding a
new replica to the already created collection with data/index is not
advisable?
  3) Is there a way to make the newly added slave node to replicate the
tlog file as it does for the data/index files from leader?
  4) Is it possible to use the Tlog files /index files from an existing
solr server to spin up a new solr cluster?


It would be much helpful for me to understand the core working of Solr
server.

Thanks,
Sripradeep P
Reply | Threaded
Open this post in threaded view
|

Re: Use of TLog

Emir Arnautović
Hi Sripradeep,
Simplified: TLog files are used to replay index updates from the last successful hard commit in case of some Solr crashes. It is used on the next Solr startup. It does not contain all updates, otherwise, it would duplicate the index size.
If you start from these premises, you will understand why it is not copied when adding replicas and why it is not needed and why you cannot use TLog to spin up a new cluster.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Nov 2019, at 06:35, Sripra deep <[hidden email]> wrote:
>
> Hi Guys,
>
> I observed a scenario with the tlog creation and usage and couldn't find
> any usage for the tlog.
>
> Solr version: 7.1.0
> Number of shards = 3
> Number of replica = 1
> I indexed the about 10k docs into the collection.
>
> Scenario 1:
>  Using add replica collection API, I created one more replica (tried with
> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
> Only the index files are pulled from master.
>  * If the tlog is not present in a replica then during ungraceful shutdown
> of the solr server how the replicas will regain the index without tlog
> files.
>  * To verify the above scenario, I killed the newly added replica server
> with kill -9 <pid> command and started back
>  also stopped the leader node.
>
> Questions:
>  1) TLog files are not used even in the case of ungraceful shutdown, where
> else it will be used?
>  2) Tlog files doesn't get copied to the newly added replica so adding a
> new replica to the already created collection with data/index is not
> advisable?
>  3) Is there a way to make the newly added slave node to replicate the
> tlog file as it does for the data/index files from leader?
>  4) Is it possible to use the Tlog files /index files from an existing
> solr server to spin up a new solr cluster?
>
>
> It would be much helpful for me to understand the core working of Solr
> server.
>
> Thanks,
> Sripradeep P

Reply | Threaded
Open this post in threaded view
|

Re: Use of TLog

Sripra deep
Hi Emir,

  Thank you so much. Now I got it cleared with the TLOG purpose.
   I am trying to copy an index of one solr cluster to use it to build
other solr cluster. I am able to make that work but Is this design okay? or
any other approach I can try to get a new cluster spin up with the same
data as in the old one.

Thanks,
Sripradeep P


On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
[hidden email]> wrote:

> Hi Sripradeep,
> Simplified: TLog files are used to replay index updates from the last
> successful hard commit in case of some Solr crashes. It is used on the next
> Solr startup. It does not contain all updates, otherwise, it would
> duplicate the index size.
> If you start from these premises, you will understand why it is not copied
> when adding replicas and why it is not needed and why you cannot use TLog
> to spin up a new cluster.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 18 Nov 2019, at 06:35, Sripra deep <[hidden email]>
> wrote:
> >
> > Hi Guys,
> >
> > I observed a scenario with the tlog creation and usage and couldn't find
> > any usage for the tlog.
> >
> > Solr version: 7.1.0
> > Number of shards = 3
> > Number of replica = 1
> > I indexed the about 10k docs into the collection.
> >
> > Scenario 1:
> >  Using add replica collection API, I created one more replica (tried with
> > both nrt and tlog) neither of the replicas doesn't pull the tlog files.
> > Only the index files are pulled from master.
> >  * If the tlog is not present in a replica then during ungraceful
> shutdown
> > of the solr server how the replicas will regain the index without tlog
> > files.
> >  * To verify the above scenario, I killed the newly added replica server
> > with kill -9 <pid> command and started back
> >  also stopped the leader node.
> >
> > Questions:
> >  1) TLog files are not used even in the case of ungraceful shutdown,
> where
> > else it will be used?
> >  2) Tlog files doesn't get copied to the newly added replica so adding a
> > new replica to the already created collection with data/index is not
> > advisable?
> >  3) Is there a way to make the newly added slave node to replicate the
> > tlog file as it does for the data/index files from leader?
> >  4) Is it possible to use the Tlog files /index files from an existing
> > solr server to spin up a new solr cluster?
> >
> >
> > It would be much helpful for me to understand the core working of Solr
> > server.
> >
> > Thanks,
> > Sripradeep P
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Use of TLog

Emir Arnautović
Hi,
Copying indices will work and it is a fine approach. An alternative would be to join new node to a cluster, use add replica to copy cores to this new node and then remove replicas from old nodes, if you want to move cores.

Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Nov 2019, at 13:22, Sripra deep <[hidden email]> wrote:
>
> Hi Emir,
>
>  Thank you so much. Now I got it cleared with the TLOG purpose.
>   I am trying to copy an index of one solr cluster to use it to build
> other solr cluster. I am able to make that work but Is this design okay? or
> any other approach I can try to get a new cluster spin up with the same
> data as in the old one.
>
> Thanks,
> Sripradeep P
>
>
> On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
> [hidden email]> wrote:
>
>> Hi Sripradeep,
>> Simplified: TLog files are used to replay index updates from the last
>> successful hard commit in case of some Solr crashes. It is used on the next
>> Solr startup. It does not contain all updates, otherwise, it would
>> duplicate the index size.
>> If you start from these premises, you will understand why it is not copied
>> when adding replicas and why it is not needed and why you cannot use TLog
>> to spin up a new cluster.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 18 Nov 2019, at 06:35, Sripra deep <[hidden email]>
>> wrote:
>>>
>>> Hi Guys,
>>>
>>> I observed a scenario with the tlog creation and usage and couldn't find
>>> any usage for the tlog.
>>>
>>> Solr version: 7.1.0
>>> Number of shards = 3
>>> Number of replica = 1
>>> I indexed the about 10k docs into the collection.
>>>
>>> Scenario 1:
>>> Using add replica collection API, I created one more replica (tried with
>>> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
>>> Only the index files are pulled from master.
>>> * If the tlog is not present in a replica then during ungraceful
>> shutdown
>>> of the solr server how the replicas will regain the index without tlog
>>> files.
>>> * To verify the above scenario, I killed the newly added replica server
>>> with kill -9 <pid> command and started back
>>> also stopped the leader node.
>>>
>>> Questions:
>>> 1) TLog files are not used even in the case of ungraceful shutdown,
>> where
>>> else it will be used?
>>> 2) Tlog files doesn't get copied to the newly added replica so adding a
>>> new replica to the already created collection with data/index is not
>>> advisable?
>>> 3) Is there a way to make the newly added slave node to replicate the
>>> tlog file as it does for the data/index files from leader?
>>> 4) Is it possible to use the Tlog files /index files from an existing
>>> solr server to spin up a new solr cluster?
>>>
>>>
>>> It would be much helpful for me to understand the core working of Solr
>>> server.
>>>
>>> Thanks,
>>> Sripradeep P
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Use of TLog

Sripra deep
Got it, Thanks for helping me Emir.

On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović <
[hidden email]> wrote:

> Hi,
> Copying indices will work and it is a fine approach. An alternative would
> be to join new node to a cluster, use add replica to copy cores to this new
> node and then remove replicas from old nodes, if you want to move cores.
>
> Regards,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 18 Nov 2019, at 13:22, Sripra deep <[hidden email]>
> wrote:
> >
> > Hi Emir,
> >
> >  Thank you so much. Now I got it cleared with the TLOG purpose.
> >   I am trying to copy an index of one solr cluster to use it to build
> > other solr cluster. I am able to make that work but Is this design okay?
> or
> > any other approach I can try to get a new cluster spin up with the same
> > data as in the old one.
> >
> > Thanks,
> > Sripradeep P
> >
> >
> > On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
> > [hidden email]> wrote:
> >
> >> Hi Sripradeep,
> >> Simplified: TLog files are used to replay index updates from the last
> >> successful hard commit in case of some Solr crashes. It is used on the
> next
> >> Solr startup. It does not contain all updates, otherwise, it would
> >> duplicate the index size.
> >> If you start from these premises, you will understand why it is not
> copied
> >> when adding replicas and why it is not needed and why you cannot use
> TLog
> >> to spin up a new cluster.
> >>
> >> HTH,
> >> Emir
> >> --
> >> Monitoring - Log Management - Alerting - Anomaly Detection
> >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >>
> >>
> >>
> >>> On 18 Nov 2019, at 06:35, Sripra deep <[hidden email]>
> >> wrote:
> >>>
> >>> Hi Guys,
> >>>
> >>> I observed a scenario with the tlog creation and usage and couldn't
> find
> >>> any usage for the tlog.
> >>>
> >>> Solr version: 7.1.0
> >>> Number of shards = 3
> >>> Number of replica = 1
> >>> I indexed the about 10k docs into the collection.
> >>>
> >>> Scenario 1:
> >>> Using add replica collection API, I created one more replica (tried
> with
> >>> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
> >>> Only the index files are pulled from master.
> >>> * If the tlog is not present in a replica then during ungraceful
> >> shutdown
> >>> of the solr server how the replicas will regain the index without tlog
> >>> files.
> >>> * To verify the above scenario, I killed the newly added replica server
> >>> with kill -9 <pid> command and started back
> >>> also stopped the leader node.
> >>>
> >>> Questions:
> >>> 1) TLog files are not used even in the case of ungraceful shutdown,
> >> where
> >>> else it will be used?
> >>> 2) Tlog files doesn't get copied to the newly added replica so adding a
> >>> new replica to the already created collection with data/index is not
> >>> advisable?
> >>> 3) Is there a way to make the newly added slave node to replicate the
> >>> tlog file as it does for the data/index files from leader?
> >>> 4) Is it possible to use the Tlog files /index files from an existing
> >>> solr server to spin up a new solr cluster?
> >>>
> >>>
> >>> It would be much helpful for me to understand the core working of Solr
> >>> server.
> >>>
> >>> Thanks,
> >>> Sripradeep P
> >>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Use of TLog

Erick Erickson
There’s also the Collections API MOVEREPLICA command that does this all as a single command.

> On Nov 19, 2019, at 5:21 AM, Sripra deep <[hidden email]> wrote:
>
> Got it, Thanks for helping me Emir.
>
> On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović <
> [hidden email]> wrote:
>
>> Hi,
>> Copying indices will work and it is a fine approach. An alternative would
>> be to join new node to a cluster, use add replica to copy cores to this new
>> node and then remove replicas from old nodes, if you want to move cores.
>>
>> Regards,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 18 Nov 2019, at 13:22, Sripra deep <[hidden email]>
>> wrote:
>>>
>>> Hi Emir,
>>>
>>> Thank you so much. Now I got it cleared with the TLOG purpose.
>>>  I am trying to copy an index of one solr cluster to use it to build
>>> other solr cluster. I am able to make that work but Is this design okay?
>> or
>>> any other approach I can try to get a new cluster spin up with the same
>>> data as in the old one.
>>>
>>> Thanks,
>>> Sripradeep P
>>>
>>>
>>> On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović <
>>> [hidden email]> wrote:
>>>
>>>> Hi Sripradeep,
>>>> Simplified: TLog files are used to replay index updates from the last
>>>> successful hard commit in case of some Solr crashes. It is used on the
>> next
>>>> Solr startup. It does not contain all updates, otherwise, it would
>>>> duplicate the index size.
>>>> If you start from these premises, you will understand why it is not
>> copied
>>>> when adding replicas and why it is not needed and why you cannot use
>> TLog
>>>> to spin up a new cluster.
>>>>
>>>> HTH,
>>>> Emir
>>>> --
>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>>>
>>>>
>>>>
>>>>> On 18 Nov 2019, at 06:35, Sripra deep <[hidden email]>
>>>> wrote:
>>>>>
>>>>> Hi Guys,
>>>>>
>>>>> I observed a scenario with the tlog creation and usage and couldn't
>> find
>>>>> any usage for the tlog.
>>>>>
>>>>> Solr version: 7.1.0
>>>>> Number of shards = 3
>>>>> Number of replica = 1
>>>>> I indexed the about 10k docs into the collection.
>>>>>
>>>>> Scenario 1:
>>>>> Using add replica collection API, I created one more replica (tried
>> with
>>>>> both nrt and tlog) neither of the replicas doesn't pull the tlog files.
>>>>> Only the index files are pulled from master.
>>>>> * If the tlog is not present in a replica then during ungraceful
>>>> shutdown
>>>>> of the solr server how the replicas will regain the index without tlog
>>>>> files.
>>>>> * To verify the above scenario, I killed the newly added replica server
>>>>> with kill -9 <pid> command and started back
>>>>> also stopped the leader node.
>>>>>
>>>>> Questions:
>>>>> 1) TLog files are not used even in the case of ungraceful shutdown,
>>>> where
>>>>> else it will be used?
>>>>> 2) Tlog files doesn't get copied to the newly added replica so adding a
>>>>> new replica to the already created collection with data/index is not
>>>>> advisable?
>>>>> 3) Is there a way to make the newly added slave node to replicate the
>>>>> tlog file as it does for the data/index files from leader?
>>>>> 4) Is it possible to use the Tlog files /index files from an existing
>>>>> solr server to spin up a new solr cluster?
>>>>>
>>>>>
>>>>> It would be much helpful for me to understand the core working of Solr
>>>>> server.
>>>>>
>>>>> Thanks,
>>>>> Sripradeep P
>>>>
>>>>
>>
>>