SolrCloud indexing

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrCloud indexing

Moshe Recanati | KMS

Hi,

We’re using SolrCloud as part of our product solution for High Availability.

During upgrade of a version we need to run full index build on our Solr data.

I would like to know if as part of SolrCloud we can manage it and make sure that items are available during the index so only once specific item is indexed it’s changing with no affect on end-user.

If there is such option please send me some guidelines on how to do it.

 

Thank you in advance,

 

Regards,

Moshe Recanati

CTO

Mobile  + 972-52-6194481

Skype    :  recanati

More at:  www.kmslh.com | LinkedIn | FB

 

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud indexing

Shawn Heisey-2
On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>
> We’re using SolrCloud as part of our product solution for High
> Availability.
>
> During upgrade of a version we need to run full index build on our
> Solr data.
>

What are you upgrading?  If it's Solr, you should pause/stop indexing
while you do the upgrade.  You'll have to stop Solr processes to upgrade
them, and even if you're using an external load balancer, it takes a
little bit of time for failover to occur.

It would be up to your indexing software to handle errors in that
situation.  There is nothing that Solr can do about that.  If your
indexing software correctly detects and handles errors, then you might
be able to restart Solr instances without a problem.

> I would like to know if as part of SolrCloud we can manage it and make
> sure that items are available during the index so only once specific
> item is indexed it’s changing with no affect on end-user.
>

I can't decipher exactly what you're asking here.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud indexing

Erick Erickson
I think you're saying you want to prove out the upgrade in some kind
of test setup then switch live traffic. What's commonly used for that
is collection aliasing. You just create a new collection and populate
it and check it out. When you're satisfied that it's doing what you
want, use the collections API CREATEALIAS command to seamlessly
switch.
Here's the sequence:
old_collection is active
create new_collection, index to it and make sure it is doing what you want
CREATEALIAS pointing old_collection at new_colletion.

The advantage here is that you have as long as you want to verify your
new collection is fine. You can also switch back if you need to.

The disadvantage is that you effectively need extra hardware. You can
mitigate this somewhat by bringing up your new collection with a
limited number of replicas and after switching you delete replicas
from the old collection and add them to the new one. Be very very
careful here though that your collection commands
(ADDREPLICA/DELETEREPLICA) don't actually operate on the new
collection!

Best,
Erick

On Sun, Apr 15, 2018 at 10:22 AM, Shawn Heisey <[hidden email]> wrote:

> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>>
>> We’re using SolrCloud as part of our product solution for High
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our Solr
>> data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing while
> you do the upgrade.  You'll have to stop Solr processes to upgrade them, and
> even if you're using an external load balancer, it takes a little bit of
> time for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.
> There is nothing that Solr can do about that.  If your indexing software
> correctly detects and handles errors, then you might be able to restart Solr
> instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and make
>> sure that items are available during the index so only once specific item is
>> indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

RE: SolrCloud indexing

Moshe Recanati | KMS
Hi Erick,
Thank you very much.
I'll check it out.

Regards,
Moshe Recanati
CTO
Mobile  + 972-52-6194481
Skype    :  recanati

More at:  www.kmslh.com | LinkedIn | FB


-----Original Message-----
From: Erick Erickson <[hidden email]>
Sent: Monday, April 16, 2018 7:01 AM
To: solr-user <[hidden email]>
Subject: Re: SolrCloud indexing

I think you're saying you want to prove out the upgrade in some kind of test setup then switch live traffic. What's commonly used for that is collection aliasing. You just create a new collection and populate it and check it out. When you're satisfied that it's doing what you want, use the collections API CREATEALIAS command to seamlessly switch.
Here's the sequence:
old_collection is active
create new_collection, index to it and make sure it is doing what you want CREATEALIAS pointing old_collection at new_colletion.

The advantage here is that you have as long as you want to verify your new collection is fine. You can also switch back if you need to.

The disadvantage is that you effectively need extra hardware. You can mitigate this somewhat by bringing up your new collection with a limited number of replicas and after switching you delete replicas from the old collection and add them to the new one. Be very very careful here though that your collection commands
(ADDREPLICA/DELETEREPLICA) don't actually operate on the new collection!

Best,
Erick

On Sun, Apr 15, 2018 at 10:22 AM, Shawn Heisey <[hidden email]> wrote:

> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>>
>> We’re using SolrCloud as part of our product solution for High
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our
>> Solr data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing
> while you do the upgrade.  You'll have to stop Solr processes to
> upgrade them, and even if you're using an external load balancer, it
> takes a little bit of time for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.
> There is nothing that Solr can do about that.  If your indexing
> software correctly detects and handles errors, then you might be able
> to restart Solr instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and
>> make sure that items are available during the index so only once
>> specific item is indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

RE: SolrCloud indexing

Moshe Recanati | KMS
In reply to this post by Shawn Heisey-2
Hi Shawn,
Thank you.
I just need to run full indexing due to massive changes in the document.

Regards,
Moshe Recanati
CTO
Mobile  + 972-52-6194481
Skype    :  recanati

More at:  www.kmslh.com | LinkedIn | FB


-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Sunday, April 15, 2018 8:23 PM
To: [hidden email]
Subject: Re: SolrCloud indexing

On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>
> We’re using SolrCloud as part of our product solution for High
> Availability.
>
> During upgrade of a version we need to run full index build on our
> Solr data.
>

What are you upgrading?  If it's Solr, you should pause/stop indexing while you do the upgrade.  You'll have to stop Solr processes to upgrade them, and even if you're using an external load balancer, it takes a little bit of time for failover to occur.

It would be up to your indexing software to handle errors in that situation.  There is nothing that Solr can do about that.  If your indexing software correctly detects and handles errors, then you might be able to restart Solr instances without a problem.

> I would like to know if as part of SolrCloud we can manage it and make
> sure that items are available during the index so only once specific
> item is indexed it’s changing with no affect on end-user.
>

I can't decipher exactly what you're asking here.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud indexing

Erick Erickson
BTW, the effort you put into creating a process to totally replace
your index won't be wasted. I pretty much guarantee that you'll have
to re-index everything upon occasion if for no other reason than the
requirements will change. "Oh, you know that field we said never would
be used for phrase searches? We changed our mind"......... "We decided
to support 14 new use cases....." ;)...

On Sun, Apr 15, 2018 at 10:32 PM, Moshe Recanati | KMS <[hidden email]> wrote:

> Hi Shawn,
> Thank you.
> I just need to run full indexing due to massive changes in the document.
>
> Regards,
> Moshe Recanati
> CTO
> Mobile  + 972-52-6194481
> Skype    :  recanati
>
> More at:  www.kmslh.com | LinkedIn | FB
>
>
> -----Original Message-----
> From: Shawn Heisey <[hidden email]>
> Sent: Sunday, April 15, 2018 8:23 PM
> To: [hidden email]
> Subject: Re: SolrCloud indexing
>
> On 4/15/2018 1:22 AM, Moshe Recanati | KMS wrote:
>>
>> We’re using SolrCloud as part of our product solution for High
>> Availability.
>>
>> During upgrade of a version we need to run full index build on our
>> Solr data.
>>
>
> What are you upgrading?  If it's Solr, you should pause/stop indexing while you do the upgrade.  You'll have to stop Solr processes to upgrade them, and even if you're using an external load balancer, it takes a little bit of time for failover to occur.
>
> It would be up to your indexing software to handle errors in that situation.  There is nothing that Solr can do about that.  If your indexing software correctly detects and handles errors, then you might be able to restart Solr instances without a problem.
>
>> I would like to know if as part of SolrCloud we can manage it and make
>> sure that items are available during the index so only once specific
>> item is indexed it’s changing with no affect on end-user.
>>
>
> I can't decipher exactly what you're asking here.
>
> Thanks,
> Shawn
>