How to optimize single shard only?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to optimize single shard only?

Marcin Rzewucki
Hi,

Do you know how to optimize index on a single shard only ? I was trying to
use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not work
- it optimizes all shards instead of just one.

Kind regards.
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

iorixxx
Hi Marcin,

just a guess, pass distrib=false ?



Ahmet


On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <[hidden email]> wrote:
Hi,

Do you know how to optimize index on a single shard only ? I was trying to
use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not work
- it optimizes all shards instead of just one.

Kind regards.

Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

YouPeng Yang
Hi Marcin

  Thanks to your mail,now I know why my cloud hangs when I just click the
optimize button on the overview page of the shard.


2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:

> Hi Marcin,
>
> just a guess, pass distrib=false ?
>
>
>
> Ahmet
>
>
> On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <[hidden email]>
> wrote:
> Hi,
>
> Do you know how to optimize index on a single shard only ? I was trying to
> use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not work
> - it optimizes all shards instead of just one.
>
> Kind regards.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

YouPeng Yang
Hi
 Maybe you can try _router_=myshard? I will check the source code ,note you
later.


2014-05-20 17:19 GMT+08:00 YouPeng Yang <[hidden email]>:

> Hi Marcin
>
>   Thanks to your mail,now I know why my cloud hangs when I just click the
> optimize button on the overview page of the shard.
>
>
> 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:
>
> Hi Marcin,
>>
>> just a guess, pass distrib=false ?
>>
>>
>>
>> Ahmet
>>
>>
>> On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <[hidden email]>
>> wrote:
>> Hi,
>>
>> Do you know how to optimize index on a single shard only ? I was trying to
>> use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not work
>> - it optimizes all shards instead of just one.
>>
>> Kind regards.
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

Marcin Rzewucki
In reply to this post by YouPeng Yang
Well, it should not hang if all is configured fine :) How many shards and
memory you have ? Note that optimize rewrites index so you might need
additional disk space for this process. Optimizing works fine however I'd
like to be able to do it on a single shard as well.


On 20 May 2014 11:19, YouPeng Yang <[hidden email]> wrote:

> Hi Marcin
>
>   Thanks to your mail,now I know why my cloud hangs when I just click the
> optimize button on the overview page of the shard.
>
>
> 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:
>
> > Hi Marcin,
> >
> > just a guess, pass distrib=false ?
> >
> >
> >
> > Ahmet
> >
> >
> > On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <[hidden email]>
> > wrote:
> > Hi,
> >
> > Do you know how to optimize index on a single shard only ? I was trying
> to
> > use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not
> work
> > - it optimizes all shards instead of just one.
> >
> > Kind regards.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

YouPeng Yang
Hi
  My DIH work indeed hangs, I have only four shards,each has a master and a
replica.Maybe jvm memory size is very low.it was 3G while the size of every
my core is almost 16GB.

 I also have found that the size of the master increased during the
optimization(you can check on the overview page of the core.).the
phenomenon is very werid. Is it because that the collection overall
optimization will comput and copy  all the docs of the whole collection.


Version Gen Size   Master (Searching)
1400501330248
 98396
   29.83 GB
 Master (Replicable)
1400501330888
 98397
-


  After I have check source code,unfortunatly,it seems the optimize action
distrib overall the collection.you can reference the
SolrCmdDistributor.distribCommit.


2014-05-20 17:27 GMT+08:00 Marcin Rzewucki <[hidden email]>:

> Well, it should not hang if all is configured fine :) How many shards and
> memory you have ? Note that optimize rewrites index so you might need
> additional disk space for this process. Optimizing works fine however I'd
> like to be able to do it on a single shard as well.
>
>
> On 20 May 2014 11:19, YouPeng Yang <[hidden email]> wrote:
>
> > Hi Marcin
> >
> >   Thanks to your mail,now I know why my cloud hangs when I just click the
> > optimize button on the overview page of the shard.
> >
> >
> > 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:
> >
> > > Hi Marcin,
> > >
> > > just a guess, pass distrib=false ?
> > >
> > >
> > >
> > > Ahmet
> > >
> > >
> > > On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <
> [hidden email]>
> > > wrote:
> > > Hi,
> > >
> > > Do you know how to optimize index on a single shard only ? I was trying
> > to
> > > use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not
> > work
> > > - it optimizes all shards instead of just one.
> > >
> > > Kind regards.
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

Marcin Rzewucki
As I wrote before index is being rewritten so it grows during optimization
and later is reduced. I guess there was OOM in your case.



On 20 May 2014 12:11, YouPeng Yang <[hidden email]> wrote:

> Hi
>   My DIH work indeed hangs, I have only four shards,each has a master and a
> replica.Maybe jvm memory size is very low.it was 3G while the size of
> every
> my core is almost 16GB.
>
>  I also have found that the size of the master increased during the
> optimization(you can check on the overview page of the core.).the
> phenomenon is very werid. Is it because that the collection overall
> optimization will comput and copy  all the docs of the whole collection.
>
>
> Version Gen Size   Master (Searching)
> 1400501330248
>  98396
>    29.83 GB
>  Master (Replicable)
> 1400501330888
>  98397
> -
>
>
>   After I have check source code,unfortunatly,it seems the optimize action
> distrib overall the collection.you can reference the
> SolrCmdDistributor.distribCommit.
>
>
> 2014-05-20 17:27 GMT+08:00 Marcin Rzewucki <[hidden email]>:
>
> > Well, it should not hang if all is configured fine :) How many shards and
> > memory you have ? Note that optimize rewrites index so you might need
> > additional disk space for this process. Optimizing works fine however I'd
> > like to be able to do it on a single shard as well.
> >
> >
> > On 20 May 2014 11:19, YouPeng Yang <[hidden email]> wrote:
> >
> > > Hi Marcin
> > >
> > >   Thanks to your mail,now I know why my cloud hangs when I just click
> the
> > > optimize button on the overview page of the shard.
> > >
> > >
> > > 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:
> > >
> > > > Hi Marcin,
> > > >
> > > > just a guess, pass distrib=false ?
> > > >
> > > >
> > > >
> > > > Ahmet
> > > >
> > > >
> > > > On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <
> > [hidden email]>
> > > > wrote:
> > > > Hi,
> > > >
> > > > Do you know how to optimize index on a single shard only ? I was
> trying
> > > to
> > > > use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not
> > > work
> > > > - it optimizes all shards instead of just one.
> > > >
> > > > Kind regards.
> > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize single shard only?

Erick Erickson
Marcin is correct. The index size on disk will perhaps double. (triple
in compound case). The reason is so you don't lose your index if the
process is interrupted.

Consider the case where you're optimizing to one segment.
1> All the current segments are copied into the new segment
2> The new segment is flushed
3> "control files" that tell Lucene what files constitute the valid
segment(s) are written.
4> the old segments are removed.

So at any point up to <3> if the system is killed, crashes, whatever,
then the old version of the index is intact and you can keep on
working, even optimizing again.

If, on the other hand, after each segment was written to the new
segment the old segment was deleted, interrupting the process (which
may be very long) would leave your index in an inconsistent state.

FWIW,
Erick

On Tue, May 20, 2014 at 4:14 AM, Marcin Rzewucki <[hidden email]> wrote:

> As I wrote before index is being rewritten so it grows during optimization
> and later is reduced. I guess there was OOM in your case.
>
>
>
> On 20 May 2014 12:11, YouPeng Yang <[hidden email]> wrote:
>
>> Hi
>>   My DIH work indeed hangs, I have only four shards,each has a master and a
>> replica.Maybe jvm memory size is very low.it was 3G while the size of
>> every
>> my core is almost 16GB.
>>
>>  I also have found that the size of the master increased during the
>> optimization(you can check on the overview page of the core.).the
>> phenomenon is very werid. Is it because that the collection overall
>> optimization will comput and copy  all the docs of the whole collection.
>>
>>
>> Version Gen Size   Master (Searching)
>> 1400501330248
>>  98396
>>    29.83 GB
>>  Master (Replicable)
>> 1400501330888
>>  98397
>> -
>>
>>
>>   After I have check source code,unfortunatly,it seems the optimize action
>> distrib overall the collection.you can reference the
>> SolrCmdDistributor.distribCommit.
>>
>>
>> 2014-05-20 17:27 GMT+08:00 Marcin Rzewucki <[hidden email]>:
>>
>> > Well, it should not hang if all is configured fine :) How many shards and
>> > memory you have ? Note that optimize rewrites index so you might need
>> > additional disk space for this process. Optimizing works fine however I'd
>> > like to be able to do it on a single shard as well.
>> >
>> >
>> > On 20 May 2014 11:19, YouPeng Yang <[hidden email]> wrote:
>> >
>> > > Hi Marcin
>> > >
>> > >   Thanks to your mail,now I know why my cloud hangs when I just click
>> the
>> > > optimize button on the overview page of the shard.
>> > >
>> > >
>> > > 2014-05-20 15:25 GMT+08:00 Ahmet Arslan <[hidden email]>:
>> > >
>> > > > Hi Marcin,
>> > > >
>> > > > just a guess, pass distrib=false ?
>> > > >
>> > > >
>> > > >
>> > > > Ahmet
>> > > >
>> > > >
>> > > > On Tuesday, May 20, 2014 10:23 AM, Marcin Rzewucki <
>> > [hidden email]>
>> > > > wrote:
>> > > > Hi,
>> > > >
>> > > > Do you know how to optimize index on a single shard only ? I was
>> trying
>> > > to
>> > > > use "optimize=true&waitFlush=true&shard.keys=myshard" but it does not
>> > > work
>> > > > - it optimizes all shards instead of just one.
>> > > >
>> > > > Kind regards.
>> > > >
>> > > >
>> > >
>> >
>>