Help! - Max Segment name reached

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Help! - Max Segment name reached

Stuart Goldberg
We have an index that has run into this bug:
https://issues.apache.org/jira/browse/LUCENE-7999

 

Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4 and
cannot upgrade.

 

By looking at the code it seems that the last segment number counter is
persisted in segment_h. When creating a new segment, it names the segment
based on the persisted counter. If this counter is larger than
Integer.MAX_VALUE how can we recover this index.

 

Is there anything we can do?

 

Stuart M Goldberg

Senior Vice President of Software Develpment
FIX Flyer LLC
 <http://www.FIXFlyer.com/> http://www.FIXFlyer.com/

NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.

 

Reply | Threaded
Open this post in threaded view
|

Re: Help! - Max Segment name reached

Uwe Schindler
Hi,

Create a new empty index in a new directory and use addIndex() using the other directory with the broken index.

This will copy all segments but renumber them.

Uwe

Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <[hidden email]>:

>We have an index that has run into this bug:
>https://issues.apache.org/jira/browse/LUCENE-7999
>
>
>
>Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
>and
>cannot upgrade.
>
>
>
>By looking at the code it seems that the last segment number counter is
>persisted in segment_h. When creating a new segment, it names the
>segment
>based on the persisted counter. If this counter is larger than
>Integer.MAX_VALUE how can we recover this index.
>
>
>
>Is there anything we can do?
>
>
>
>Stuart M Goldberg
>
>Senior Vice President of Software Develpment
>FIX Flyer LLC
> <http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
>
>NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
>RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
>DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
>THIS
>INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE
>THIS
>E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

RE: Help! - Max Segment name reached

Stuart Goldberg
Thanks, I will try that.

Why haven't more people run into this issue? The next segment number is persisted, so if an index has a long life it should eventually run into this problem.

Stuart M Goldberg
Senior Vice President of Software Develpment
FIX Flyer LLC
http://www.FIXFlyer.com/
NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.

-----Original Message-----
From: Uwe Schindler <[hidden email]>
Sent: Tuesday, April 17, 2018 4:02 PM
To: [hidden email]
Subject: Re: Help! - Max Segment name reached

Hi,

Create a new empty index in a new directory and use addIndex() using the other directory with the broken index.

This will copy all segments but renumber them.

Uwe

Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <[hidden email]>:

>We have an index that has run into this bug:
>https://issues.apache.org/jira/browse/LUCENE-7999
>
>
>
>Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
>and cannot upgrade.
>
>
>
>By looking at the code it seems that the last segment number counter is
>persisted in segment_h. When creating a new segment, it names the
>segment based on the persisted counter. If this counter is larger than
>Integer.MAX_VALUE how can we recover this index.
>
>
>
>Is there anything we can do?
>
>
>
>Stuart M Goldberg
>
>Senior Vice President of Software Develpment FIX Flyer LLC  
><http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
>
>NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
>RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
>DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
>THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE
>INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE
>DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Help! - Max Segment name reached

Michael McCandless-2
Well I think as time goes on we'll see more and more people running into it
;)

But you really need to commit at a surprisingly high rate, and have a
surprisingly long lived index, to overflow the int that holds the segment
number.  E.g. if you commit once per second, it should take ~68 years to
overflow the int.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Apr 17, 2018 at 4:04 PM, Stuart Goldberg <[hidden email]>
wrote:

> Thanks, I will try that.
>
> Why haven't more people run into this issue? The next segment number is
> persisted, so if an index has a long life it should eventually run into
> this problem.
>
> Stuart M Goldberg
> Senior Vice President of Software Develpment
> FIX Flyer LLC
> http://www.FIXFlyer.com/
> NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
> -----Original Message-----
> From: Uwe Schindler <[hidden email]>
> Sent: Tuesday, April 17, 2018 4:02 PM
> To: [hidden email]
> Subject: Re: Help! - Max Segment name reached
>
> Hi,
>
> Create a new empty index in a new directory and use addIndex() using the
> other directory with the broken index.
>
> This will copy all segments but renumber them.
>
> Uwe
>
> Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <
> [hidden email]>:
> >We have an index that has run into this bug:
> >https://issues.apache.org/jira/browse/LUCENE-7999
> >
> >
> >
> >Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
> >and cannot upgrade.
> >
> >
> >
> >By looking at the code it seems that the last segment number counter is
> >persisted in segment_h. When creating a new segment, it names the
> >segment based on the persisted counter. If this counter is larger than
> >Integer.MAX_VALUE how can we recover this index.
> >
> >
> >
> >Is there anything we can do?
> >
> >
> >
> >Stuart M Goldberg
> >
> >Senior Vice President of Software Develpment FIX Flyer LLC
> ><http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
> >
> >NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> >RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> >WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> >DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
> >THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE
> >INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE
> >DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
> >
> >
>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>