Lucene index upgrade from 4.6 to 8 facing issue

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti

Hi All,

 

I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying to upgrade tool using:

java -cp lucene-core.jar:lucene-backward-codecs.jar \

org.apache.lucene.index.IndexUpgrader -delete-prior-commits  \

/scratch/***/workspaces/trunk/****/indexes/4.6/

Script is working fine from 4.6 to 5.5.55.5.5 to 6.6.66.6.6 to 7.7.2. But When I am trying to upgrade from 7.7.2 to 8.0.0 it is throwing error as below.

Exception in thread "main" org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/scratch/jbaviset/workspaces/trunk/directorlocal/config/casemanagement/indexes/4.6/sd/segments_9"))): This index was initially created with Lucene 6.x while the current version is 8.0.0 and Lucene only supports reading the current and previous major versions.. This version of Lucene only supports indexes created with release 7.0 and later.

    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:318)

    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)

    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:846)

    at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:167)

    at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:78)


We want to upgrade from 4.6 to 8.0.0. Till 7.7.2 it is upgrading without any issues, but when I'm using for Lucene version 7.7.2 in my code indexing is working fine. But Lucene 8.0.0 is not working.

Please help me in upgrading 8.0.0

Please let me know any option other than reindexing.

 

Thanks,

Jyothsna

 

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Shawn Heisey
On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never
guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was
written into the index.  The recorded version was preserved by the
upgrader for version 7.  When the index was subsequently read by version
8, it complained because the original index was not written by version 7
or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Erick Erickson
No workaround exists that I know of.

Best,
Erick

> On Sep 26, 2019, at 2:21 PM, Jyothsna Bavisetti <[hidden email]> wrote:
>
> Hi Shawn,
>
> Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?
>
> Thanks,
> Jyothsna
> -----Original Message-----
> From: Shawn Heisey <[hidden email]>
> Sent: Thursday, September 26, 2019 11:35 PM
> To: [hidden email]
> Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue
>
> On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
>> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
>> to upgrade tool using:
>>
>> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>>
>> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \
>
> <snip>
>
>> Please let me know any option other than reindexing.
>
> If you're upgrading more than one major version, you must reindex.
> Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.
>
> When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.
>
> Thanks,
> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
In reply to this post by Jyothsna Bavisetti
Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Jörn Franke
I think it highly depends on how you want to do it. You could continue to run 4.6 and have another instance(a) with 8.2+. Then you just Index as you used to do it until now on the new one. Once you have all the data indexed you switch you lt application to 8.2. Of course you need to test it in dev, test etc environments.
Maybe in order to reduce time for the future and it you don’t use a database import. Convert your “rich” documents (eg pdf, wird, excel) in a suitable text format (or html) and start indexing from there. This will save you a lot of time on converting those documents in case of reindexing.

Then depending on the use case it could be also worthy to think about not indexing anymore old data (=> datensparsamkeit).

> Am 01.10.2019 um 23:19 schrieb Jyothsna Bavisetti <[hidden email]>:
>
> Hi Shawn,
>
> Any doc or links for re indexing process. We are using Lucene core 8.0.0.
>
>
> Thanks,
> Jyothsna
>
> -----Original Message-----
> From: Jyothsna Bavisetti
> Sent: Thursday, September 26, 2019 11:51 PM
> To: [hidden email]
> Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue
>
> Hi Shawn,
>
> Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?
>
> Thanks,
> Jyothsna
> -----Original Message-----
> From: Shawn Heisey <[hidden email]>
> Sent: Thursday, September 26, 2019 11:35 PM
> To: [hidden email]
> Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue
>
>> On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
>> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
>> to upgrade tool using:
>>
>> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>>
>> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \
>
> <snip>
>
>> Please let me know any option other than reindexing.
>
> If you're upgrading more than one major version, you must reindex.
> Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.
>
> When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.
>
> Thanks,
> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]