Lucene index upgrade from 4.6 to 8 facing issue

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti

Hi All,

 

I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying to upgrade tool using:

java -cp lucene-core.jar:lucene-backward-codecs.jar \

org.apache.lucene.index.IndexUpgrader -delete-prior-commits  \

/scratch/***/workspaces/trunk/****/indexes/4.6/

Script is working fine from 4.6 to 5.5.55.5.5 to 6.6.66.6.6 to 7.7.2. But When I am trying to upgrade from 7.7.2 to 8.0.0 it is throwing error as below.

Exception in thread "main" org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/scratch/jbaviset/workspaces/trunk/directorlocal/config/casemanagement/indexes/4.6/sd/segments_9"))): This index was initially created with Lucene 6.x while the current version is 8.0.0 and Lucene only supports reading the current and previous major versions.. This version of Lucene only supports indexes created with release 7.0 and later.

    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:318)

    at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)

    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:846)

    at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:167)

    at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:78)


We want to upgrade from 4.6 to 8.0.0. Till 7.7.2 it is upgrading without any issues, but when I'm using for Lucene version 7.7.2 in my code indexing is working fine. But Lucene 8.0.0 is not working.

Please help me in upgrading 8.0.0

Please let me know any option other than reindexing.

 

Thanks,

Jyothsna

 

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Shawn Heisey
On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never
guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was
written into the index.  The recorded version was preserved by the
upgrader for version 7.  When the index was subsequently read by version
8, it complained because the original index was not written by version 7
or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Erick Erickson
No workaround exists that I know of.

Best,
Erick

> On Sep 26, 2019, at 2:21 PM, Jyothsna Bavisetti <[hidden email]> wrote:
>
> Hi Shawn,
>
> Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?
>
> Thanks,
> Jyothsna
> -----Original Message-----
> From: Shawn Heisey <[hidden email]>
> Sent: Thursday, September 26, 2019 11:35 PM
> To: [hidden email]
> Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue
>
> On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
>> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
>> to upgrade tool using:
>>
>> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>>
>> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \
>
> <snip>
>
>> Please let me know any option other than reindexing.
>
> If you're upgrading more than one major version, you must reindex.
> Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.
>
> When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.
>
> Thanks,
> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
In reply to this post by Jyothsna Bavisetti
Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene index upgrade from 4.6 to 8 facing issue

Jörn Franke
I think it highly depends on how you want to do it. You could continue to run 4.6 and have another instance(a) with 8.2+. Then you just Index as you used to do it until now on the new one. Once you have all the data indexed you switch you lt application to 8.2. Of course you need to test it in dev, test etc environments.
Maybe in order to reduce time for the future and it you don’t use a database import. Convert your “rich” documents (eg pdf, wird, excel) in a suitable text format (or html) and start indexing from there. This will save you a lot of time on converting those documents in case of reindexing.

Then depending on the use case it could be also worthy to think about not indexing anymore old data (=> datensparsamkeit).

> Am 01.10.2019 um 23:19 schrieb Jyothsna Bavisetti <[hidden email]>:
>
> Hi Shawn,
>
> Any doc or links for re indexing process. We are using Lucene core 8.0.0.
>
>
> Thanks,
> Jyothsna
>
> -----Original Message-----
> From: Jyothsna Bavisetti
> Sent: Thursday, September 26, 2019 11:51 PM
> To: [hidden email]
> Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue
>
> Hi Shawn,
>
> Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?
>
> Thanks,
> Jyothsna
> -----Original Message-----
> From: Shawn Heisey <[hidden email]>
> Sent: Thursday, September 26, 2019 11:35 PM
> To: [hidden email]
> Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue
>
>> On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
>> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
>> to upgrade tool using:
>>
>> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>>
>> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \
>
> <snip>
>
>> Please let me know any option other than reindexing.
>
> If you're upgrading more than one major version, you must reindex.
> Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.
>
> When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.
>
> Thanks,
> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
In reply to this post by Jyothsna Bavisetti
Hi All,

1. After Upgrading Lucene from 4.6 to 8, facing issue in search process.
2.We are creating 5 different folder for indexing with different index id ( 5 folders from 5 different tables). During search process we will join all these data to display.  After upgrading we are facing issue in search data.
3. When we are applying filter for different fields from different index_id. It is searching for only one field.
4. I am seeing difference in Join Query.
JoinUtil.createJoinQuery(fromField, false, toField, Query, srch, ScoreMode.None);
Query formation with different versions:
Lucene 4.6.0:
TermsQuery{field=case.id}
fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter)
Lucene 8.0.0:
TermsQuery{field=case.idfromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000}

fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter is also visible.)

Please Suggest me , as new to the Lucene unable to predict it.

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Thank you,


Thanks,
Jyothsna

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
In latest version form query is included in TermsQuery . So no difference in TermsQuery.
Still unable to figure it out the reason why filter is not applicable when form field data is in different indexing folder.
Please suggest any ideas to debug.

Thanks,
Jyothsna



-----Original Message-----
From: Jyothsna Bavisetti <[hidden email]>
Sent: Wednesday, October 30, 2019 11:53 AM
To: '[hidden email]' <[hidden email]>
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi All,

1. After Upgrading Lucene from 4.6 to 8, facing issue in search process.
2.We are creating 5 different folder for indexing with different index id ( 5 folders from 5 different tables). During search process we will join all these data to display.  After upgrading we are facing issue in search data.
3. When we are applying filter for different fields from different index_id. It is searching for only one field.
4. I am seeing difference in Join Query.
JoinUtil.createJoinQuery(fromField, false, toField, Query, srch, ScoreMode.None); Query formation with different versions:
Lucene 4.6.0:
TermsQuery{field=case.id}
fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter)

Lucene 8.0.0:
TermsQuery{field=case.id fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000}

fromQuery=+(history.attribute:SExtendedAttribute1) +history.modifiedBy:3ff0000000000000  (different parameter is also visible.)

Actual Filter Condition:
[Filter:
  condition: (&(case.caseType in (Scase))(case.sourceName in (SDEDUPLICATE))(&(history.attribute in (SExtendedAttribute1))(history.modifiedBy eq 3ff0000000000000))) ]



Please Suggest me , as new to the Lucene unable to predict it.


Thank you.


Thanks,
Jyothsna
-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue


Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti
In reply to this post by Jyothsna Bavisetti
Hi All,

Please help me with below error:
java.lang.IllegalStateException: unexpected docvalues type NONE for field 'case.id' (expected one of [BINARY, SORTED]). Re-index with correct docvalues type.
        at org.apache.lucene.index.DocValues.checkField(DocValues.java:317)
        at org.apache.lucene.index.DocValues.getBinary(DocValues.java:352)
        at org.apache.lucene.search.join.DocValuesTermsCollector.lambda$binaryDocValues$0(DocValuesTermsCollector.java:48)
        at org.apache.lucene.search.join.DocValuesTermsCollector.doSetNextReader(DocValuesTermsCollector.java:44)
        at org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:33)
        at org.apache.lucene.search.join.GenericTermsCollector$2.getLeafCollector(GenericTermsCollector.java:128)


any pointers for this issue please let me know.

Thank you!

Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene index upgrade from 4.6 to 8 facing issue

Jyothsna Bavisetti


Hi All,

Could please share some points when to go with longBitset vs FixedBitSet.

I read below Points:

BitSet of fixed length (numBits), backed by accessible (getBits()) long[], accessed with a long index. Use it only if you intend to store more than 2.1B bits, otherwise you should use FixedBitSet.
NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.


Thanks,
Jyothsna
-----Original Message-----
From: Jyothsna Bavisetti
Sent: Wednesday, October 2, 2019 2:49 AM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Any doc or links for re indexing process. We are using Lucene core 8.0.0.


Thanks,
Jyothsna

-----Original Message-----
From: Jyothsna Bavisetti
Sent: Thursday, September 26, 2019 11:51 PM
To: [hidden email]
Subject: RE: Lucene index upgrade from 4.6 to 8 facing issue

Hi Shawn,

Re-indexing is costly transaction in my use case as it takes more than three days. Please let me know if any work around?

Thanks,
Jyothsna
-----Original Message-----
From: Shawn Heisey <[hidden email]>
Sent: Thursday, September 26, 2019 11:35 PM
To: [hidden email]
Subject: Re: Lucene index upgrade from 4.6 to 8 facing issue

On 9/26/2019 11:41 AM, Jyothsna Bavisetti wrote:
> I am trying to upgrade Lucene index from 4.6 to 8.0.0. When I'm trying
> to upgrade tool using:
>
> java -cp lucene-core.jar:lucene-backward-codecs.jar \
>
> org.apache.lucene.index.IndexUpgrader-delete-prior-commits  \

<snip>

> Please let me know any option other than reindexing.

If you're upgrading more than one major version, you must reindex.
Multiple major version upgrades have always been discouraged and never guaranteed to work, but now such upgrades are explicitly denied.

When you used the IndexUpgrader from Lucene 6, the Lucene version was written into the index.  The recorded version was preserved by the upgrader for version 7.  When the index was subsequently read by version 8, it complained because the original index was not written by version 7 or later.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]