Query over migrating a solr database from 7.7.1 to 8.7.0

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 


smime.p7s (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi There

 

As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.

 

It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.

 

Many Thanks

 

Matthew

 

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 

From: Flowerday, Matthew J <[hidden email]>
Sent: 07 January 2021 12:25
To: [hidden email]
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 


smime.p7s (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Query over migrating a solr database from 7.7.1 to 8.7.0

matthew sporleder
Did you commit?

> On Jan 9, 2021, at 5:44 AM, Flowerday, Matthew J <[hidden email]> wrote:
>
> 
> Hi There
>  
> As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.
>  
> It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.
>  
> Many Thanks
>  
> Matthew
>  
>  
> Matthew Flowerday | Consultant | ULEAF
> Unisys | 01908 774830| [hidden email]
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX
>  
> <image001.png>
>  
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.
> <image002.jpg>
>  
> <image003.jpg>
>  
> <image004.jpg>
> <image005.jpg>
> <image006.jpg>
> <image007.jpg>
>  
> From: Flowerday, Matthew J <[hidden email]>
> Sent: 07 January 2021 12:25
> To: [hidden email]
> Subject: Query over migrating a solr database from 7.7.1 to 8.7.0
>  
> Hi There
>  
> I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).
>  
> On my local windows machine I have a single solr server 7.7.1 installation
>  
> I upgraded in the following manner
>  
> Installed windows solr 8.7.0 on my machine in a different folder
> Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
> Brought up the solr
> Checked that queries work through the Solr Admin Tool and our application
>  
> This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.
>  
> An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):
>  
> {
>   "responseHeader":{
>     "status":0,
>     "QTime":4,
>     "params":{
>       "q":"id:9901020319M01-N26",
>       "_":"1610016003669"}},
>   "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[
>       {
>         "id":"9901020319M01-N26",
>         "groupId":"9901020319M01",
>         "urn":"N26",
>         "specification":"nominal",
>         "owningGroupId":"9901020319M01",
>         "description":"N26, Yates, Mike, Alan, Richard, MALE",
>         "group_t":"9901020319M01",
>         "nominalUrn_t":"N26",
>         "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
>         "dateTimeCreated_dt":"2020-12-30T12:00:53Z",
>         "title_t":"Captain",
>         "surname_t":"Yates",
>         "qualifier_t":"Voyager",
>         "forename1_t":"Mike",
>         "forename2_t":"Alan",
>         "forename3_t":"Richard",
>         "sex_t":"MALE",
>         "orderedType_t":"Nominal",
>         "_version_":1687507566832123904},
>       {
>         "id":"9901020319M01-N26",
>         "groupId":"9901020319M01",
>         "urn":"N26",
>         "specification":"nominal",
>         "owningGroupId":"9901020319M01",
>         "description":"N26, Yates, Mike, Alan, Richard, MALE",
>         "group_t":"9901020319M01",
>         "nominalUrn_t":"N26",
>         "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",
>         "dateTimeCreated_dt":"2020-12-30T12:00:53Z",
>         "title_t":"Captain",
>         "surname_t":"Yates",
>         "qualifier_t":"Voyager enterprise defiant yorktown xx yy",
>         "forename1_t":"Mike",
>         "forename2_t":"Alan",
>         "forename3_t":"Richard",
>         "sex_t":"MALE",
>         "orderedType_t":"Nominal",
>         "_version_":1688224966566215680}]
>   }}
>  
> I checked the solrconfig.xml file and it does have a uniqueKey set up
>  
>               <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
>                  
> <uniqueKey>id</uniqueKey>
>  
> I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).
>  
> Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).
>  
> Many thanks
>  
> Matthew
>  
> Matthew Flowerday | Consultant | ULEAF
> Unisys | 01908 774830| [hidden email]
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX
>  
> <image001.png>
>  
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.
> <image002.jpg>
>  
> <image003.jpg>
>  
> <image004.jpg>
> <image005.jpg>
> <image006.jpg>
> <image007.jpg>
>  
Reply | Threaded
Open this post in threaded view
|

Re:Query over migrating a solr database from 7.7.1 to 8.7.0

xiefengchang
In reply to this post by Flowerday, Matthew J

can you show the update request?  






At 2021-01-07 20:25:13, "Flowerday, Matthew J" <[hidden email]> wrote:

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 



 

Reply | Threaded
Open this post in threaded view
|

RE: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi There

 

Thanks for contacting me.

 

I carried out this analysis of the solr log from the updates I carried out at the time:

 

Looking at the update requests sent to Solr. The first update of an existing record generated

 

2021-01-07 06:04:18.958 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688206792619720704)]} 0 59

2021-01-07 06:04:19.186 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:04:19.196 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 1 ms

2021-01-07 06:04:19.198 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 228

 

And the record was duplicated:

 

 

The next update generated

 

2021-01-07 06:10:59.786 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688207212953993216)]} 0 20

2021-01-07 06:10:59.974 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:10:59.982 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:10:59.998 INFO  (qtp1458091526-26) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 208

 

Which looks the same as the previous command – so no real difference here.

 

And then the records looked like

 

 

And this shows that the original (7.7.1) item is untouched and only the 8.6.3 item is updated on subsequent updates.

 

A brand new record being sent to solr generate this dialog

 

2021-01-07 06:20:10.645 INFO  (qtp1458091526-25) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X15 (1688207790576762880), 9901020319M01-DI21 (1688207790587248640)]} 0 15

2021-01-07 06:20:10.798 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:20:10.802 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:20:10.803 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 153

 

And this has a similar update request line as the others – so no differences here. Solr just seems to leave the migrated records as is and just creates a duplicate when they are updated for some reason.

 

I hope this is what you are after.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 

From: xiefengchang <[hidden email]>
Sent: 10 January 2021 08:44
To: [hidden email]
Subject: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

can you show the update request?  

 

 

 

 

 

At 2021-01-07 20:25:13, "Flowerday, Matthew J" <[hidden email]> wrote:

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 




 


smime.p7s (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

matthew sporleder
I think the general advice is to do a full re-index on a major version upgrade.  Also - did you ever commit?

On Sun, Jan 10, 2021 at 11:13 AM Flowerday, Matthew J <[hidden email]> wrote:

Hi There

 

Thanks for contacting me.

 

I carried out this analysis of the solr log from the updates I carried out at the time:

 

Looking at the update requests sent to Solr. The first update of an existing record generated

 

2021-01-07 06:04:18.958 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688206792619720704)]} 0 59

2021-01-07 06:04:19.186 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:04:19.196 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 1 ms

2021-01-07 06:04:19.198 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 228

 

And the record was duplicated:

 

 

The next update generated

 

2021-01-07 06:10:59.786 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688207212953993216)]} 0 20

2021-01-07 06:10:59.974 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:10:59.982 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:10:59.998 INFO  (qtp1458091526-26) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 208

 

Which looks the same as the previous command – so no real difference here.

 

And then the records looked like

 

 

And this shows that the original (7.7.1) item is untouched and only the 8.6.3 item is updated on subsequent updates.

 

A brand new record being sent to solr generate this dialog

 

2021-01-07 06:20:10.645 INFO  (qtp1458091526-25) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X15 (1688207790576762880), 9901020319M01-DI21 (1688207790587248640)]} 0 15

2021-01-07 06:20:10.798 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:20:10.802 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:20:10.803 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 153

 

And this has a similar update request line as the others – so no differences here. Solr just seems to leave the migrated records as is and just creates a duplicate when they are updated for some reason.

 

I hope this is what you are after.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: xiefengchang <[hidden email]>
Sent: 10 January 2021 08:44
To: [hidden email]
Subject: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

can you show the update request?  

 

 

 

 

 

At 2021-01-07 20:25:13, "Flowerday, Matthew J" <[hidden email]> wrote:

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 




 

Reply | Threaded
Open this post in threaded view
|

RE: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi There

 

Thanks for replying to my query. Yes I had seen the notes saying that on upgrading to a new major release the advice is to wipe and re-index. But I did see this on Major Changes in Solr 8 | Apache Solr Reference Guide 8.7

 

Upgrade Prerequisites

If using SolrCloud, you must be on Solr 7.3.0 or higher. Solr’s LeaderInRecovery (LIR) functionality changed significantly in Solr 7.3. While these changes were back-compatible for all subsequent 7.x releases, that compatibility has been removed in 8.0. In order to upgrade to Solr 8.x, all nodes of your cluster must be running Solr 7.3 or higher. If an upgrade is attempted with nodes running versions earlier than 7.3, documents could be lost.

If you are not using Solr in SolrCloud mode (you use Standalone Mode instead), we expect you can upgrade to Solr 8 from any 7.x version without major issues.

So I was hoping to carry out just an upgrade as our customer’s data would take 2-3 weeks to re-index from scratch.

 

You mentioned about committing after carrying out an update. Were you referring to the running of the IndexUpgrader? This is what carried out

 

  • With solr service stopped
  • Open a CMD window
  • cd c:\solr-8.7.0\server\solr-webapp\webapp\WEB-INF\lib
  • "C:\Program Files\Java\jre11_0_2\bin"\java -cp lucene-core-8.7.0.jar;lucene-backward-codecs-8.7.0.jar org.apache.lucene.index.IndexUpgrader -verbose C:\solr-8.7.0\server\solr\uleaf\data\index

And it generated output of the form

 

MS 0 [2021-01-09T10:16:54.405441300Z; main]: initDynamicDefaults spins=true maxThreadCount=1 maxMergeCount=6

IFD 0 [2021-01-09T10:16:54.450123300Z; main]: init: current segments file is "segments_1s"; deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@7d68ef40

IFD 0 [2021-01-09T10:16:54.451123400Z; main]: init: load commit "segments_1s"

IFD 0 [2021-01-09T10:16:54.456124300Z; main]: init: seg=_1l set nextWriteFieldInfosGen=2 vs current=1

IFD 0 [2021-01-09T10:16:54.456727400Z; main]: init: seg=_1l set nextWriteDocValuesGen=2 vs current=1

IFD 0 [2021-01-09T10:16:54.462160900Z; main]: now checkpoint "_1i(7.7.1):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=7.7.1, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1609332625247}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] _1j(7.7.1):C16:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=7.7.1, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1609335456070}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] _1k(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610016791080}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]

 

Ending with

 

IFD 0 [2021-01-09T10:16:55.448195200Z; main]: now checkpoint "_1q(8.7.0):C17:[diagnostics={os=Windows 10, java.version=11.0.2, os.arch=amd64, java.runtime.version=11.0.2+9, source=merge, os.version=10.0, java.vendor=AdoptOpenJDK, java.vm.version=11.0.2+9, lucene.version=8.7.0, mergeMaxNumSegments=1, mergeFactor=2, timestamp=1610187414535}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=i2dct7nwpvi35aycws4u8gwu _1k(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610016791080}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=dop9d815p1kjrk1phi955lbg8 _1l(8.7.0):C2/1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610097123034}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1 :id=dop9d815p1kjrk1phi955lbgh _1m(8.7.0):C2:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610097143494}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=dop9d815p1kjrk1phi955lbgg _1n(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610097253797}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=dop9d815p1kjrk1phi955lbgl _1p(8.7.0):C1:[diagnostics={java.vendor=AdoptOpenJDK, os=Windows 10, java.version=11.0.2, java.vm.version=11.0.2+9, lucene.version=8.7.0, os.arch=amd64, java.runtime.version=11.0.2+9, source=flush, os.version=10.0, timestamp=1610097594499}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] :id=dop9d815p1kjrk1phi955lbgt" [6 segments ; isCommit = false]

IFD 0 [2021-01-09T10:16:55.457043500Z; main]: 10 msec to checkpoint

 

I was not aware about the need to carry out an commit after this. Please can you show me how to do this. I checked the Index folder before and after running the tool and the number of files reduced – so it seemed to have done some ‘combining’.

 

If I run the tool again it seems to run through similar dialog and does not report that nothing needs correcting.

 

Also please could you confirm if my steps to upgrade a solr database to a new release are correct (just in case what I did caused this issue).

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 

From: matthew sporleder <[hidden email]>
Sent: 10 January 2021 21:38
To: [hidden email]
Subject: Re: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

I think the general advice is to do a full re-index on a major version upgrade.  Also - did you ever commit?

 

On Sun, Jan 10, 2021 at 11:13 AM Flowerday, Matthew J <[hidden email]> wrote:

Hi There

 

Thanks for contacting me.

 

I carried out this analysis of the solr log from the updates I carried out at the time:

 

Looking at the update requests sent to Solr. The first update of an existing record generated

 

2021-01-07 06:04:18.958 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688206792619720704)]} 0 59

2021-01-07 06:04:19.186 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:04:19.196 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 1 ms

2021-01-07 06:04:19.198 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 228

 

And the record was duplicated:

 

 

The next update generated

 

2021-01-07 06:10:59.786 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X11 (1688207212953993216)]} 0 20

2021-01-07 06:10:59.974 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:10:59.982 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:10:59.998 INFO  (qtp1458091526-26) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 208

 

Which looks the same as the previous command – so no real difference here.

 

And then the records looked like

 

 

And this shows that the original (7.7.1) item is untouched and only the 8.6.3 item is updated on subsequent updates.

 

A brand new record being sent to solr generate this dialog

 

2021-01-07 06:20:10.645 INFO  (qtp1458091526-25) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9901020319M01-X15 (1688207790576762880), 9901020319M01-DI21 (1688207790587248640)]} 0 15

2021-01-07 06:20:10.798 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.QuerySenderListener QuerySenderListener done.

2021-01-07 06:20:10.802 INFO  (searcherExecutor-15-thread-1-processing-x:uleaf) [   x:uleaf] o.a.s.c.SolrCore [uleaf]  Registered new searcher autowarm time: 0 ms

2021-01-07 06:20:10.803 INFO  (qtp1458091526-23) [   x:uleaf] o.a.s.u.p.LogUpdateProcessorFactory [uleaf]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 153

 

And this has a similar update request line as the others – so no differences here. Solr just seems to leave the migrated records as is and just creates a duplicate when they are updated for some reason.

 

I hope this is what you are after.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: xiefengchang <[hidden email]>
Sent: 10 January 2021 08:44
To:
[hidden email]
Subject: Re:Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

can you show the update request?  

 

 

 

 

 

At 2021-01-07 20:25:13, "Flowerday, Matthew J" <[hidden email]> wrote:

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

 



 

 


smime.p7s (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Query over migrating a solr database from 7.7.1 to 8.7.0

Dyer, Jim
In reply to this post by Flowerday, Matthew J

When we upgraded from 7.x to 8.x, I ran into an issue similar to yours:  when updating an existing document in the index, the document would be duplicated instead of replaced as expected.  The solution was to add a "_root_" field to schema.xml like this:

 

<field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

It appeared that when a feature was added for nested documents, this field somehow became mandatory in order for updates to work properly, at least in some cases.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Saturday, January 9, 2021 4:44 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.

 

It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.

 

Many Thanks

 

Matthew

 

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Flowerday, Matthew J <[hidden email]>
Sent: 07 January 2021 12:25
To: [hidden email]
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

Reply | Threaded
Open this post in threaded view
|

RE: Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi Jim

 

Thanks for getting back to me.

 

I checked the schema.xml that we are using and it has the line you mentioned:

 

    <field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

And this is the only reference (apart from within a comment) for _root_ In the schema.xml. Does your schema.xml have further references to _root_ that I could need? I also checked out solrconfig.xml file for any references to _root_ and there are none.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 

From: Dyer, Jim <[hidden email]>
Sent: 11 January 2021 22:58
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

When we upgraded from 7.x to 8.x, I ran into an issue similar to yours:  when updating an existing document in the index, the document would be duplicated instead of replaced as expected.  The solution was to add a "_root_" field to schema.xml like this:

 

<field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

It appeared that when a feature was added for nested documents, this field somehow became mandatory in order for updates to work properly, at least in some cases.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Saturday, January 9, 2021 4:44 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.

 

It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.

 

Many Thanks

 

Matthew

 

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Flowerday, Matthew J <[hidden email]>
Sent: 07 January 2021 12:25
To: [hidden email]
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 


smime.p7s (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Query over migrating a solr database from 7.7.1 to 8.7.0

Dyer, Jim

I think if you have _root_ in schema.xml you should look elsewhere.  My memory is merely adding this one line to schema.xml took care of our problem.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Tuesday, January 12, 2021 3:23 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi Jim

 

Thanks for getting back to me.

 

I checked the schema.xml that we are using and it has the line you mentioned:

 

    <field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

And this is the only reference (apart from within a comment) for _root_ In the schema.xml. Does your schema.xml have further references to _root_ that I could need? I also checked out solrconfig.xml file for any references to _root_ and there are none.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Dyer, Jim <[hidden email]>
Sent: 11 January 2021 22:58
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

When we upgraded from 7.x to 8.x, I ran into an issue similar to yours:  when updating an existing document in the index, the document would be duplicated instead of replaced as expected.  The solution was to add a "_root_" field to schema.xml like this:

 

<field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

It appeared that when a feature was added for nested documents, this field somehow became mandatory in order for updates to work properly, at least in some cases.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Saturday, January 9, 2021 4:44 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.

 

It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.

 

Many Thanks

 

Matthew

 

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Flowerday, Matthew J <[hidden email]>
Sent: 07 January 2021 12:25
To: [hidden email]
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

Reply | Threaded
Open this post in threaded view
|

RE: Query over migrating a solr database from 7.7.1 to 8.7.0

Flowerday, Matthew J

Hi Jim

 

Thanks for looking into it for me.

 

I did some more testing and if I created a base solr 7.7.1 database using the ‘out of the box’ schema.xml and solrconfig and add this item manually using the Solr Admin tool documents/XML

 

<doc>

<field name="id">ABCD-N1</field>

<field name="title_t">A test</field>

</doc>

 

And then update it using

 

<doc>

<field name="id">ABCD-N1</field>

<field name="title_t">A test updated</field>

</doc>

 

It correctly updates and deletes the old copy.

 

I then ‘migrated’ it to solr 8.7.0 and updated the record in the same manner (using documents/XML) with this

 

<doc>

<field name="id">ABCD-N1</field>

<field name="title_t">A test updated again</field>

</doc>

 

It created a new record without deleting the old record

 

{

  "responseHeader":{

    "status":0,

    "QTime":1,

    "params":{

      "q":"*:*",

      "_":"1610703647168"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"ABCD-N1",

        "title_t":"A test updated",

        "_version_":1688944583266795520},

      {

        "id":"ABCD-N1",

        "title_t":"A test updated again",

        "_version_":1688950299184594944}]

  }}

 

It is almost as if the delete of the record from the segment set up 7.7.1 is not recognised.

 

When I updated the record again using

 

<doc>

<field name="id">ABCD-N1</field>

<field name="title_t">A test updated again and again</field>

</doc>

 

It updated the newly created record  and deleted the old version.

 

{

  "responseHeader":{

    "status":0,

    "QTime":1,

    "params":{

      "q":"*:*",

      "_":"1610703647168"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"ABCD-N1",

        "title_t":"A test updated",

        "_version_":1688944583266795520},

      {

        "id":"ABCD-N1",

        "title_t":"A test updated again and again",

        "_version_":1688950897568120832}]

  }}

 

I did further testing by turning on lucene TRACE on my database and first update generated

 

2021-01-15 09:38:30.138 INFO  (qtp1458091526-18) [   x:uleaf] o.a.s.u.LoggingInfoStream [BD][qtp1458091526-18]: now apply del packet ([hidden email]) to 10 segments, mergeGen 0

2021-01-15 09:38:30.138 INFO  (qtp1458091526-18) [   x:uleaf] o.a.s.u.LoggingInfoStream [BD][qtp1458091526-18]: applyTermDeletes took 0.44 msec for 10 segments and 1 del terms; 0 new deletions

 

Whilst the second update generated

 

2021-01-15 09:44:21.543 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.LoggingInfoStream [BD][qtp1458091526-17]: now apply del packet ([hidden email]) to 11 segments, mergeGen 0

2021-01-15 09:44:21.544 INFO  (qtp1458091526-17) [   x:uleaf] o.a.s.u.LoggingInfoStream [BD][qtp1458091526-17]: applyTermDeletes took 0.29 msec for 11 segments and 1 del terms; 1 new deletions

 

 

I think that it does not seem to find the document to delete in the old segment.

 

Could this be a bug in Solr?

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

unisys_logo

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

Grey_LI  Grey_TW Grey_YTGrey_FBGrey_VimeoGrey_UB

 

From: Dyer, Jim <[hidden email]>
Sent: 13 January 2021 18:21
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

I think if you have _root_ in schema.xml you should look elsewhere.  My memory is merely adding this one line to schema.xml took care of our problem.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Tuesday, January 12, 2021 3:23 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi Jim

 

Thanks for getting back to me.

 

I checked the schema.xml that we are using and it has the line you mentioned:

 

    <field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

And this is the only reference (apart from within a comment) for _root_ In the schema.xml. Does your schema.xml have further references to _root_ that I could need? I also checked out solrconfig.xml file for any references to _root_ and there are none.

 

Many Thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Dyer, Jim <[hidden email]>
Sent: 11 January 2021 22:58
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

EXTERNAL EMAIL - Be cautious of all links and attachments.

When we upgraded from 7.x to 8.x, I ran into an issue similar to yours:  when updating an existing document in the index, the document would be duplicated instead of replaced as expected.  The solution was to add a "_root_" field to schema.xml like this:

 

<field name="_root_" type="string" indexed="true" stored="false" docValues="false" />

 

It appeared that when a feature was added for nested documents, this field somehow became mandatory in order for updates to work properly, at least in some cases.

 

From: Flowerday, Matthew J <[hidden email]>
Sent: Saturday, January 9, 2021 4:44 AM
To: [hidden email]
Subject: RE: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

As a test I stopped Solr and ran the IndexUpgrader tool on the database to see if this might fix the issue. It completed OK but unfortunately the issue still occurs – a new version of the record on solr is created rather than updating the original record.

 

It looks to me as if the record created under 7.7.1 is somehow not being ‘marked as deleted’ in the way that records created under 8.7.0 are. Is there a way for these records to be marked as deleted when they are updated.

 

Many Thanks

 

Matthew

 

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 

From: Flowerday, Matthew J <[hidden email]>
Sent: 07 January 2021 12:25
To: [hidden email]
Subject: Query over migrating a solr database from 7.7.1 to 8.7.0

 

Hi There

 

I have recently upgraded a solr database from 7.7.1 to 8.7.0 and not wiped the database and re-indexed (as this would take too long to run on site).

 

On my local windows machine I have a single solr server 7.7.1 installation

 

I upgraded in the following manner

 

  • Installed windows solr 8.7.0 on my machine in a different folder
  • Copied the core related folder (holding conf, data, lib, core.properties) from 7.7.1 to the new 8.7.0 folder
  • Brought up the solr
  • Checked that queries work through the Solr Admin Tool and our application

 

This all worked fine until I tried to update a record which had been created under 7.7.1. Instead of marking the old record as deleted it effectively created a new copy of the record with the change in and left the old image as still visible. When I updated the record again it then correctly updated the new 8.7.0 version without leaving the old image behind. If I created a new record and then updated it the solr record would be updated correctly. The issue only seemed to affect the old 7.7.1 created records.

 

An example of the duplication as follows (the first record is 7.7.1 created version and the second record is the 8.7.0 version after carrying out an update):

 

{

  "responseHeader":{

    "status":0,

    "QTime":4,

    "params":{

      "q":"id:9901020319M01-N26",

      "_":"1610016003669"}},

  "response":{"numFound":2,"start":0,"numFoundExact":true,"docs":[

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1687507566832123904},

      {

        "id":"9901020319M01-N26",

        "groupId":"9901020319M01",

        "urn":"N26",

        "specification":"nominal",

        "owningGroupId":"9901020319M01",

        "description":"N26, Yates, Mike, Alan, Richard, MALE",

        "group_t":"9901020319M01",

        "nominalUrn_t":"N26",

        "dateTimeCreated_dtr":"2020-12-30T12:00:53Z",

        "dateTimeCreated_dt":"2020-12-30T12:00:53Z",

        "title_t":"Captain",

        "surname_t":"Yates",

        "qualifier_t":"Voyager enterprise defiant yorktown xx yy",

        "forename1_t":"Mike",

        "forename2_t":"Alan",

        "forename3_t":"Richard",

        "sex_t":"MALE",

        "orderedType_t":"Nominal",

        "_version_":1688224966566215680}]

  }}

 

I checked the solrconfig.xml file and it does have a uniqueKey set up

 

              <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

                 

<uniqueKey>id</uniqueKey>

 

I was wondering if this behaviour is expected and if there is a way to make sure that records created under a previous version are updated correctly (so that the old data is deleted when updated).

 

Also am I upgrading solr correctly as it could be that the way I have upgraded it might be causing this issue (I tried hunting through the solr documentation online but struggled to find window upgrade notes and the above steps I worked out by trial and error).

 

Many thanks

 

Matthew

 

Matthew Flowerday | Consultant | ULEAF

Unisys | 01908 774830| [hidden email]

Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17 8LX

 

 

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all devices.

   

 


smime.p7s (10K) Download Attachment