Query on changing FieldType

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Query on changing FieldType

Shubham Goswami
Hello Community

I have indexed some documents for which solr has taken its type="plongs" by
auto guessing but i am trying to change its type="pint" and re-indexing the
same data with the same id and indexing the data with different id where id
is unique key but it is showing error.

Can somebody please let me know if it is possible or not, if not possible
then why it is not possible as i am using different id as well ? if
possible then how we could achieve it ?
Any help will be appreciated. Thanks in advance.

--
*Thanks & Regards*
Shubham Goswami
Enterprise Software Engineer
*HotWax Systems*
*Enterprise open source experts*
cell: +91-7803886288
office: 0731-409-3684
http://www.hotwaxsystems.com
Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Emir Arnautović
Hi Shubham,
Changing type is not allowed without full reindexing. If you do something like that, Solr will end up with segments with different types for the same field. Remember that segments are immutable and that reindexing some document will be in new segment, but old segment will still be there and at query type Solr will have mismatch between what is stated in schema and what is in segment. In order to change type you have to do full reindexing - create a new collection and reindex all documents.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 22 Oct 2019, at 09:25, Shubham Goswami <[hidden email]> wrote:
>
> Hello Community
>
> I have indexed some documents for which solr has taken its type="plongs" by
> auto guessing but i am trying to change its type="pint" and re-indexing the
> same data with the same id and indexing the data with different id where id
> is unique key but it is showing error.
>
> Can somebody please let me know if it is possible or not, if not possible
> then why it is not possible as i am using different id as well ? if
> possible then how we could achieve it ?
> Any help will be appreciated. Thanks in advance.
>
> --
> *Thanks & Regards*
> Shubham Goswami
> Enterprise Software Engineer
> *HotWax Systems*
> *Enterprise open source experts*
> cell: +91-7803886288
> office: 0731-409-3684
> http://www.hotwaxsystems.com

Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Shubham Goswami
Hi Emir

Thanks for the reply, i got your point.
But is there any other way to do like one field could have two or more
different types defined ?
or  if i talk about my previous query, can we index some data for the same
field with different unique id after replacing the type ?

Thanks again
Shubham

On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
[hidden email]> wrote:

> Hi Shubham,
> Changing type is not allowed without full reindexing. If you do something
> like that, Solr will end up with segments with different types for the same
> field. Remember that segments are immutable and that reindexing some
> document will be in new segment, but old segment will still be there and at
> query type Solr will have mismatch between what is stated in schema and
> what is in segment. In order to change type you have to do full reindexing
> - create a new collection and reindex all documents.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 22 Oct 2019, at 09:25, Shubham Goswami <[hidden email]>
> wrote:
> >
> > Hello Community
> >
> > I have indexed some documents for which solr has taken its type="plongs"
> by
> > auto guessing but i am trying to change its type="pint" and re-indexing
> the
> > same data with the same id and indexing the data with different id where
> id
> > is unique key but it is showing error.
> >
> > Can somebody please let me know if it is possible or not, if not possible
> > then why it is not possible as i am using different id as well ? if
> > possible then how we could achieve it ?
> > Any help will be appreciated. Thanks in advance.
> >
> > --
> > *Thanks & Regards*
> > Shubham Goswami
> > Enterprise Software Engineer
> > *HotWax Systems*
> > *Enterprise open source experts*
> > cell: +91-7803886288
> > office: 0731-409-3684
> > http://www.hotwaxsystems.com
>
>

--
*Thanks & Regards*
Shubham Goswami
Enterprise Software Engineer
*HotWax Systems*
*Enterprise open source experts*
cell: +91-7803886288
office: 0731-409-3684
http://www.hotwaxsystems.com
Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Emir Arnautović
Hi Shubham,
No you cannot. What you can do is to use copy field or update request processor to store is as some other field and use that in your query and ignore the old one that will eventually disappear as the result of segment merges.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]> wrote:
>
> Hi Emir
>
> Thanks for the reply, i got your point.
> But is there any other way to do like one field could have two or more
> different types defined ?
> or  if i talk about my previous query, can we index some data for the same
> field with different unique id after replacing the type ?
>
> Thanks again
> Shubham
>
> On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
> [hidden email]> wrote:
>
>> Hi Shubham,
>> Changing type is not allowed without full reindexing. If you do something
>> like that, Solr will end up with segments with different types for the same
>> field. Remember that segments are immutable and that reindexing some
>> document will be in new segment, but old segment will still be there and at
>> query type Solr will have mismatch between what is stated in schema and
>> what is in segment. In order to change type you have to do full reindexing
>> - create a new collection and reindex all documents.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 22 Oct 2019, at 09:25, Shubham Goswami <[hidden email]>
>> wrote:
>>>
>>> Hello Community
>>>
>>> I have indexed some documents for which solr has taken its type="plongs"
>> by
>>> auto guessing but i am trying to change its type="pint" and re-indexing
>> the
>>> same data with the same id and indexing the data with different id where
>> id
>>> is unique key but it is showing error.
>>>
>>> Can somebody please let me know if it is possible or not, if not possible
>>> then why it is not possible as i am using different id as well ? if
>>> possible then how we could achieve it ?
>>> Any help will be appreciated. Thanks in advance.
>>>
>>> --
>>> *Thanks & Regards*
>>> Shubham Goswami
>>> Enterprise Software Engineer
>>> *HotWax Systems*
>>> *Enterprise open source experts*
>>> cell: +91-7803886288
>>> office: 0731-409-3684
>>> http://www.hotwaxsystems.com
>>
>>
>
> --
> *Thanks & Regards*
> Shubham Goswami
> Enterprise Software Engineer
> *HotWax Systems*
> *Enterprise open source experts*
> cell: +91-7803886288
> office: 0731-409-3684
> http://www.hotwaxsystems.com

Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Shubham Goswami
Hi Emir

As you have mentioned above we cannot change field type after indexing once
and we have to do dull re-indexing again, I tried to change field type from
plong to pint which has implemented class solr.LongPointField and
solr.IntPointField respectively and it was showing error as expected.
        But when i changed field types from pint/plong to any type which
has implemented class solr.TextField, in this case its working fine and i
am able to index the documents after changing its fieldtype with same and
different id.

So i want to know if is there any compatibility with implemented classes ?

Thanks
Shubham

On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
[hidden email]> wrote:

> Hi Shubham,
> No you cannot. What you can do is to use copy field or update request
> processor to store is as some other field and use that in your query and
> ignore the old one that will eventually disappear as the result of segment
> merges.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]>
> wrote:
> >
> > Hi Emir
> >
> > Thanks for the reply, i got your point.
> > But is there any other way to do like one field could have two or more
> > different types defined ?
> > or  if i talk about my previous query, can we index some data for the
> same
> > field with different unique id after replacing the type ?
> >
> > Thanks again
> > Shubham
> >
> > On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
> > [hidden email]> wrote:
> >
> >> Hi Shubham,
> >> Changing type is not allowed without full reindexing. If you do
> something
> >> like that, Solr will end up with segments with different types for the
> same
> >> field. Remember that segments are immutable and that reindexing some
> >> document will be in new segment, but old segment will still be there
> and at
> >> query type Solr will have mismatch between what is stated in schema and
> >> what is in segment. In order to change type you have to do full
> reindexing
> >> - create a new collection and reindex all documents.
> >>
> >> HTH,
> >> Emir
> >> --
> >> Monitoring - Log Management - Alerting - Anomaly Detection
> >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >>
> >>
> >>
> >>> On 22 Oct 2019, at 09:25, Shubham Goswami <[hidden email]>
> >> wrote:
> >>>
> >>> Hello Community
> >>>
> >>> I have indexed some documents for which solr has taken its
> type="plongs"
> >> by
> >>> auto guessing but i am trying to change its type="pint" and re-indexing
> >> the
> >>> same data with the same id and indexing the data with different id
> where
> >> id
> >>> is unique key but it is showing error.
> >>>
> >>> Can somebody please let me know if it is possible or not, if not
> possible
> >>> then why it is not possible as i am using different id as well ? if
> >>> possible then how we could achieve it ?
> >>> Any help will be appreciated. Thanks in advance.
> >>>
> >>> --
> >>> *Thanks & Regards*
> >>> Shubham Goswami
> >>> Enterprise Software Engineer
> >>> *HotWax Systems*
> >>> *Enterprise open source experts*
> >>> cell: +91-7803886288
> >>> office: 0731-409-3684
> >>> http://www.hotwaxsystems.com
> >>
> >>
> >
> > --
> > *Thanks & Regards*
> > Shubham Goswami
> > Enterprise Software Engineer
> > *HotWax Systems*
> > *Enterprise open source experts*
> > cell: +91-7803886288
> > office: 0731-409-3684
> > http://www.hotwaxsystems.com
>
>

--
*Thanks & Regards*
Shubham Goswami
Enterprise Software Engineer
*HotWax Systems*
*Enterprise open source experts*
cell: +91-7803886288
office: 0731-409-3684
http://www.hotwaxsystems.com
Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Jason Gerlowski
Hi Shubbham,

Emir gave you accurate advice - you cannot (safely) change field types
without reindexing.  You may avoid errors for a time, and searches may
even return the results you expect.  But the type-change is still a
ticking time bomb...Solr might try to merge segments down the road or
do some other operation and blow up in unexpected ways.  For more
information on why this is, see the documentation here:
https://lucene.apache.org/solr/guide/8_2/reindexing.html.

Unfortunately there's no way around it.  This, by the way, is why the
community strongly recommends against using schema-guessing mode for
anything other than experimentation.

Best of luck,

Jason

On Tue, Oct 22, 2019 at 7:42 AM Shubham Goswami
<[hidden email]> wrote:

>
> Hi Emir
>
> As you have mentioned above we cannot change field type after indexing once
> and we have to do dull re-indexing again, I tried to change field type from
> plong to pint which has implemented class solr.LongPointField and
> solr.IntPointField respectively and it was showing error as expected.
>         But when i changed field types from pint/plong to any type which
> has implemented class solr.TextField, in this case its working fine and i
> am able to index the documents after changing its fieldtype with same and
> different id.
>
> So i want to know if is there any compatibility with implemented classes ?
>
> Thanks
> Shubham
>
> On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
> [hidden email]> wrote:
>
> > Hi Shubham,
> > No you cannot. What you can do is to use copy field or update request
> > processor to store is as some other field and use that in your query and
> > ignore the old one that will eventually disappear as the result of segment
> > merges.
> >
> > HTH,
> > Emir
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> >
> > > On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]>
> > wrote:
> > >
> > > Hi Emir
> > >
> > > Thanks for the reply, i got your point.
> > > But is there any other way to do like one field could have two or more
> > > different types defined ?
> > > or  if i talk about my previous query, can we index some data for the
> > same
> > > field with different unique id after replacing the type ?
> > >
> > > Thanks again
> > > Shubham
> > >
> > > On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
> > > [hidden email]> wrote:
> > >
> > >> Hi Shubham,
> > >> Changing type is not allowed without full reindexing. If you do
> > something
> > >> like that, Solr will end up with segments with different types for the
> > same
> > >> field. Remember that segments are immutable and that reindexing some
> > >> document will be in new segment, but old segment will still be there
> > and at
> > >> query type Solr will have mismatch between what is stated in schema and
> > >> what is in segment. In order to change type you have to do full
> > reindexing
> > >> - create a new collection and reindex all documents.
> > >>
> > >> HTH,
> > >> Emir
> > >> --
> > >> Monitoring - Log Management - Alerting - Anomaly Detection
> > >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> > >>
> > >>
> > >>
> > >>> On 22 Oct 2019, at 09:25, Shubham Goswami <[hidden email]>
> > >> wrote:
> > >>>
> > >>> Hello Community
> > >>>
> > >>> I have indexed some documents for which solr has taken its
> > type="plongs"
> > >> by
> > >>> auto guessing but i am trying to change its type="pint" and re-indexing
> > >> the
> > >>> same data with the same id and indexing the data with different id
> > where
> > >> id
> > >>> is unique key but it is showing error.
> > >>>
> > >>> Can somebody please let me know if it is possible or not, if not
> > possible
> > >>> then why it is not possible as i am using different id as well ? if
> > >>> possible then how we could achieve it ?
> > >>> Any help will be appreciated. Thanks in advance.
> > >>>
> > >>> --
> > >>> *Thanks & Regards*
> > >>> Shubham Goswami
> > >>> Enterprise Software Engineer
> > >>> *HotWax Systems*
> > >>> *Enterprise open source experts*
> > >>> cell: +91-7803886288
> > >>> office: 0731-409-3684
> > >>> http://www.hotwaxsystems.com
> > >>
> > >>
> > >
> > > --
> > > *Thanks & Regards*
> > > Shubham Goswami
> > > Enterprise Software Engineer
> > > *HotWax Systems*
> > > *Enterprise open source experts*
> > > cell: +91-7803886288
> > > office: 0731-409-3684
> > > http://www.hotwaxsystems.com
> >
> >
>
> --
> *Thanks & Regards*
> Shubham Goswami
> Enterprise Software Engineer
> *HotWax Systems*
> *Enterprise open source experts*
> cell: +91-7803886288
> office: 0731-409-3684
> http://www.hotwaxsystems.com
Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Shubham Goswami
Hi Jason

Thanks for the response.
You are right that re-indexing is required after making any changes to
Schema even i am re-indexing the docs in which i have
changed the fieldtypes, but here Emir is talking about full re-indexing
i.e. deleting the existing/core and creating new one that is
time consuming i think. My doubt is that i am not able to change the type
which has implementation classes like LongPointField/IntPointField to the
type with implementation classes LongPointField/IntPointField.t

         But i am able to change into Text related fields like TextFields
and from TextFields to any other Int/Long type fields.
So i just want to know that what is exact dependency on these classes so
that iam able to change types of some fields ?

Thanks
Shubham

On Tue, Oct 22, 2019 at 6:29 PM Jason Gerlowski <[hidden email]>
wrote:

> Hi Shubbham,
>
> Emir gave you accurate advice - you cannot (safely) change field types
> without reindexing.  You may avoid errors for a time, and searches may
> even return the results you expect.  But the type-change is still a
> ticking time bomb...Solr might try to merge segments down the road or
> do some other operation and blow up in unexpected ways.  For more
> information on why this is, see the documentation here:
> https://lucene.apache.org/solr/guide/8_2/reindexing.html.
>
> Unfortunately there's no way around it.  This, by the way, is why the
> community strongly recommends against using schema-guessing mode for
> anything other than experimentation.
>
> Best of luck,
>
> Jason
>
> On Tue, Oct 22, 2019 at 7:42 AM Shubham Goswami
> <[hidden email]> wrote:
> >
> > Hi Emir
> >
> > As you have mentioned above we cannot change field type after indexing
> once
> > and we have to do dull re-indexing again, I tried to change field type
> from
> > plong to pint which has implemented class solr.LongPointField and
> > solr.IntPointField respectively and it was showing error as expected.
> >         But when i changed field types from pint/plong to any type which
> > has implemented class solr.TextField, in this case its working fine and i
> > am able to index the documents after changing its fieldtype with same and
> > different id.
> >
> > So i want to know if is there any compatibility with implemented classes
> ?
> >
> > Thanks
> > Shubham
> >
> > On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
> > [hidden email]> wrote:
> >
> > > Hi Shubham,
> > > No you cannot. What you can do is to use copy field or update request
> > > processor to store is as some other field and use that in your query
> and
> > > ignore the old one that will eventually disappear as the result of
> segment
> > > merges.
> > >
> > > HTH,
> > > Emir
> > > --
> > > Monitoring - Log Management - Alerting - Anomaly Detection
> > > Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> > >
> > >
> > >
> > > > On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]
> >
> > > wrote:
> > > >
> > > > Hi Emir
> > > >
> > > > Thanks for the reply, i got your point.
> > > > But is there any other way to do like one field could have two or
> more
> > > > different types defined ?
> > > > or  if i talk about my previous query, can we index some data for the
> > > same
> > > > field with different unique id after replacing the type ?
> > > >
> > > > Thanks again
> > > > Shubham
> > > >
> > > > On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
> > > > [hidden email]> wrote:
> > > >
> > > >> Hi Shubham,
> > > >> Changing type is not allowed without full reindexing. If you do
> > > something
> > > >> like that, Solr will end up with segments with different types for
> the
> > > same
> > > >> field. Remember that segments are immutable and that reindexing some
> > > >> document will be in new segment, but old segment will still be there
> > > and at
> > > >> query type Solr will have mismatch between what is stated in schema
> and
> > > >> what is in segment. In order to change type you have to do full
> > > reindexing
> > > >> - create a new collection and reindex all documents.
> > > >>
> > > >> HTH,
> > > >> Emir
> > > >> --
> > > >> Monitoring - Log Management - Alerting - Anomaly Detection
> > > >> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> > > >>
> > > >>
> > > >>
> > > >>> On 22 Oct 2019, at 09:25, Shubham Goswami <
> [hidden email]>
> > > >> wrote:
> > > >>>
> > > >>> Hello Community
> > > >>>
> > > >>> I have indexed some documents for which solr has taken its
> > > type="plongs"
> > > >> by
> > > >>> auto guessing but i am trying to change its type="pint" and
> re-indexing
> > > >> the
> > > >>> same data with the same id and indexing the data with different id
> > > where
> > > >> id
> > > >>> is unique key but it is showing error.
> > > >>>
> > > >>> Can somebody please let me know if it is possible or not, if not
> > > possible
> > > >>> then why it is not possible as i am using different id as well ? if
> > > >>> possible then how we could achieve it ?
> > > >>> Any help will be appreciated. Thanks in advance.
> > > >>>
> > > >>> --
> > > >>> *Thanks & Regards*
> > > >>> Shubham Goswami
> > > >>> Enterprise Software Engineer
> > > >>> *HotWax Systems*
> > > >>> *Enterprise open source experts*
> > > >>> cell: +91-7803886288
> > > >>> office: 0731-409-3684
> > > >>> http://www.hotwaxsystems.com
> > > >>
> > > >>
> > > >
> > > > --
> > > > *Thanks & Regards*
> > > > Shubham Goswami
> > > > Enterprise Software Engineer
> > > > *HotWax Systems*
> > > > *Enterprise open source experts*
> > > > cell: +91-7803886288
> > > > office: 0731-409-3684
> > > > http://www.hotwaxsystems.com
> > >
> > >
> >
> > --
> > *Thanks & Regards*
> > Shubham Goswami
> > Enterprise Software Engineer
> > *HotWax Systems*
> > *Enterprise open source experts*
> > cell: +91-7803886288
> > office: 0731-409-3684
> > http://www.hotwaxsystems.com
>


--
Kind Regards,
Shubham Goswami
Enterprise Software Engineer
mobile: +91 7803886288
email: *[hidden email]*
*www.hotwax.co <http://www.hotwax.co/>*
Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Emir Arnautović
Hi Shubham,
My guess that it might be working for text because it uses o.toString() so there are no runtime errors while in case of others, it has to assume some class so it does class casting. You can check in logs what sort of error happens. But in any case, like Jason pointed out, that is a problem that is just waiting to happen somewhere and the only way to make sure it does not happen is to do full reindexing or to create a new field (with a new name) and stop using the one that is wrong. Different field types are indexed in different structures and with different defaults (e.g. for docValues) and I would not rely on some features working after field type changed.

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 23 Oct 2019, at 08:18, Shubham Goswami <[hidden email]> wrote:
>
> Hi Jason
>
> Thanks for the response.
> You are right that re-indexing is required after making any changes to
> Schema even i am re-indexing the docs in which i have
> changed the fieldtypes, but here Emir is talking about full re-indexing
> i.e. deleting the existing/core and creating new one that is
> time consuming i think. My doubt is that i am not able to change the type
> which has implementation classes like LongPointField/IntPointField to the
> type with implementation classes LongPointField/IntPointField.t
>
>         But i am able to change into Text related fields like TextFields
> and from TextFields to any other Int/Long type fields.
> So i just want to know that what is exact dependency on these classes so
> that iam able to change types of some fields ?
>
> Thanks
> Shubham
>
> On Tue, Oct 22, 2019 at 6:29 PM Jason Gerlowski <[hidden email]>
> wrote:
>
>> Hi Shubbham,
>>
>> Emir gave you accurate advice - you cannot (safely) change field types
>> without reindexing.  You may avoid errors for a time, and searches may
>> even return the results you expect.  But the type-change is still a
>> ticking time bomb...Solr might try to merge segments down the road or
>> do some other operation and blow up in unexpected ways.  For more
>> information on why this is, see the documentation here:
>> https://lucene.apache.org/solr/guide/8_2/reindexing.html.
>>
>> Unfortunately there's no way around it.  This, by the way, is why the
>> community strongly recommends against using schema-guessing mode for
>> anything other than experimentation.
>>
>> Best of luck,
>>
>> Jason
>>
>> On Tue, Oct 22, 2019 at 7:42 AM Shubham Goswami
>> <[hidden email]> wrote:
>>>
>>> Hi Emir
>>>
>>> As you have mentioned above we cannot change field type after indexing
>> once
>>> and we have to do dull re-indexing again, I tried to change field type
>> from
>>> plong to pint which has implemented class solr.LongPointField and
>>> solr.IntPointField respectively and it was showing error as expected.
>>>        But when i changed field types from pint/plong to any type which
>>> has implemented class solr.TextField, in this case its working fine and i
>>> am able to index the documents after changing its fieldtype with same and
>>> different id.
>>>
>>> So i want to know if is there any compatibility with implemented classes
>> ?
>>>
>>> Thanks
>>> Shubham
>>>
>>> On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
>>> [hidden email]> wrote:
>>>
>>>> Hi Shubham,
>>>> No you cannot. What you can do is to use copy field or update request
>>>> processor to store is as some other field and use that in your query
>> and
>>>> ignore the old one that will eventually disappear as the result of
>> segment
>>>> merges.
>>>>
>>>> HTH,
>>>> Emir
>>>> --
>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>> Solr & Elasticsearch Consulting Support Training -
>> http://sematext.com/
>>>>
>>>>
>>>>
>>>>> On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]
>>>
>>>> wrote:
>>>>>
>>>>> Hi Emir
>>>>>
>>>>> Thanks for the reply, i got your point.
>>>>> But is there any other way to do like one field could have two or
>> more
>>>>> different types defined ?
>>>>> or  if i talk about my previous query, can we index some data for the
>>>> same
>>>>> field with different unique id after replacing the type ?
>>>>>
>>>>> Thanks again
>>>>> Shubham
>>>>>
>>>>> On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Hi Shubham,
>>>>>> Changing type is not allowed without full reindexing. If you do
>>>> something
>>>>>> like that, Solr will end up with segments with different types for
>> the
>>>> same
>>>>>> field. Remember that segments are immutable and that reindexing some
>>>>>> document will be in new segment, but old segment will still be there
>>>> and at
>>>>>> query type Solr will have mismatch between what is stated in schema
>> and
>>>>>> what is in segment. In order to change type you have to do full
>>>> reindexing
>>>>>> - create a new collection and reindex all documents.
>>>>>>
>>>>>> HTH,
>>>>>> Emir
>>>>>> --
>>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>>>> Solr & Elasticsearch Consulting Support Training -
>> http://sematext.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 22 Oct 2019, at 09:25, Shubham Goswami <
>> [hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hello Community
>>>>>>>
>>>>>>> I have indexed some documents for which solr has taken its
>>>> type="plongs"
>>>>>> by
>>>>>>> auto guessing but i am trying to change its type="pint" and
>> re-indexing
>>>>>> the
>>>>>>> same data with the same id and indexing the data with different id
>>>> where
>>>>>> id
>>>>>>> is unique key but it is showing error.
>>>>>>>
>>>>>>> Can somebody please let me know if it is possible or not, if not
>>>> possible
>>>>>>> then why it is not possible as i am using different id as well ? if
>>>>>>> possible then how we could achieve it ?
>>>>>>> Any help will be appreciated. Thanks in advance.
>>>>>>>
>>>>>>> --
>>>>>>> *Thanks & Regards*
>>>>>>> Shubham Goswami
>>>>>>> Enterprise Software Engineer
>>>>>>> *HotWax Systems*
>>>>>>> *Enterprise open source experts*
>>>>>>> cell: +91-7803886288
>>>>>>> office: 0731-409-3684
>>>>>>> http://www.hotwaxsystems.com
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>> Shubham Goswami
>>>>> Enterprise Software Engineer
>>>>> *HotWax Systems*
>>>>> *Enterprise open source experts*
>>>>> cell: +91-7803886288
>>>>> office: 0731-409-3684
>>>>> http://www.hotwaxsystems.com
>>>>
>>>>
>>>
>>> --
>>> *Thanks & Regards*
>>> Shubham Goswami
>>> Enterprise Software Engineer
>>> *HotWax Systems*
>>> *Enterprise open source experts*
>>> cell: +91-7803886288
>>> office: 0731-409-3684
>>> http://www.hotwaxsystems.com
>>
>
>
> --
> Kind Regards,
> Shubham Goswami
> Enterprise Software Engineer
> mobile: +91 7803886288
> email: *[hidden email]*
> *www.hotwax.co <http://www.hotwax.co/>*

Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Erick Erickson
Really, just don’t do this. Please. As others have pointed out, it may look like it works, but it won’t. I’ve spent many hours tracking down why clients got weird errors after making changes like this, sometimes weeks later. Or more accurately, if you choose to change field types without reindexing, please don’t ask for others to troubleshoot it when something blows up.

As far as creating a new core, if that takes a significant amount of time relative to re-indexing, then you must be working with a very small index. Those operations should take a couple of minutes, tops.

Best,
Erick

> On Oct 23, 2019, at 5:13 AM, Emir Arnautović <[hidden email]> wrote:
>
> Hi Shubham,
> My guess that it might be working for text because it uses o.toString() so there are no runtime errors while in case of others, it has to assume some class so it does class casting. You can check in logs what sort of error happens. But in any case, like Jason pointed out, that is a problem that is just waiting to happen somewhere and the only way to make sure it does not happen is to do full reindexing or to create a new field (with a new name) and stop using the one that is wrong. Different field types are indexed in different structures and with different defaults (e.g. for docValues) and I would not rely on some features working after field type changed.
>
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 23 Oct 2019, at 08:18, Shubham Goswami <[hidden email]> wrote:
>>
>> Hi Jason
>>
>> Thanks for the response.
>> You are right that re-indexing is required after making any changes to
>> Schema even i am re-indexing the docs in which i have
>> changed the fieldtypes, but here Emir is talking about full re-indexing
>> i.e. deleting the existing/core and creating new one that is
>> time consuming i think. My doubt is that i am not able to change the type
>> which has implementation classes like LongPointField/IntPointField to the
>> type with implementation classes LongPointField/IntPointField.t
>>
>>        But i am able to change into Text related fields like TextFields
>> and from TextFields to any other Int/Long type fields.
>> So i just want to know that what is exact dependency on these classes so
>> that iam able to change types of some fields ?
>>
>> Thanks
>> Shubham
>>
>> On Tue, Oct 22, 2019 at 6:29 PM Jason Gerlowski <[hidden email]>
>> wrote:
>>
>>> Hi Shubbham,
>>>
>>> Emir gave you accurate advice - you cannot (safely) change field types
>>> without reindexing.  You may avoid errors for a time, and searches may
>>> even return the results you expect.  But the type-change is still a
>>> ticking time bomb...Solr might try to merge segments down the road or
>>> do some other operation and blow up in unexpected ways.  For more
>>> information on why this is, see the documentation here:
>>> https://lucene.apache.org/solr/guide/8_2/reindexing.html.
>>>
>>> Unfortunately there's no way around it.  This, by the way, is why the
>>> community strongly recommends against using schema-guessing mode for
>>> anything other than experimentation.
>>>
>>> Best of luck,
>>>
>>> Jason
>>>
>>> On Tue, Oct 22, 2019 at 7:42 AM Shubham Goswami
>>> <[hidden email]> wrote:
>>>>
>>>> Hi Emir
>>>>
>>>> As you have mentioned above we cannot change field type after indexing
>>> once
>>>> and we have to do dull re-indexing again, I tried to change field type
>>> from
>>>> plong to pint which has implemented class solr.LongPointField and
>>>> solr.IntPointField respectively and it was showing error as expected.
>>>>       But when i changed field types from pint/plong to any type which
>>>> has implemented class solr.TextField, in this case its working fine and i
>>>> am able to index the documents after changing its fieldtype with same and
>>>> different id.
>>>>
>>>> So i want to know if is there any compatibility with implemented classes
>>> ?
>>>>
>>>> Thanks
>>>> Shubham
>>>>
>>>> On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
>>>> [hidden email]> wrote:
>>>>
>>>>> Hi Shubham,
>>>>> No you cannot. What you can do is to use copy field or update request
>>>>> processor to store is as some other field and use that in your query
>>> and
>>>>> ignore the old one that will eventually disappear as the result of
>>> segment
>>>>> merges.
>>>>>
>>>>> HTH,
>>>>> Emir
>>>>> --
>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>>> Solr & Elasticsearch Consulting Support Training -
>>> http://sematext.com/
>>>>>
>>>>>
>>>>>
>>>>>> On 22 Oct 2019, at 10:53, Shubham Goswami <[hidden email]
>>>>
>>>>> wrote:
>>>>>>
>>>>>> Hi Emir
>>>>>>
>>>>>> Thanks for the reply, i got your point.
>>>>>> But is there any other way to do like one field could have two or
>>> more
>>>>>> different types defined ?
>>>>>> or  if i talk about my previous query, can we index some data for the
>>>>> same
>>>>>> field with different unique id after replacing the type ?
>>>>>>
>>>>>> Thanks again
>>>>>> Shubham
>>>>>>
>>>>>> On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> Hi Shubham,
>>>>>>> Changing type is not allowed without full reindexing. If you do
>>>>> something
>>>>>>> like that, Solr will end up with segments with different types for
>>> the
>>>>> same
>>>>>>> field. Remember that segments are immutable and that reindexing some
>>>>>>> document will be in new segment, but old segment will still be there
>>>>> and at
>>>>>>> query type Solr will have mismatch between what is stated in schema
>>> and
>>>>>>> what is in segment. In order to change type you have to do full
>>>>> reindexing
>>>>>>> - create a new collection and reindex all documents.
>>>>>>>
>>>>>>> HTH,
>>>>>>> Emir
>>>>>>> --
>>>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>>>>> Solr & Elasticsearch Consulting Support Training -
>>> http://sematext.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 22 Oct 2019, at 09:25, Shubham Goswami <
>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello Community
>>>>>>>>
>>>>>>>> I have indexed some documents for which solr has taken its
>>>>> type="plongs"
>>>>>>> by
>>>>>>>> auto guessing but i am trying to change its type="pint" and
>>> re-indexing
>>>>>>> the
>>>>>>>> same data with the same id and indexing the data with different id
>>>>> where
>>>>>>> id
>>>>>>>> is unique key but it is showing error.
>>>>>>>>
>>>>>>>> Can somebody please let me know if it is possible or not, if not
>>>>> possible
>>>>>>>> then why it is not possible as i am using different id as well ? if
>>>>>>>> possible then how we could achieve it ?
>>>>>>>> Any help will be appreciated. Thanks in advance.
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Thanks & Regards*
>>>>>>>> Shubham Goswami
>>>>>>>> Enterprise Software Engineer
>>>>>>>> *HotWax Systems*
>>>>>>>> *Enterprise open source experts*
>>>>>>>> cell: +91-7803886288
>>>>>>>> office: 0731-409-3684
>>>>>>>> http://www.hotwaxsystems.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>> Shubham Goswami
>>>>>> Enterprise Software Engineer
>>>>>> *HotWax Systems*
>>>>>> *Enterprise open source experts*
>>>>>> cell: +91-7803886288
>>>>>> office: 0731-409-3684
>>>>>> http://www.hotwaxsystems.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>> Shubham Goswami
>>>> Enterprise Software Engineer
>>>> *HotWax Systems*
>>>> *Enterprise open source experts*
>>>> cell: +91-7803886288
>>>> office: 0731-409-3684
>>>> http://www.hotwaxsystems.com
>>>
>>
>>
>> --
>> Kind Regards,
>> Shubham Goswami
>> Enterprise Software Engineer
>> mobile: +91 7803886288
>> email: *[hidden email]*
>> *www.hotwax.co <http://www.hotwax.co/>*
>

Reply | Threaded
Open this post in threaded view
|

Re: Query on changing FieldType

Shubham Goswami
Hello Erick/Emir

Thanks for your valuable suggestions. I will it keep in mind while doing
such operations.

Best,
Shubham

On Wed, Oct 23, 2019 at 5:56 PM Erick Erickson <[hidden email]>
wrote:

> Really, just don’t do this. Please. As others have pointed out, it may
> look like it works, but it won’t. I’ve spent many hours tracking down why
> clients got weird errors after making changes like this, sometimes weeks
> later. Or more accurately, if you choose to change field types without
> reindexing, please don’t ask for others to troubleshoot it when something
> blows up.
>
> As far as creating a new core, if that takes a significant amount of time
> relative to re-indexing, then you must be working with a very small index.
> Those operations should take a couple of minutes, tops.
>
> Best,
> Erick
>
> > On Oct 23, 2019, at 5:13 AM, Emir Arnautović <
> [hidden email]> wrote:
> >
> > Hi Shubham,
> > My guess that it might be working for text because it uses o.toString()
> so there are no runtime errors while in case of others, it has to assume
> some class so it does class casting. You can check in logs what sort of
> error happens. But in any case, like Jason pointed out, that is a problem
> that is just waiting to happen somewhere and the only way to make sure it
> does not happen is to do full reindexing or to create a new field (with a
> new name) and stop using the one that is wrong. Different field types are
> indexed in different structures and with different defaults (e.g. for
> docValues) and I would not rely on some features working after field type
> changed.
> >
> > Thanks,
> > Emir
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> >
> >> On 23 Oct 2019, at 08:18, Shubham Goswami <[hidden email]>
> wrote:
> >>
> >> Hi Jason
> >>
> >> Thanks for the response.
> >> You are right that re-indexing is required after making any changes to
> >> Schema even i am re-indexing the docs in which i have
> >> changed the fieldtypes, but here Emir is talking about full re-indexing
> >> i.e. deleting the existing/core and creating new one that is
> >> time consuming i think. My doubt is that i am not able to change the
> type
> >> which has implementation classes like LongPointField/IntPointField to
> the
> >> type with implementation classes LongPointField/IntPointField.t
> >>
> >>        But i am able to change into Text related fields like TextFields
> >> and from TextFields to any other Int/Long type fields.
> >> So i just want to know that what is exact dependency on these classes so
> >> that iam able to change types of some fields ?
> >>
> >> Thanks
> >> Shubham
> >>
> >> On Tue, Oct 22, 2019 at 6:29 PM Jason Gerlowski <[hidden email]>
> >> wrote:
> >>
> >>> Hi Shubbham,
> >>>
> >>> Emir gave you accurate advice - you cannot (safely) change field types
> >>> without reindexing.  You may avoid errors for a time, and searches may
> >>> even return the results you expect.  But the type-change is still a
> >>> ticking time bomb...Solr might try to merge segments down the road or
> >>> do some other operation and blow up in unexpected ways.  For more
> >>> information on why this is, see the documentation here:
> >>> https://lucene.apache.org/solr/guide/8_2/reindexing.html.
> >>>
> >>> Unfortunately there's no way around it.  This, by the way, is why the
> >>> community strongly recommends against using schema-guessing mode for
> >>> anything other than experimentation.
> >>>
> >>> Best of luck,
> >>>
> >>> Jason
> >>>
> >>> On Tue, Oct 22, 2019 at 7:42 AM Shubham Goswami
> >>> <[hidden email]> wrote:
> >>>>
> >>>> Hi Emir
> >>>>
> >>>> As you have mentioned above we cannot change field type after indexing
> >>> once
> >>>> and we have to do dull re-indexing again, I tried to change field type
> >>> from
> >>>> plong to pint which has implemented class solr.LongPointField and
> >>>> solr.IntPointField respectively and it was showing error as expected.
> >>>>       But when i changed field types from pint/plong to any type which
> >>>> has implemented class solr.TextField, in this case its working fine
> and i
> >>>> am able to index the documents after changing its fieldtype with same
> and
> >>>> different id.
> >>>>
> >>>> So i want to know if is there any compatibility with implemented
> classes
> >>> ?
> >>>>
> >>>> Thanks
> >>>> Shubham
> >>>>
> >>>> On Tue, Oct 22, 2019 at 2:46 PM Emir Arnautović <
> >>>> [hidden email]> wrote:
> >>>>
> >>>>> Hi Shubham,
> >>>>> No you cannot. What you can do is to use copy field or update request
> >>>>> processor to store is as some other field and use that in your query
> >>> and
> >>>>> ignore the old one that will eventually disappear as the result of
> >>> segment
> >>>>> merges.
> >>>>>
> >>>>> HTH,
> >>>>> Emir
> >>>>> --
> >>>>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>>>> Solr & Elasticsearch Consulting Support Training -
> >>> http://sematext.com/
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 22 Oct 2019, at 10:53, Shubham Goswami <
> [hidden email]
> >>>>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Emir
> >>>>>>
> >>>>>> Thanks for the reply, i got your point.
> >>>>>> But is there any other way to do like one field could have two or
> >>> more
> >>>>>> different types defined ?
> >>>>>> or  if i talk about my previous query, can we index some data for
> the
> >>>>> same
> >>>>>> field with different unique id after replacing the type ?
> >>>>>>
> >>>>>> Thanks again
> >>>>>> Shubham
> >>>>>>
> >>>>>> On Tue, Oct 22, 2019 at 1:23 PM Emir Arnautović <
> >>>>>> [hidden email]> wrote:
> >>>>>>
> >>>>>>> Hi Shubham,
> >>>>>>> Changing type is not allowed without full reindexing. If you do
> >>>>> something
> >>>>>>> like that, Solr will end up with segments with different types for
> >>> the
> >>>>> same
> >>>>>>> field. Remember that segments are immutable and that reindexing
> some
> >>>>>>> document will be in new segment, but old segment will still be
> there
> >>>>> and at
> >>>>>>> query type Solr will have mismatch between what is stated in schema
> >>> and
> >>>>>>> what is in segment. In order to change type you have to do full
> >>>>> reindexing
> >>>>>>> - create a new collection and reindex all documents.
> >>>>>>>
> >>>>>>> HTH,
> >>>>>>> Emir
> >>>>>>> --
> >>>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>>>>>> Solr & Elasticsearch Consulting Support Training -
> >>> http://sematext.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 22 Oct 2019, at 09:25, Shubham Goswami <
> >>> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hello Community
> >>>>>>>>
> >>>>>>>> I have indexed some documents for which solr has taken its
> >>>>> type="plongs"
> >>>>>>> by
> >>>>>>>> auto guessing but i am trying to change its type="pint" and
> >>> re-indexing
> >>>>>>> the
> >>>>>>>> same data with the same id and indexing the data with different id
> >>>>> where
> >>>>>>> id
> >>>>>>>> is unique key but it is showing error.
> >>>>>>>>
> >>>>>>>> Can somebody please let me know if it is possible or not, if not
> >>>>> possible
> >>>>>>>> then why it is not possible as i am using different id as well ?
> if
> >>>>>>>> possible then how we could achieve it ?
> >>>>>>>> Any help will be appreciated. Thanks in advance.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Thanks & Regards*
> >>>>>>>> Shubham Goswami
> >>>>>>>> Enterprise Software Engineer
> >>>>>>>> *HotWax Systems*
> >>>>>>>> *Enterprise open source experts*
> >>>>>>>> cell: +91-7803886288
> >>>>>>>> office: 0731-409-3684
> >>>>>>>> http://www.hotwaxsystems.com
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Thanks & Regards*
> >>>>>> Shubham Goswami
> >>>>>> Enterprise Software Engineer
> >>>>>> *HotWax Systems*
> >>>>>> *Enterprise open source experts*
> >>>>>> cell: +91-7803886288
> >>>>>> office: 0731-409-3684
> >>>>>> http://www.hotwaxsystems.com
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> *Thanks & Regards*
> >>>> Shubham Goswami
> >>>> Enterprise Software Engineer
> >>>> *HotWax Systems*
> >>>> *Enterprise open source experts*
> >>>> cell: +91-7803886288
> >>>> office: 0731-409-3684
> >>>> http://www.hotwaxsystems.com
> >>>
> >>
> >>
> >> --
> >> Kind Regards,
> >> Shubham Goswami
> >> Enterprise Software Engineer
> >> mobile: +91 7803886288
> >> email: *[hidden email]*
> >> *www.hotwax.co <http://www.hotwax.co/>*
> >
>
>

--
Kind Regards,
Shubham Goswami
Enterprise Software Engineer
mobile: +91 7803886288
email: *[hidden email]*
*www.hotwax.co <http://www.hotwax.co/>*