Help for importing large data (approx. 8GB) from old solr version to new solr version

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Help for importing large data (approx. 8GB) from old solr version to new solr version

Ken Walker
Hello,

We are using solr version 8.2.0 in our production server.

We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
version but we have faced out of memory error while importing data and
then we have extended memory in our server and then again start
importing process but it has work too slowy for 8GB data ( it has
taken more than 2 days for importing data from solr 8.2.0 version to
solr 8.3.1 version).

Could you please help me how we can do it fast for importing 8GB data
from old solr version to new solr version?

We are using below command for importing data from one solr version to
another solr version
$ curl 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'

Thanks in advance!
- Ken
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Shawn Heisey-2
On 12/23/2019 9:58 PM, Ken Walker wrote:
> We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> version but we have faced out of memory error while importing data and
> then we have extended memory in our server and then again start
> importing process but it has work too slowy for 8GB data ( it has
> taken more than 2 days for importing data from solr 8.2.0 version to
> solr 8.3.1 version).
>
> Could you please help me how we can do it fast for importing 8GB data
> from old solr version to new solr version?

Why not just copy the index and use it directly rather than importing
it?  Solr 8.x can directly use indexes built by versions back to 7.0.0.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Ken Walker
Hello Shawn,

Thanks for your reply!

Actually we don't know how its works ( just copy the index ) so could
you please give us some reference urls or any steps for it?

Thanks in advance
- Ken

On Tue, Dec 24, 2019 at 11:56 AM Shawn Heisey <[hidden email]> wrote:

>
> On 12/23/2019 9:58 PM, Ken Walker wrote:
> > We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> > version but we have faced out of memory error while importing data and
> > then we have extended memory in our server and then again start
> > importing process but it has work too slowy for 8GB data ( it has
> > taken more than 2 days for importing data from solr 8.2.0 version to
> > solr 8.3.1 version).
> >
> > Could you please help me how we can do it fast for importing 8GB data
> > from old solr version to new solr version?
>
> Why not just copy the index and use it directly rather than importing
> it?  Solr 8.x can directly use indexes built by versions back to 7.0.0.
>
> Thanks,
> Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Jörn Franke
In reply to this post by Ken Walker
It seems that you got this handed over with little documentation. You have to explore what the import handler does. This is a custom configuration that you need to check how it works.

Then as already said. You can simply install another version of Solr if you are within a Solr major version 8.x in Linux is simply a symbolic link pointing from one Solr version to the other. In this way you can easily switch back as well.

Finally, check your memory consumption. Normally heap is significant smaller then the total available memory as the non-heap memory is used by Solr for caching.

If you have 8g mb of heap I would expect that the total amount of memory available is more than 32 gb.
As always it depends, but maybe you can give more details on no of cores, heap memory, total memory and if other processes than Solr run on the machine.

> Am 24.12.2019 um 05:59 schrieb Ken Walker <[hidden email]>:
>
> Hello,
>
> We are using solr version 8.2.0 in our production server.
>
> We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> version but we have faced out of memory error while importing data and
> then we have extended memory in our server and then again start
> importing process but it has work too slowy for 8GB data ( it has
> taken more than 2 days for importing data from solr 8.2.0 version to
> solr 8.3.1 version).
>
> Could you please help me how we can do it fast for importing 8GB data
> from old solr version to new solr version?
>
> We are using below command for importing data from one solr version to
> another solr version
> $ curl 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
>
> Thanks in advance!
> - Ken
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Ken Walker
Hello Jörn,

Thanks for your reply!

As per Shawn "Why not just copy the index and use it directly rather
than importing it?  Solr 8.x can directly use indexes built by
versions back to 7.0.0." in previous mail comment.

Is it possible and how we can do that ?

Thanks in advance
- Ken

On Tue, Dec 24, 2019 at 3:26 PM Jörn Franke <[hidden email]> wrote:

>
> It seems that you got this handed over with little documentation. You have to explore what the import handler does. This is a custom configuration that you need to check how it works.
>
> Then as already said. You can simply install another version of Solr if you are within a Solr major version 8.x in Linux is simply a symbolic link pointing from one Solr version to the other. In this way you can easily switch back as well.
>
> Finally, check your memory consumption. Normally heap is significant smaller then the total available memory as the non-heap memory is used by Solr for caching.
>
> If you have 8g mb of heap I would expect that the total amount of memory available is more than 32 gb.
> As always it depends, but maybe you can give more details on no of cores, heap memory, total memory and if other processes than Solr run on the machine.
>
> > Am 24.12.2019 um 05:59 schrieb Ken Walker <[hidden email]>:
> >
> > Hello,
> >
> > We are using solr version 8.2.0 in our production server.
> >
> > We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> > version but we have faced out of memory error while importing data and
> > then we have extended memory in our server and then again start
> > importing process but it has work too slowy for 8GB data ( it has
> > taken more than 2 days for importing data from solr 8.2.0 version to
> > solr 8.3.1 version).
> >
> > Could you please help me how we can do it fast for importing 8GB data
> > from old solr version to new solr version?
> >
> > We are using below command for importing data from one solr version to
> > another solr version
> > $ curl 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
> >
> > Thanks in advance!
> > - Ken
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Erick Erickson
Here’s the very simplest way:
1> shut down your 8.2 Solr instance
2> install your 8.3.1 instance on the same machine
3> when you start your 8.3.1 instance, specify the environment variable SOLR_HOME to point to the same one you used in 8.2

If you don’t know what SOLR_HOME used to point to, bring up your 8.2 instance first and look at the admin UI, your environment variables will point there.

NOTE: If yo do it this way, you ma _NOT_ have both 8.2 and 8.3.1 running the same time.

Best,
Erick

> On Dec 24, 2019, at 5:54 AM, Ken Walker <[hidden email]> wrote:
>
> Hello Jörn,
>
> Thanks for your reply!
>
> As per Shawn "Why not just copy the index and use it directly rather
> than importing it?  Solr 8.x can directly use indexes built by
> versions back to 7.0.0." in previous mail comment.
>
> Is it possible and how we can do that ?
>
> Thanks in advance
> - Ken
>
> On Tue, Dec 24, 2019 at 3:26 PM Jörn Franke <[hidden email]> wrote:
>>
>> It seems that you got this handed over with little documentation. You have to explore what the import handler does. This is a custom configuration that you need to check how it works.
>>
>> Then as already said. You can simply install another version of Solr if you are within a Solr major version 8.x in Linux is simply a symbolic link pointing from one Solr version to the other. In this way you can easily switch back as well.
>>
>> Finally, check your memory consumption. Normally heap is significant smaller then the total available memory as the non-heap memory is used by Solr for caching.
>>
>> If you have 8g mb of heap I would expect that the total amount of memory available is more than 32 gb.
>> As always it depends, but maybe you can give more details on no of cores, heap memory, total memory and if other processes than Solr run on the machine.
>>
>>> Am 24.12.2019 um 05:59 schrieb Ken Walker <[hidden email]>:
>>>
>>> Hello,
>>>
>>> We are using solr version 8.2.0 in our production server.
>>>
>>> We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
>>> version but we have faced out of memory error while importing data and
>>> then we have extended memory in our server and then again start
>>> importing process but it has work too slowy for 8GB data ( it has
>>> taken more than 2 days for importing data from solr 8.2.0 version to
>>> solr 8.3.1 version).
>>>
>>> Could you please help me how we can do it fast for importing 8GB data
>>> from old solr version to new solr version?
>>>
>>> We are using below command for importing data from one solr version to
>>> another solr version
>>> $ curl 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
>>>
>>> Thanks in advance!
>>> - Ken

Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Ken Walker
Hello Erick,

Thanks for your reply!

You mean that, we should follow below steps right?
Here is the data directory path :
solr/solr-8.2.0/server/solr/product/item_core/data

STEPS :-
1. Stop old solr-8.2.0 server
2. Copy data directory (from old solr version to new solr version)
copy solr/solr-8.2.1/server/solr/product/item_core/data to
solr/solr-8.3.1/server/solr/product/item_core/data
3. Start new solr version solr-8.3.1

Is it correct way to copy just index only from old to new solr version?
Is it lose any data or anything break in new solr version ?

Thanks in advance!
-Ken
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Erick Erickson
Should work. At any rate, just try it. Since all you’re doing is copying data, even if the new installation doesn’t work you still have the original.

> On Dec 25, 2019, at 1:35 AM, Ken Walker <[hidden email]> wrote:
>
> Hello Erick,
>
> Thanks for your reply!
>
> You mean that, we should follow below steps right?
> Here is the data directory path :
> solr/solr-8.2.0/server/solr/product/item_core/data
>
> STEPS :-
> 1. Stop old solr-8.2.0 server
> 2. Copy data directory (from old solr version to new solr version)
> copy solr/solr-8.2.1/server/solr/product/item_core/data to
> solr/solr-8.3.1/server/solr/product/item_core/data
> 3. Start new solr version solr-8.3.1
>
> Is it correct way to copy just index only from old to new solr version?
> Is it lose any data or anything break in new solr version ?
>
> Thanks in advance!
> -Ken

Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

dhastings
Exactly. Although I’m a bit curious why your going a .1 version up, I always wait until an x2, so I won’t be upgrading until 9.3

> On Dec 25, 2019, at 9:45 AM, Erick Erickson <[hidden email]> wrote:
>
> Should work. At any rate, just try it. Since all you’re doing is copying data, even if the new installation doesn’t work you still have the original.
>
>> On Dec 25, 2019, at 1:35 AM, Ken Walker <[hidden email]> wrote:
>>
>> Hello Erick,
>>
>> Thanks for your reply!
>>
>> You mean that, we should follow below steps right?
>> Here is the data directory path :
>> solr/solr-8.2.0/server/solr/product/item_core/data
>>
>> STEPS :-
>> 1. Stop old solr-8.2.0 server
>> 2. Copy data directory (from old solr version to new solr version)
>> copy solr/solr-8.2.1/server/solr/product/item_core/data to
>> solr/solr-8.3.1/server/solr/product/item_core/data
>> 3. Start new solr version solr-8.3.1
>>
>> Is it correct way to copy just index only from old to new solr version?
>> Is it lose any data or anything break in new solr version ?
>>
>> Thanks in advance!
>> -Ken
>
Reply | Threaded
Open this post in threaded view
|

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

Paras Lehana
Hi Ken,

I also recommend at least reading if not following "Taking Solr to
Production":
https://lucene.apache.org/solr/guide/8_4/taking-solr-to-production.html.

Following this cleared my doubts regarding upgradation and core referencing
while made upgradation very easy and fast.

While starting Solr, you can also define Solr Home (where your older core
lives) by using -s option.



On Wed, 25 Dec 2019 at 21:44, David Hastings <[hidden email]> wrote:

> Exactly. Although I’m a bit curious why your going a .1 version up, I
> always wait until an x2, so I won’t be upgrading until 9.3
>
> > On Dec 25, 2019, at 9:45 AM, Erick Erickson <[hidden email]>
> wrote:
> >
> > Should work. At any rate, just try it. Since all you’re doing is
> copying data, even if the new installation doesn’t work you still have the
> original.
> >
> >> On Dec 25, 2019, at 1:35 AM, Ken Walker <[hidden email]> wrote:
> >>
> >> Hello Erick,
> >>
> >> Thanks for your reply!
> >>
> >> You mean that, we should follow below steps right?
> >> Here is the data directory path :
> >> solr/solr-8.2.0/server/solr/product/item_core/data
> >>
> >> STEPS :-
> >> 1. Stop old solr-8.2.0 server
> >> 2. Copy data directory (from old solr version to new solr version)
> >> copy solr/solr-8.2.1/server/solr/product/item_core/data to
> >> solr/solr-8.3.1/server/solr/product/item_core/data
> >> 3. Start new solr version solr-8.3.1
> >>
> >> Is it correct way to copy just index only from old to new solr version?
> >> Is it lose any data or anything break in new solr version ?
> >>
> >> Thanks in advance!
> >> -Ken
> >
>


--
--
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
*
*

 <https://www.facebook.com/IndiaMART/videos/578196442936091/>