too often delta imports performance effect

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

too often delta imports performance effect

adeelmahmood
we are trying to setup solr for a website where data gets updated pretty frequently and I want to have those changes reflected in solr indexes sooner than nighly delta-imports .. so I am thinking we will probably want to set it up to have delta imports running every 15 mins or so .. and solr search will obviously be in use while this is going on .. first of all does solr works well with adding new data or updating existing data while people are doing searches in it
secondly are these delta imports are gonna cause any significant performance degradation in solr search
any help is appreciated
Reply | Threaded
Open this post in threaded view
|

Re: too often delta imports performance effect

Jan Høydahl / Cominvent
Hi,

This all depends on actual volumes, HW, architecture etc.
What exactly is "pretty frequently", how many document updates/adds per 15 minutes?

Solr is designed to be able to do indexing and search in parallel, so you don't need to fear this, unless you are already pushing the limits of what your setup can handle. The best way to go is to start out and then optimize when you see bottlenecks.

Here is a pointer to Wiki about indexing performance:
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed

--
Jan Høydahl  - search architect
Cominvent AS - www.cominvent.com

On 14. feb. 2010, at 23.56, adeelmahmood wrote:

>
> we are trying to setup solr for a website where data gets updated pretty
> frequently and I want to have those changes reflected in solr indexes sooner
> than nighly delta-imports .. so I am thinking we will probably want to set
> it up to have delta imports running every 15 mins or so .. and solr search
> will obviously be in use while this is going on .. first of all does solr
> works well with adding new data or updating existing data while people are
> doing searches in it
> secondly are these delta imports are gonna cause any significant performance
> degradation in solr search
> any help is appreciated
> --
> View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: too often delta imports performance effect

adeelmahmood
thank you .. that helps .. actually its not that many updates .. close to 10 fields probably and may be 50 doc updates per 15 .. so i am assuming that by handling indexing and searching in parallel you mean that if its updating some data .. it will continue to show old data until new data has been finalized(committed) or something like that ??

Jan Høydahl / Cominvent wrote
Hi,

This all depends on actual volumes, HW, architecture etc.
What exactly is "pretty frequently", how many document updates/adds per 15 minutes?

Solr is designed to be able to do indexing and search in parallel, so you don't need to fear this, unless you are already pushing the limits of what your setup can handle. The best way to go is to start out and then optimize when you see bottlenecks.

Here is a pointer to Wiki about indexing performance:
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed

--
Jan Høydahl  - search architect
Cominvent AS - www.cominvent.com

On 14. feb. 2010, at 23.56, adeelmahmood wrote:

>
> we are trying to setup solr for a website where data gets updated pretty
> frequently and I want to have those changes reflected in solr indexes sooner
> than nighly delta-imports .. so I am thinking we will probably want to set
> it up to have delta imports running every 15 mins or so .. and solr search
> will obviously be in use while this is going on .. first of all does solr
> works well with adding new data or updating existing data while people are
> doing searches in it
> secondly are these delta imports are gonna cause any significant performance
> degradation in solr search
> any help is appreciated
> --
> View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: too often delta imports performance effect

Nick Jenkin-2
Yes, the old data will show until there has been a commit executed. 50
docs isn't many so you should be fine
-Nick

On Mon, Feb 15, 2010 at 11:41 AM, adeelmahmood <[hidden email]> wrote:

>
> thank you .. that helps .. actually its not that many updates .. close to 10
> fields probably and may be 50 doc updates per 15 .. so i am assuming that by
> handling indexing and searching in parallel you mean that if its updating
> some data .. it will continue to show old data until new data has been
> finalized(committed) or something like that ??
>
>
> Jan Høydahl / Cominvent wrote:
>>
>> Hi,
>>
>> This all depends on actual volumes, HW, architecture etc.
>> What exactly is "pretty frequently", how many document updates/adds per 15
>> minutes?
>>
>> Solr is designed to be able to do indexing and search in parallel, so you
>> don't need to fear this, unless you are already pushing the limits of what
>> your setup can handle. The best way to go is to start out and then
>> optimize when you see bottlenecks.
>>
>> Here is a pointer to Wiki about indexing performance:
>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>>
>> --
>> Jan Høydahl  - search architect
>> Cominvent AS - www.cominvent.com
>>
>> On 14. feb. 2010, at 23.56, adeelmahmood wrote:
>>
>>>
>>> we are trying to setup solr for a website where data gets updated pretty
>>> frequently and I want to have those changes reflected in solr indexes
>>> sooner
>>> than nighly delta-imports .. so I am thinking we will probably want to
>>> set
>>> it up to have delta imports running every 15 mins or so .. and solr
>>> search
>>> will obviously be in use while this is going on .. first of all does solr
>>> works well with adding new data or updating existing data while people
>>> are
>>> doing searches in it
>>> secondly are these delta imports are gonna cause any significant
>>> performance
>>> degradation in solr search
>>> any help is appreciated
>>> --
>>> View this message in context:
>>> http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27588472.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>