dynamicField Scaling

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

dynamicField Scaling

mark angelillo
Hello,

I've got a Solr index running and I want to use a dynamicField to  
store n different sorting fields. The field that is used to actually  
sort the results will be determined by the application that is  
querying the index.

I'm wondering if anyone has done something similar to this, or if  
anyone has an idea of how Solr will perform as the number n of  
sorting fields grows larger. Is there a way to make sure this doesn't  
start to slow the index down? Is there any information out there  
about the number of dynamicFields that can be declared in this way  
before the entire index suffers? Is there such a limit?

(I'm assuming the number of documents in the index will eventually be  
around 500k -- perhaps more in the future.)

TIA,
Mark Angelillo
Reply | Threaded
Open this post in threaded view
|

Re: dynamicField Scaling

Mike Klaas
On 3/7/07, mark angelillo <[hidden email]> wrote:

> Hello,
>
> I've got a Solr index running and I want to use a dynamicField to
> store n different sorting fields. The field that is used to actually
> sort the results will be determined by the application that is
> querying the index.
>
> I'm wondering if anyone has done something similar to this, or if
> anyone has an idea of how Solr will perform as the number n of
> sorting fields grows larger. Is there a way to make sure this doesn't
> start to slow the index down? Is there any information out there
> about the number of dynamicFields that can be declared in this way
> before the entire index suffers? Is there such a limit?

It's not realy about the number of dynamic fields.  The key variable
is the number of sort fields.  To sort efficiently, solr needs to
maintain a cache of field values.  This consumes memory per-field on
the order of

D x S + U

where D is the document count, S is the the size of the data type (eg.
4bytes for ints, 8 bytes for doubles, 4/8 bytes for anything else
[pointers]), and U is the cumulative size of the unique field values
(if sorting on a non-primitive type, like Strings).

If you have sufficient memory to store this data for each field you
are sorting on, you shouldn't have any problems.

best,
-Mike
Reply | Threaded
Open this post in threaded view
|

Re: dynamicField Scaling

mark angelillo
On Mar 7, 2007, at 2:17 PM, Mike Klaas wrote:

> On 3/7/07, mark angelillo <[hidden email]> wrote:
>> Hello,
>>
>> I've got a Solr index running and I want to use a dynamicField to
>> store n different sorting fields. The field that is used to actually
>> sort the results will be determined by the application that is
>> querying the index.
>>
>> I'm wondering if anyone has done something similar to this, or if
>> anyone has an idea of how Solr will perform as the number n of
>> sorting fields grows larger. Is there a way to make sure this doesn't
>> start to slow the index down? Is there any information out there
>> about the number of dynamicFields that can be declared in this way
>> before the entire index suffers? Is there such a limit?
>
> It's not realy about the number of dynamic fields.  The key variable
> is the number of sort fields.  To sort efficiently, solr needs to
> maintain a cache of field values.  This consumes memory per-field on
> the order of
>
> D x S + U
>
> where D is the document count, S is the the size of the data type (eg.
> 4bytes for ints, 8 bytes for doubles, 4/8 bytes for anything else
> [pointers]), and U is the cumulative size of the unique field values
> (if sorting on a non-primitive type, like Strings).
>
> If you have sufficient memory to store this data for each field you
> are sorting on, you shouldn't have any problems.
>
> best,
> -Mike


Okay, makes sense.

Thanks,
Mark