Maximum number of fields in a solrCloud collection config

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Maximum number of fields in a solrCloud collection config

Sharif Shahriar
This post was updated on .
Is there any limitation on how many field can be added in a solrcloud
collection configset?
After adding 13,500 fields when I want to add new fields, it shows
-"Error persisting managed schema at /configs/*/managed-schema"
-"zkClient has disconnected"

Thank you,
Sharif



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Number of fields in a solrCloud collection config

Shawn Heisey-2
On 7/11/2018 2:05 AM, Sharif Shahriar wrote:
> Is there any limitation on how many field can be added in a solrcloud
> collection configset?
> After adding 24,520 fields when I wan't to add new fields, it shows
> -"Error persisting managed schema at /configs/*/managed-schema"
> -"zkClient has disconnected"

There is no hard limit on the number of fields.  It is likely that you
are running into something else.   Without detailed logs it's difficult
to know for sure what the problem is, but I do have one idea:

The most likely limit that I can think of that you'd be running into is
the limit on the size of an individual entry in the ZooKeeper database
-- your schema file is getting too big.  By default this limit is right
around one megabyte.  Changing it is possible, but it must be done on
all ZK servers and every single ZK client (which includes Solr).

https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#Experimental+Options%2FFeatures

If that's not the problem, then we will need to see your logs to figure
out what the problem might be.  To properly interpret the log, the exact
Solr version must be known.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Number of fields in a solrCloud collection config

Erick Erickson
As Shawn says, there's no hard limit, but having that many fields
almost always indicates a flawed design.

I'd pretty strongly suggest that you reconsider your design with an
eye towards reducing the field count to
something less than, say, 1,000. That number is fairly arbitrary, and
ideally I'd want it even smaller
but it's not a bad target.

And do note that if you elect to do this, the meta-data will still be
hanging around and you must
create a new collection and re-index from scratch to purge it.

Best,
Erick

On Wed, Jul 11, 2018 at 6:55 AM, Shawn Heisey <[hidden email]> wrote:

> On 7/11/2018 2:05 AM, Sharif Shahriar wrote:
>>
>> Is there any limitation on how many field can be added in a solrcloud
>> collection configset?
>> After adding 24,520 fields when I wan't to add new fields, it shows
>> -"Error persisting managed schema at /configs/*/managed-schema"
>> -"zkClient has disconnected"
>
>
> There is no hard limit on the number of fields.  It is likely that you are
> running into something else.   Without detailed logs it's difficult to know
> for sure what the problem is, but I do have one idea:
>
> The most likely limit that I can think of that you'd be running into is the
> limit on the size of an individual entry in the ZooKeeper database -- your
> schema file is getting too big.  By default this limit is right around one
> megabyte.  Changing it is possible, but it must be done on all ZK servers
> and every single ZK client (which includes Solr).
>
> https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#Experimental+Options%2FFeatures
>
> If that's not the problem, then we will need to see your logs to figure out
> what the problem might be.  To properly interpret the log, the exact Solr
> version must be known.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Number of fields in a solrCloud collection config

Sharif Shahriar
In reply to this post by Shawn Heisey-2
Thanks a lot Shawn for your reply. I'm using SolrCloud v7.3 and using
schema-less approach. I add documents and new fields automatically added in
managed-schema files. I've tried it several times and it stops at around
13,500 fields.
If I try to add fields using SolrCloud API I can add them, but cannot add
using schemaless approach documents adding.

Regards,
Sharif
 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Number of fields in a solrCloud collection config

Sharif Shahriar
In reply to this post by Erick Erickson
Hi Erick,
Thank you for your suggestions.

Regards,
Sharif



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html