number of documents exceed 2147483519

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

number of documents exceed 2147483519

interma
Hi
I'm using solr-cloud (ver 6.6), got an error:
org.apache.solr.common.SolrException: Exception writing document id (null) to the index; possible analysis error: number of documents in the index cannot exceed 2147483519

After googled it, I know the number is exceed one solr shard limit.
The collection has 64 shards, so I think total limit is 20B*64=128B

My question is:
I don't want to recreate index (then split to more shards) and also don't want to delete docs.
Can I using the "SPLITSHARD" api to fix this issue?
https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard

After split each shard (now 128 shards), I think the total limit is increasing to 256B, right?

Thanks.


Collections API | Apache Solr Reference Guide 6.6<https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard>
The Collections API is used to enable you to create, remove, or reload collections, but in the context of SolrCloud you can also use it to create collections with a specific number of shards and replicas.
lucene.apache.org


Reply | Threaded
Open this post in threaded view
|

Re: number of documents exceed 2147483519

interma
I was trying "SPLITSHARD" in my test env and encountered a strange behavior:
I created a 1M doc collection and it contained 8 shards, then split shard1, after split:

  *   SPLITSHARD returned success.
  *   looks good
     *   shard1 (became inactive) -> shard1_0 and shard1_1
     *   range 80000000-9fffffff -> 80000000-8fffffff and 90000000-9fffffff
  *   but
     *   the doc number is increasing: docnum of shard1_0 (70702) + docnum of shard1_1(67980) > docnum of shard1(124818)

I tested many times and this issue happened every time. Why?

Thanks.








________________________________
From: Hongxu Ma <[hidden email]>
Sent: Monday, March 16, 2020 16:46
To: [hidden email] <[hidden email]>
Subject: number of documents exceed 2147483519

Hi
I'm using solr-cloud (ver 6.6), got an error:
org.apache.solr.common.SolrException: Exception writing document id (null) to the index; possible analysis error: number of documents in the index cannot exceed 2147483519

After googled it, I know the number is exceed one solr shard limit.
The collection has 64 shards, so I think total limit is 20B*64=128B

My question is:
I don't want to recreate index (then split to more shards) and also don't want to delete docs.
Can I using the "SPLITSHARD" api to fix this issue?
https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard

After split each shard (now 128 shards), I think the total limit is increasing to 256B, right?

Thanks.


Collections API | Apache Solr Reference Guide 6.6<https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-splitshard>
The Collections API is used to enable you to create, remove, or reload collections, but in the context of SolrCloud you can also use it to create collections with a specific number of shards and replicas.
lucene.apache.org