Re: Solr Size Limitation upto 32 kb limitation

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Solr Size Limitation upto 32 kb limitation

Bernd Fehling
Hi,

assuming you have a fieldType for "text_general" defined in your schema, change from:

<field name="FileContent" type="text_general" indexed="true" stored="true" />

to:

<field name="FileContent" type="text_general" indexed="true" stored="true" multiValued="true" />


Regards,
Bernd


Am 18.01.19 um 11:51 schrieb Kranthi Kumar K:

> Hi team,
>
> Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I've tried the suggested one's and still we are unable to import files having            size  >32 kb, it is displaying same error.
>
> Below link has the suggested solutions. Please have a look once.
>
> http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html
>
>
>    1.  As per Erick Erickson, I've changed the string type to Text type based and still the issue occurs .
>
> I've changed from :
>
>
>
> <field name="FileContent" type="string_rev" indexed="true" stored="true" />
>
>
>
> Changed to:
>
>
>
> <field name="FileContent" type="text" indexed="true" stored="true" />
>
>
>
> If we do so, it is showing error in the log, please find the error in the attachment.
>
>
>
> If I change to:
>
>
>
> <field name="FileContent" type="text_general" indexed="true" stored="true" />
>
>
>
> It is not showing any error , but the issue still exists.
>
>
>
>    1.  As per Jan Hoydahl, I have gone through the link that you have provided and checked 'requestParsers' tag in solrconfig.xml,
>
>
>
> RequestParsers tag in our application is as follows:
>
>
>
> '<requestParsers enableRemoteStreaming="true"
>
>                      multipartUploadLimitInKB="2048000"
>
>                      formdataUploadLimitInKB="2048"
>
>                      addHttpRequestToContext="false"/>'
>
> Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb.
>
>
>
>    1.  As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as,
> 'If you are trying to add larger content then you have to "chop" that
> by yourself and add it as multivalued. Can be done within a self written loader. '
>
>
> I'm a newbie to Solr and I didn't get what exactly 'self written loader' is?
>
>
>
> Could you please provide us sample code, that helps us to go further?
>
>
> [image001]
> Thanks & Regards,
> Kranthi Kumar.K,
> Software Engineer,
> Ccube Fintech Global Services Pvt Ltd.,
> Email/Skype: [hidden email]<mailto:[hidden email]>,
> Mobile: +91-8978078449.
>
>
> From: Kranthi Kumar K <[hidden email]>
> Sent: Thursday, January 17, 2019 12:43 PM
> To: [hidden email]; [hidden email]
> Cc: Ananda Babu medida <[hidden email]>; Srinivasa Reddy Karri <[hidden email]>; Michelle Ngo <[hidden email]>
> Subject: Re: Solr Size Limitation upto 32 kb limitation
>
>
> Hi Team,
>
>
>
> Can we have any updates on the below issue? We are awaiting your reply.
>
>
>
> Thanks,
>
> Kranthi kumar.K
>
> ________________________________
> From: Kranthi Kumar K
> Sent: Friday, January 4, 2019 5:01:38 PM
> To: [hidden email]<mailto:[hidden email]>
> Cc: Ananda Babu medida; Srinivasa Reddy Karri
> Subject: Solr Size Limitation upto 32 kb limitation
>
>
> Hi team,
>
>
>
> We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions:
>
>
>
>    1.  java.lang.illegalargumentexception
>    2.  org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception
>
>
>
> Please find the attached screenshot for reference.
>
>
>
> We have searched for solutions in many forums and didn't find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the 'field' from sting to  'text_general' might solve the issue. Please have a look in the below forum:
>
>
>
> https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t
>
>
>
> Schema.xml:
>
> Changed from:
>
> '<field name="text" type="string_rev" indexed="true" stored="false" multiValued="true" />'
>
>
>
> Changed to:
>
> '<field name="text" type="text_general " indexed="true" stored="false" multiValued="true" />'
>
>
>
> We have tried it but still it is not importing the files > 32 KB or 32766 bytes.
>
>
>
> Could you please let us know the solution to fix this issue? We'll be awaiting your reply.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr Size Limitation upto 32 kb limitation

Jörn Franke
I guess selfwritten loader means that you split up the file in 32 kB chunks or smaller and then posts each of those 32 kB chunks to the multi valued field.

> Am 18.01.2019 um 11:51 schrieb Kranthi Kumar K <[hidden email]>:
>
> Hi team,
>  
> Thank you Erick Erickson ,Bernd Fehling , Jan Hoydahl for your suggested solutions. I’ve tried the suggested one’s and still we are unable to import files having            size  >32 kb, it is displaying same error.
>  
> Below link has the suggested solutions. Please have a look once.
>  
> http://lucene.472066.n3.nabble.com/Solr-Size-Limitation-upto-32-KB-files-td4419779.html
>  
> As per Erick Erickson, I’ve changed the string type to Text type based and still the issue occurs .
> I’ve changed from :
>  
> <field name="FileContent" type="string_rev" indexed="true" stored="true" />
>  
> Changed to:
>  
> <field name="FileContent" type="text" indexed="true" stored="true" />
>  
> If we do so, it is showing error in the log, please find the error in the attachment.
>  
> If I change to:
>  
> <field name="FileContent" type="text_general" indexed="true" stored="true" />
>  
> It is not showing any error , but the issue still exists.
>  
> As per Jan Hoydahl, I have gone through the link that you have provided and checked ‘requestParsers’ tag in solrconfig.xml,
>  
> RequestParsers tag in our application is as follows:
>  
> ‘<requestParsers enableRemoteStreaming="true"
>                     multipartUploadLimitInKB="2048000"
>                     formdataUploadLimitInKB="2048"
>                     addHttpRequestToContext="false"/>’
> Request parsers, which we are using and in the link you have provided are similar. And still we are unable to import the files size >32 kb.
>  
> As per Bernd Fehling, we are using Solr 4.10.2. you have mentioned as,
> ‘If you are trying to add larger content then you have to "chop" that
> by yourself and add it as multivalued. Can be done within a self written loader. ’
>  
> I’m a newbie to Solr and I didn’t get what exactly ‘self written loader’ is?
>  
> Could you please provide us sample code, that helps us to go further?
>  
>  
> <image001.png>
>
> Thanks & Regards,
> Kranthi Kumar.K,
> Software Engineer,
> Ccube Fintech Global Services Pvt Ltd.,
> Email/Skype: [hidden email],
> Mobile: +91-8978078449.
>  
>  
> From: Kranthi Kumar K <[hidden email]>
> Sent: Thursday, January 17, 2019 12:43 PM
> To: [hidden email]; [hidden email]
> Cc: Ananda Babu medida <[hidden email]>; Srinivasa Reddy Karri <[hidden email]>; Michelle Ngo <[hidden email]>
> Subject: Re: Solr Size Limitation upto 32 kb limitation
>  
> Hi Team,
>
>  
>
> Can we have any updates on the below issue? We are awaiting your reply.
>
>  
>
> Thanks,
>
> Kranthi kumar.K
>
> From: Kranthi Kumar K
> Sent: Friday, January 4, 2019 5:01:38 PM
> To: [hidden email]
> Cc: Ananda Babu medida; Srinivasa Reddy Karri
> Subject: Solr Size Limitation upto 32 kb limitation
>  
> Hi team,
>
>  
>
> We are currently using Solr 4.2.1 version in our project and everything is going well. But recently, we are facing an issue with Solr Data Import. It is not importing the files with size greater than 32766 bytes (i.e, 32 kb) and showing 2 exceptions:
>
>  
>
> java.lang.illegalargumentexception
> org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception
>  
>
> Please find the attached screenshot for reference.
>
>  
>
> We have searched for solutions in many forums and didn’t find the exact solution for this issue. Interestingly, we found in the article, by changing the type of the ‘field’ from sting to  ‘text_general’ might solve the issue. Please have a look in the below forum:
>
>  
>
> https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t 
>
>  
>
> Schema.xml:
>
> Changed from:
>
> ‘<field name="text" type="string_rev" indexed="true" stored="false" multiValued="true" />’
>
>  
>
> Changed to:
>
> ‘<field name="text" type="text_general " indexed="true" stored="false" multiValued="true" />’
>
>  
>
> We have tried it but still it is not importing the files > 32 KB or 32766 bytes.
>
>  
>
> Could you please let us know the solution to fix this issue? We’ll be awaiting your reply.
>
>  
> <solr size.PNG>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]