Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

solr2020
Hi,

We are getting the  below error message while trying to delete 30k records from solr.

Error occured while invoking endpoint on Solr:
org.apache.solr.client.solrj.SolrServerException: Error executing query
        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:98)
        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:36)
        at com.ntrs.wpf.search.batch.BatchProcessorImpl.run(BatchProcessorImpl.java:57)
        at com.ntrs.wpf.search.batch.SolrIndexingController.purgePositions(SolrIndexingController.java:235)
        at com.ntrs.wpf.search.batch.SolrIndexingController.execute(SolrIndexingController.java:106)
        at com.ntrs.wpf.search.batch.BatchFrameworkManager.main(BatchFrameworkManager.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:56)
Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109)
        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:387)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
        ... 10 more


we are forming the delete query as below.
 
(Account:123+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
(Account:345+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
(Account:569+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
(Account:789+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+..........+OR+30k th record)

But we are able to delete small number of records without any issues.

Can anyone please help us on this.
Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

Walter Underwood
60 in ASCII is '<'. Is it returning XML? Or maybe an error message?

wunder

On Jan 2, 2014, at 9:22 PM, gpssolr2020 <[hidden email]> wrote:

> Hi,
>
> We are getting the  below error message while trying to delete 30k records
> from solr.
>
> Error occured while invoking endpoint on Solr:
> org.apache.solr.client.solrj.SolrServerException: Error executing query
> at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:98)
> at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:36)
> at
> com.ntrs.wpf.search.batch.BatchProcessorImpl.run(BatchProcessorImpl.java:57)
> at
> com.ntrs.wpf.search.batch.SolrIndexingController.purgePositions(SolrIndexingController.java:235)
> at
> com.ntrs.wpf.search.batch.SolrIndexingController.execute(SolrIndexingController.java:106)
> at
> com.ntrs.wpf.search.batch.BatchFrameworkManager.main(BatchFrameworkManager.java:63)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:56)
> Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60)
> or the data in not in 'javabin' format
> at
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109)
> at
> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:387)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
> at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
> ... 10 more
>
>
> we are forming the delete query as below.
>
> (Account:123+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:345+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:569+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:789+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+..........+OR+30k
> th record)
>
> But we are able to delete small number of records without any issues.
>
> Can anyone please help us on this.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Invalid-version-expected-2-but-60-or-the-data-in-not-in-javabin-format-exception-while-deleting-30k-s-tp4109259.html
> Sent from the Solr - User mailing list archive at Nabble.com.

--
Walter Underwood
[hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

Shawn Heisey-4
In reply to this post by solr2020
On 1/2/2014 10:22 PM, gpssolr2020 wrote:
> Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60)
> or the data in not in 'javabin' format

<snip>

> (Account:123+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:345+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:569+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
> (Account:789+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+..........+OR+30k
> th record)
>
> But we are able to delete small number of records without any issues.
>
> Can anyone please help us on this.

Walter is right about the error message.  This indicates that Solr is
returning an error response that's in HTML or XML format rather than the
expected javabin.  The server log should actually contain more
information about what actually went wrong.

This is probably happening because the request is too big.  In my
indexing code, I send deleteByQuery requests in batches of 1000 for this
very reason.

If that date range is the same for all of the deletes in a batch, you
could make the request considerably smaller by using the following format:

DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"] AND Account:(123
OR 345 OR 569 OR 789 OR .... 30000th-value)

The other option is to increase the max form size of your servlet
container.  In 4.1 or later, this is actually controlled by Solr
directly, not the container config, with the formdataUploadLimitInKB
attribute on the requestParsers tag in solrconfig.xml.  The default is
2048, or 2MB.  With 30000 clauses like you have described, the request
will be a little bit more than 2MB.

There is a bug in the early Jetty 8 versions (that included with Solr
4.0) that prevented the form size from being set in the jetty config:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130

The Jetty 6 that is included with Solr 1.x and 3.x can be properly
configured.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

Mark Miller-3
Just an FYI, newer version of Solr will deploy the proper error message rather than that cryptic one.

- Mark

On Jan 3, 2014, at 12:54 AM, Shawn Heisey <[hidden email]> wrote:

> On 1/2/2014 10:22 PM, gpssolr2020 wrote:
>> Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60)
>> or the data in not in 'javabin' format
>
> <snip>
>
>> (Account:123+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
>> (Account:345+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
>> (Account:569+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+
>> (Account:789+AND+DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"])+OR+..........+OR+30k
>> th record)
>>
>> But we are able to delete small number of records without any issues.
>>
>> Can anyone please help us on this.
>
> Walter is right about the error message.  This indicates that Solr is
> returning an error response that's in HTML or XML format rather than the
> expected javabin.  The server log should actually contain more
> information about what actually went wrong.
>
> This is probably happening because the request is too big.  In my
> indexing code, I send deleteByQuery requests in batches of 1000 for this
> very reason.
>
> If that date range is the same for all of the deletes in a batch, you
> could make the request considerably smaller by using the following format:
>
> DATE:["2013-11-29T00:00:00Z"+TO+"2013-11-29T23:59:59Z"] AND Account:(123
> OR 345 OR 569 OR 789 OR .... 30000th-value)
>
> The other option is to increase the max form size of your servlet
> container.  In 4.1 or later, this is actually controlled by Solr
> directly, not the container config, with the formdataUploadLimitInKB
> attribute on the requestParsers tag in solrconfig.xml.  The default is
> 2048, or 2MB.  With 30000 clauses like you have described, the request
> will be a little bit more than 2MB.
>
> There is a bug in the early Jetty 8 versions (that included with Solr
> 4.0) that prevented the form size from being set in the jetty config:
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
>
> The Jetty 6 that is included with Solr 1.x and 3.x can be properly
> configured.
>
> Thanks,
> Shawn
>

Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

solr2020
In reply to this post by Shawn Heisey-4
Thanks all for the reply.

we are working on that to reduce the delete query size.

But after that we faced one more issue.The issue is our batch process is able to delete 16k records but we got an OOM exception in one server.(out of 4 server in solrcloud).We are using solr 4.2 and zookeeper 3.4.5. and we are using sunjdk. we increased the heap size from 12 to 16 GB but no luck.we have 250GB ram dedicated only for solr and we have 100 million records.

we have We are getting the following error during the deletion process.


org.apache.solr.common.SolrException; Error opening realtime searcher for deleteByQuery:java.lang.OutOfMemoryError: Java heap space


 org.apache.solr.common.SolrException; Error while trying to recover. core=pe_shard1_replica1:java.lang.IllegalStateException: this writ
er hit an OutOfMemoryError; cannot commit

 org.apache.solr.common.SolrException; auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot
 commit


ERROR -  org.apache.solr.common.SolrException; org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are disabled.


Note:Initially we got the same error when we index more than one million record at a time.So to fix that we divided data into two chunks and it is working fine now.

Please advise ..


Thanks...
Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

Yago Riveiro
You have your GC tuned?

In the pass I had a lot of problems with zookeeper as a result of GC pauses because my heap was too big.

Increase your heap to 20G or more, and use some of the configurations exposed in this thread http://wiki.apache.org/solr/ShawnHeisey

The first works fine for me with a heap with 24G.

--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Tuesday, January 7, 2014 at 3:54 PM, gpssolr2020 wrote:

> Thanks all for the reply.
>
> we are working on that to reduce the delete query size.
>
> But after that we faced one more issue.The issue is our batch process is
> able to delete 16k records but we got an OOM exception in one server.(out of
> 4 server in solrcloud).We are using solr 4.2 and zookeeper 3.4.5. and we are
> using sunjdk. we increased the heap size from 12 to 16 GB but no luck.we
> have 250GB ram dedicated only for solr and we have 100 million records.
>
> we have We are getting the following error during the deletion process.
>
>
> org.apache.solr.common.SolrException; Error opening realtime searcher for
> deleteByQuery:java.lang.OutOfMemoryError: Java heap space
>
>
> org.apache.solr.common.SolrException; Error while trying to recover.
> core=pe_shard1_replica1:java.lang.IllegalStateException: this writ
> er hit an OutOfMemoryError; cannot commit
>
> org.apache.solr.common.SolrException; auto commit
> error...:java.lang.IllegalStateException: this writer hit an
> OutOfMemoryError; cannot
> commit
>
>
> ERROR - org.apache.solr.common.SolrException;
> org.apache.solr.common.SolrException: Cannot talk to ZooKeeper - Updates are
> disabled.
>
>
> Note:Initially we got the same error when we index more than one million
> record at a time.So to fix that we divided data into two chunks and it is
> working fine now.
>
> Please advise ..
>
>
> Thanks...
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Invalid-version-expected-2-but-60-or-the-data-in-not-in-javabin-format-exception-while-deleting-30k-s-tp4109259p4109993.html
> Sent from the Solr - User mailing list archive at Nabble.com (http://Nabble.com).
>
>


Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format exception while deleting 30k records

solr2020
Thanks. We will try with more heap.

And we noticed that zookeeper(open jdk) and Solr(sun jdk) is using different jvm. Will this really cause this OOM issue ?.