Sending compressed (gzip) UpdateRequest with SolrJ

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Sending compressed (gzip) UpdateRequest with SolrJ

Gael Jourdan-Weil
Hello,

I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.

Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587

By the way, is there any documentation about javabin? I could only find one on the "old wiki".

Thanks,
Gaël
Reply | Threaded
Open this post in threaded view
|

RE: Sending compressed (gzip) UpdateRequest with SolrJ

Gael Jourdan-Weil
Answering to myself on this one.

Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.

Seems quite surprising to me though.

-----
 
Hello,

I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.

Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
- https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587

By the way, is there any documentation about javabin? I could only find one on the "old wiki".

Thanks,
Gaël
Reply | Threaded
Open this post in threaded view
|

Re: Sending compressed (gzip) UpdateRequest with SolrJ

matthew sporleder
jetty supports http gzip and I've added it to solr before in my own
installs (and submitted patches to do so by default to solr) but I
don't know about the handling for solrj.

IME compression helps a little, sometimes a lot, and never hurts.
Even the admin interface benefits a lot from regular old http gzip

On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
<[hidden email]> wrote:

>
> Answering to myself on this one.
>
> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>
> Seems quite surprising to me though.
>
> -----
>
> Hello,
>
> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>
> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>
> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>
> Thanks,
> Gaël
Reply | Threaded
Open this post in threaded view
|

RE: Sending compressed (gzip) UpdateRequest with SolrJ

Gael Jourdan-Weil
You're right Matthew.

Jetty supports it for responses but for requests it doesn't seem to be the default.
However I found a configuration not documented that needs to be set in the GzipHandler for it to work: inflateBufferSize.

For SolrJ it still hacky to send gzip requests, maybe easier to use a regular http call..

---

De : matthew sporleder <[hidden email]>
Envoyé : jeudi 7 janvier 2021 16:43
À : [hidden email] <[hidden email]>
Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ
 
jetty supports http gzip and I've added it to solr before in my own
installs (and submitted patches to do so by default to solr) but I
don't know about the handling for solrj.

IME compression helps a little, sometimes a lot, and never hurts.
Even the admin interface benefits a lot from regular old http gzip

On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
<[hidden email]> wrote:

>
> Answering to myself on this one.
>
> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>
> Seems quite surprising to me though.
>
> -----
>
> Hello,
>
> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>
> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>
> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>
> Thanks,
> Gaël
Reply | Threaded
Open this post in threaded view
|

Re: Sending compressed (gzip) UpdateRequest with SolrJ

Walter Underwood
Years ago, working on the Ultraseek spider, we did a bunch of tests on compressed HTTP.
I expected it to be a big win, but the results were really inconclusive. Sometimes it was faster,
sometimes it was slower. We left it turned off.

It is an absolute win for serving already-compressed static content with Apache or whatever.
For dynamic content, it will increase some amount of delay as stuff is compressed before
sending. If the content already fits in one or two packets, it is just extra overhead. For really
large data, it helps with transmission time, but the processing time for large data probably
overwhelms the network time.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Jan 8, 2021, at 12:01 AM, Gael Jourdan-Weil <[hidden email]> wrote:
>
> You're right Matthew.
>
> Jetty supports it for responses but for requests it doesn't seem to be the default.
> However I found a configuration not documented that needs to be set in the GzipHandler for it to work: inflateBufferSize.
>
> For SolrJ it still hacky to send gzip requests, maybe easier to use a regular http call..
>
> ---
>
> De : matthew sporleder <[hidden email]>
> Envoyé : jeudi 7 janvier 2021 16:43
> À : [hidden email] <[hidden email]>
> Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ
>  
> jetty supports http gzip and I've added it to solr before in my own
> installs (and submitted patches to do so by default to solr) but I
> don't know about the handling for solrj.
>
> IME compression helps a little, sometimes a lot, and never hurts.
> Even the admin interface benefits a lot from regular old http gzip
>
> On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil
> <[hidden email]> wrote:
>>
>> Answering to myself on this one.
>>
>> Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now.
>> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now.
>>
>> Seems quite surprising to me though.
>>
>> -----
>>
>> Hello,
>>
>> I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ.
>>
>> Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default?
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin)
>> - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587
>>
>> By the way, is there any documentation about javabin? I could only find one on the "old wiki".
>>
>> Thanks,
>> Gaël