[jira] [Commented] (NUTCH-2733) protocol-okhttp: add support for Brotli compression (Content-Encoding)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (NUTCH-2733) protocol-okhttp: add support for Brotli compression (Content-Encoding)

Akash (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011864#comment-17011864 ]

ASF GitHub Bot commented on NUTCH-2733:
---------------------------------------

sebastian-nagel commented on pull request #492: NUTCH-2733 protocol-okhttp: add support for Brotli compression (Content-Encoding)
URL: https://github.com/apache/nutch/pull/492
 
 
   - upgrade to okhttp 4.3.2
   - add dependency to okhttp3.brotli and enable Brotli compression
   
   ```
   $> bin/nutch parsechecker -Dstore.http.headers=true -Dstore.http.request=true \
          -Dplugin.includes='protocol-okhttp|parse-html'  https://www.youtube.com/
   ...
   _request_=GET / HTTP/1.1
   ...
   Accept-Encoding: br,gzip
   ...
   
   _response.headers_=HTTP/1.1 200 OK
   Content-Encoding: br
   ...
   ```
   
   I'll run a test crawl with some additional metrics soon.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


> protocol-okhttp: add support for Brotli compression (Content-Encoding)
> ----------------------------------------------------------------------
>
>                 Key: NUTCH-2733
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2733
>             Project: Nutch
>          Issue Type: Improvement
>          Components: plugin, protocol
>    Affects Versions: 1.15
>            Reporter: Sebastian Nagel
>            Priority: Major
>              Labels: easytask, help-wanted, needs-test
>             Fix For: 1.17
>
>
> [Okhttp 4.1.0|https://square.github.io/okhttp/changelog/#version-410] adds support for Brotli compression. We might upgrade the okhttp dependency to enable support for the "brotli" Content-Encoding.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)