[jira] [Commented] (NUTCH-2511) SitemapProcessor limited by http.content.limit

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (NUTCH-2511) SitemapProcessor limited by http.content.limit

Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951904#comment-16951904 ]

Hudson commented on NUTCH-2511:

SUCCESS: Integrated in Jenkins build Nutch-trunk #3651 (See [https://builds.apache.org/job/Nutch-trunk/3651/])
Nutch-2511 Support large sitemaps (github: [https://github.com/apache/nutch/commit/83ea2070e5bf34c0548847c33b3e4a85f4c9c923])
* (edit) src/java/org/apache/nutch/util/SitemapProcessor.java

> SitemapProcessor limited by http.content.limit
> ----------------------------------------------
>                 Key: NUTCH-2511
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2511
>             Project: Nutch
>          Issue Type: Bug
>          Components: sitemap
>    Affects Versions: 1.14
>            Reporter: Yossi Tamari
>            Priority: Major
>             Fix For: 1.17
> Because SitemapProcessor uses the HTTP protocol plugin, which limits the size of a response to http.content.limit (64KB by default), it can only handle sitemaps smaller than that size. 
> I don't believe that is the intent of the users by setting http.content.limit - they want to limit document size, not sitemap size. The spec specifically says that sitemaps can be up to 50MB.

This message was sent by Atlassian Jira