[jira] [Updated] (NUTCH-2791) domainstats, protocolstats and crawlcomplete do not handle GCS URLs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (NUTCH-2791) domainstats, protocolstats and crawlcomplete do not handle GCS URLs

Sergey Smolyakov (Jira)

     [ https://issues.apache.org/jira/browse/NUTCH-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Mézard updated NUTCH-2791:
----------------------------------
    Priority: Trivial  (was: Minor)

> domainstats, protocolstats and crawlcomplete do not handle GCS URLs
> -------------------------------------------------------------------
>
>                 Key: NUTCH-2791
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2791
>             Project: Nutch
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 1.17
>            Reporter: Patrick Mézard
>            Priority: Trivial
>
> I am running Nutch in GCP Dataproc. The domainstats, protocolstats and crawlcomplete commands do not resolve crawldb paths correctly from GCS URLs.
> Also:
>  * protocolstats has an off-by-one error when resolving the numReducers argument
>  * crawlcomplete -inputDirs is inconsistent with other command: where domainstats expect directories containing a "current" path, crawlcomplete looks for directories containing a "crawldb/current" path.
> I will send a PR after creating the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)