[jira] [Commented] (NUTCH-2635) Generator writes unneeded temporary output

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (NUTCH-2635) Generator writes unneeded temporary output

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641209#comment-16641209 ]

Hudson commented on NUTCH-2635:
-------------------------------

SUCCESS: Integrated in Jenkins build Nutch-trunk #3558 (See [https://builds.apache.org/job/Nutch-trunk/3558/])
NUTCH-2635 Generator writes unneeded temporary output - output is (snagel: [https://github.com/apache/nutch/commit/113c58e85f23ebbe2031ec56cca2dd194523f9ef])
* (edit) src/java/org/apache/nutch/crawl/Generator.java


> Generator writes unneeded temporary output
> ------------------------------------------
>
>                 Key: NUTCH-2635
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2635
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 1.15
>            Reporter: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.16
>
>
> Generator writes the temporary output of the Selector job/step twice (see [line 516|https://github.com/apache/nutch/blob/branch-1.15/src/java/org/apache/nutch/crawl/Generator.java#L516]). Not a big issue when generating small fetch lists but may be when working on large data. The temporary output looks like:
> {noformat}
> % tree -h generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/
> enerate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/
> |-- [4.0K]  fetchlist-1
> |   `-- [ 25M]  part-r-00000
> `-- [ 77M]  part-r-00000
> 1 directory, 2 files
> % file generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/part-r-00000
> generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/part-r-00000: ASCII text
> % file generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/fetchlist-1/part-r-00000
> generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/fetchlist-1/part-r-00000: Apache Hadoop Sequence file version 6
> {noformat}
> The unneeded output is plain-text which explains its larger size compared to the Hadoop Sequence file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)