[jira] [Commented] (NUTCH-1559) parse-metatags duplicates extracted metatags

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (NUTCH-1559) parse-metatags duplicates extracted metatags

Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953786#comment-16953786 ]

Sebastian Nagel commented on NUTCH-1559:

After a closer the look: the issue is not limited to parse-tika (also applies to parse-html). The comment in MetaTagsParser is just misleading. But I agree to remove the lines which extract <meta> tag values from ParseData's metadata.

Of course, the situation is somewhat complex but I see no counter arguments for this solution:

1. parser plugins must add some common <meta> tags to ParseData's metadata because the metadata is required for indexing (eg. "robots" required to handle the "noindex" directive, see NUTCH-1553). This will then necessarily lead to duplicate "metatag.*" values.

2. it is not guaranteed that a parser adds all available <meta> tags to ParseData's metadata, so MetaTagsParser must always look into the HTMLMetaTags passed to the filter(...) method to ensure that the <meta> tags are really found.

3. parse filter plugins (eg. "headings" see NUTCH-1005) may add key-value pairs to ParseData's metadata. The index-meta plugin is still able to index them without the "metatag." prefix.

The alternative would be to implement a deduplication which would require additional computations.

Any objections to finally apply [~jcocking]'s patch? - I'll open a PR including a unit test.

> parse-metatags duplicates extracted metatags
> --------------------------------------------
>                 Key: NUTCH-1559
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1559
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.6
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.17
>         Attachments: NUTCH-1559-1.x.patch
> If the plugin parse-metatags is used in combination with parse-tika, the extracted metatags (the pairs <name, value>) are duplicated.
> The metatags are found twice in parse.getData().getParseMeta() and in metaTags.getGeneralTags(). Is this necessary? Maybe we should fix parse-tika in this point?

This message was sent by Atlassian Jira