[jira] Commented: (NUTCH-369) StringUtil.resolveEncodingAlias is unuseful.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-369) StringUtil.resolveEncodingAlias is unuseful.

Clark Perkins (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530798 ]

Hudson commented on NUTCH-369:
------------------------------

Integrated in Nutch-Nightly #219 (See [http://lucene.zones.apache.org:8080/hudson/job/Nutch-Nightly/219/])

> StringUtil.resolveEncodingAlias  is unuseful.
> ---------------------------------------------
>
>                 Key: NUTCH-369
>                 URL: https://issues.apache.org/jira/browse/NUTCH-369
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.9.0
>         Environment: all
>            Reporter: King Kong
>            Assignee: Doğacan Güney
>            Priority: Minor
>         Attachments: patch.diff, remover.diff
>
>
> After we defined encoding alias map in StringUtil , but parse html use orginal encoding also.
> I found it is reading charset from  meta in nekohtml which HtmlParser  used .
> we can set it's feature "http://cyberneko.org/html/features/scanner/ignore-specified-charset" to true
> that nekohtml will use encoding we set;
> concretely,
>   private DocumentFragment parseNeko(InputSource input) throws Exception {
>     DOMFragmentParser parser = new DOMFragmentParser();
>     // some plugins, e.g., creativecommons, need to examine html comments
>     try {
>    + parser.setFeature("http://cyberneko.org/html/features/scanner/ignore-specified-charset",true);
>       parser.setFeature("http://apache.org/xml/features/include-comments",
>               true);
>       ....
> BTW, It must be add on front of try block,because the following sentence  (parser.setFeature("http://apache.org/xml/features/include-comments",
>               true);) will throw exception.
>      

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.