[jira] Created: (NUTCH-426) parse-js skips parsing if found URL fails java.net.URL parse

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-426) parse-js skips parsing if found URL fails java.net.URL parse

JIRA jira@apache.org
parse-js skips parsing if found URL fails java.net.URL parse
------------------------------------------------------------

                 Key: NUTCH-426
                 URL: https://issues.apache.org/jira/browse/NUTCH-426
             Project: Nutch
          Issue Type: Bug
          Components: fetcher
    Affects Versions: 0.9.0
            Reporter: [hidden email]
            Priority: Minor


The parse-js plugin in getJSLinks tries a regex looking for likely URLs against a string of javascript.  Any matches that do not begin 'www' are given to java.net.URL with base URL to test 'URLness'.  Often this test will fail.  Currently, when it fails, nutch skips processing any more of the javascript snippet logs an error.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-426) parse-js skips parsing if found URL fails java.net.URL parse

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462307 ]

[hidden email] commented on NUTCH-426:
-----------------------------------------

Just attached a patch that catches the MalformedURLException, logs the failure at trace level, and then continues to pickup on any more matches found in remaining javascript.

> parse-js skips parsing if found URL fails java.net.URL parse
> ------------------------------------------------------------
>
>                 Key: NUTCH-426
>                 URL: https://issues.apache.org/jira/browse/NUTCH-426
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: [hidden email]
>            Priority: Minor
>         Attachments: nutch426.patch
>
>
> The parse-js plugin in getJSLinks tries a regex looking for likely URLs against a string of javascript.  Any matches that do not begin 'www' are given to java.net.URL with base URL to test 'URLness'.  Often this test will fail.  Currently, when it fails, nutch skips processing any more of the javascript snippet logs an error.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-426) parse-js skips parsing if found URL fails java.net.URL parse

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

[hidden email] updated NUTCH-426:
------------------------------------

    Attachment: nutch426.patch

> parse-js skips parsing if found URL fails java.net.URL parse
> ------------------------------------------------------------
>
>                 Key: NUTCH-426
>                 URL: https://issues.apache.org/jira/browse/NUTCH-426
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: [hidden email]
>            Priority: Minor
>         Attachments: nutch426.patch
>
>
> The parse-js plugin in getJSLinks tries a regex looking for likely URLs against a string of javascript.  Any matches that do not begin 'www' are given to java.net.URL with base URL to test 'URLness'.  Often this test will fail.  Currently, when it fails, nutch skips processing any more of the javascript snippet logs an error.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (NUTCH-426) parse-js skips parsing if found URL fails java.net.URL parse

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrzej Bialecki  closed NUTCH-426.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.9.0
         Assignee: Andrzej Bialecki

Fixed in rev. 493085. Thank you!

> parse-js skips parsing if found URL fails java.net.URL parse
> ------------------------------------------------------------
>
>                 Key: NUTCH-426
>                 URL: https://issues.apache.org/jira/browse/NUTCH-426
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: [hidden email]
>         Assigned To: Andrzej Bialecki
>            Priority: Minor
>             Fix For: 0.9.0
>
>         Attachments: nutch426.patch
>
>
> The parse-js plugin in getJSLinks tries a regex looking for likely URLs against a string of javascript.  Any matches that do not begin 'www' are given to java.net.URL with base URL to test 'URLness'.  Often this test will fail.  Currently, when it fails, nutch skips processing any more of the javascript snippet logs an error.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira