[jira] Created: (NUTCH-411) Parse ignores meta refresh redirection

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-411) Parse ignores meta refresh redirection

JIRA jira@apache.org
Parse ignores meta refresh redirection
--------------------------------------

                 Key: NUTCH-411
                 URL: http://issues.apache.org/jira/browse/NUTCH-411
             Project: Nutch
          Issue Type: Bug
    Affects Versions: 0.9.0
            Reporter: Dogacan Güney
            Priority: Minor


If fetching and parsing are run as seperate jobs, then redirection coming from meta refresh tag (i.e. <meta http-equiv="refresh" content="0;url=foo/">) is ignored, resulting in the loss of that ("foo/") url.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-411) Parse ignores meta refresh redirection

JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/NUTCH-411?page=comments#action_12454649 ]
           
Dogacan Güney commented on NUTCH-411:
-------------------------------------

My not-necessarily-correct patch for this. We add the new url as a newly discovered url (so it gets initialScore), which is different from what happens if we parse in fetcher.

I believe that in the long term, nutch should associate source url with the redirected url. But this patch (or a more correct version of this:D) can be applied so that we do not lose urls in the short term.

> Parse ignores meta refresh redirection
> --------------------------------------
>
>                 Key: NUTCH-411
>                 URL: http://issues.apache.org/jira/browse/NUTCH-411
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Dogacan Güney
>            Priority: Minor
>
> If fetching and parsing are run as seperate jobs, then redirection coming from meta refresh tag (i.e. <meta http-equiv="refresh" content="0;url=foo/">) is ignored, resulting in the loss of that ("foo/") url.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-411) Parse ignores meta refresh redirection

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/NUTCH-411?page=all ]

Dogacan Güney updated NUTCH-411:
--------------------------------

    Attachment: parse-redirect.patch

> Parse ignores meta refresh redirection
> --------------------------------------
>
>                 Key: NUTCH-411
>                 URL: http://issues.apache.org/jira/browse/NUTCH-411
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Dogacan Güney
>            Priority: Minor
>         Attachments: parse-redirect.patch
>
>
> If fetching and parsing are run as seperate jobs, then redirection coming from meta refresh tag (i.e. <meta http-equiv="refresh" content="0;url=foo/">) is ignored, resulting in the loss of that ("foo/") url.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (NUTCH-411) Parse ignores meta refresh redirection

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/NUTCH-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doğacan Güney closed NUTCH-411.
-------------------------------

       Resolution: Fixed
    Fix Version/s: 1.0.0
         Assignee: Doğacan Güney

This is fixed as part of NUTCH-547.

> Parse ignores meta refresh redirection
> --------------------------------------
>
>                 Key: NUTCH-411
>                 URL: https://issues.apache.org/jira/browse/NUTCH-411
>             Project: Nutch
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Doğacan Güney
>            Assignee: Doğacan Güney
>            Priority: Minor
>             Fix For: 1.0.0
>
>         Attachments: parse-redirect.patch
>
>
> If fetching and parsing are run as seperate jobs, then redirection coming from meta refresh tag (i.e. <meta http-equiv="refresh" content="0;url=foo/">) is ignored, resulting in the loss of that ("foo/") url.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.