[jira] Created: (NUTCH-243) Some meta-refresh urls get ignored due to matching regular expression

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-243) Some meta-refresh urls get ignored due to matching regular expression

kuladeep (Jira)
Some meta-refresh urls get ignored due to matching regular expression
---------------------------------------------------------------------

         Key: NUTCH-243
         URL: http://issues.apache.org/jira/browse/NUTCH-243
     Project: Nutch
        Type: Bug

  Components: fetcher  
    Versions: 0.8-dev    
    Reporter: Dennis Kubes
    Priority: Minor


On fetching of pages with meta-refresh tags the url is taken at face value without any filtering.  Some urls, such as those used by struts return with a jsessionid or with query strings.  Examples are:

http://www.somesite.com;jsessionid=3123123412ADBE3344...
http://www.somesite.com?querystring=value

The RegexURLFilter will match these urls according to the following regex inside of the regex-urlfilter.txt file:

-[?*!@=]

Should these urls be cleaned up to allow processing and not match the previous URL filter or should they be ignored as they currently are?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-243) Some meta-refresh urls get ignored due to matching regular expression

kuladeep (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-243?page=all ]

Dennis Kubes updated NUTCH-243:
-------------------------------

    Priority: Trivial  (was: Minor)

This is resolved by NUTCH-255

> Some meta-refresh urls get ignored due to matching regular expression
> ---------------------------------------------------------------------
>
>          Key: NUTCH-243
>          URL: http://issues.apache.org/jira/browse/NUTCH-243
>      Project: Nutch
>         Type: Bug

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Dennis Kubes
>     Priority: Trivial

>
> On fetching of pages with meta-refresh tags the url is taken at face value without any filtering.  Some urls, such as those used by struts return with a jsessionid or with query strings.  Examples are:
> http://www.somesite.com;jsessionid=3123123412ADBE3344...
> http://www.somesite.com?querystring=value
> The RegexURLFilter will match these urls according to the following regex inside of the regex-urlfilter.txt file:
> -[?*!@=]
> Should these urls be cleaned up to allow processing and not match the previous URL filter or should they be ignored as they currently are?

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira