[jira] Created: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
URLFilter-suffix management of the url path when the url contains some query parameters
---------------------------------------------------------------------------------------

                 Key: NUTCH-489
                 URL: https://issues.apache.org/jira/browse/NUTCH-489
             Project: Nutch
          Issue Type: Improvement
    Affects Versions: 0.9.0
         Environment: Windows, Java 1.5
            Reporter: Emmanuel Joke


The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).

So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.

I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)

     [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Joke updated NUTCH-489:
--------------------------------

    Attachment: SuffixURLFilter.java.patch
                suffix-urlfilter.txt.patch

> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
In reply to this post by Parth (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497770 ]

Doğacan Güney commented on NUTCH-489:
-------------------------------------

This is obviously useful but:

* Your patches both in this issue and in NUTCH-488 have whitespace issues. Nutch uses (or at least, tries to use) 2-space indentation. You seem to use a tab-indentation, so your patches end up removing code only to re-add it with wrong indentation.

* I don't understand why new P option only works after I. What if I want this filter to be case-sensitive and to filter on path?

* (Not related to your patch) It seems even a "-I" line makes SuffixURLFilter ignore case. Is that intended?


> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
In reply to this post by Parth (Jira)

     [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Joke updated NUTCH-489:
--------------------------------

    Attachment: SuffixURLFilter_v2.java.patch

My mistake...

I've added a new patchwhich is supposed to:
1- have a correct indentation
2- manage to read the option I and P without any order or constraint.

Hope it will help

> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
In reply to this post by Parth (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498113 ]

Doğacan Güney commented on NUTCH-489:
-------------------------------------

Hmm.. Won't it now cause Nutch to filter on path on a line like this:
-(jpg|JPG)

Because, you just check if line contains a 'P', and line above does.

Perhaps, a better option would be to require P and I be the only characters on their line, like:
+I
+P
# but not +IP

> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
In reply to this post by Parth (Jira)

    [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499777 ]

Doğacan Güney commented on NUTCH-489:
-------------------------------------

Please ignore my last comment. I don't know what I was on when I wrote that. Your patch seems to be correct.

Andrzej, (hope you are reading this:) isn't javadoc in SuffixURLFilter wrong? It says "...., you should use "+.jpg" instead.", but AFAICS, that line would have no effect except possibly switching mode to accept. Since, first character after '+' is not 'I',  filter will just skip rest of the line. Am I missing something here?

> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (NUTCH-489) URLFilter-suffix management of the url path when the url contains some query parameters

Parth (Jira)
In reply to this post by Parth (Jira)

     [ https://issues.apache.org/jira/browse/NUTCH-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doğacan Güney resolved NUTCH-489.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 1.0.0
         Assignee: Doğacan Güney

Committed in rev. 548429 with some modifications:

1) Fix minor whitespace issues.

2) Filtering on path is disabled by default. One will have to uncomment line in suffix-urlfilter.txt to enable it.

3) Updated unit test case.

> URLFilter-suffix management of the url path when the url contains some query parameters
> ---------------------------------------------------------------------------------------
>
>                 Key: NUTCH-489
>                 URL: https://issues.apache.org/jira/browse/NUTCH-489
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: Windows, Java 1.5
>            Reporter: Emmanuel Joke
>            Assignee: Doğacan Güney
>             Fix For: 1.0.0
>
>         Attachments: suffix-urlfilter.txt.patch, SuffixURLFilter.java.patch, SuffixURLFilter_v2.java.patch
>
>
> The current filter compares only on string level. It try to apply the filter to the full URL (path + query parameters).
> So, even if we have define in our filter to exclude all js extension, it won't exclude this URL http://www.toto.com/from.js?id=5.
> I've added a new parameter in the filter which can be use to configure the filter to exclude URL based on the url path.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.