[jira] Created: (NUTCH-249) black- white list url filtering

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-249) black- white list url filtering

Hudson (Jira)
black- white list url filtering
-------------------------------

         Key: NUTCH-249
         URL: http://issues.apache.org/jira/browse/NUTCH-249
     Project: Nutch
        Type: Improvement

  Components: fetcher  
    Versions: 0.8-dev    
    Reporter: Stefan Groschupf
    Priority: Trivial
     Fix For: 0.8-dev


Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-249) black- white list url filtering

Hudson (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-249?page=all ]

Stefan Groschupf updated NUTCH-249:
-----------------------------------

    Attachment: blackWhiteListV2.patch

A concept tryout of black- white list filtering. I'm looking for beta tester and improvement suggestions. (Especially I'm looking for terminus suggestions)
Such a filter mechanism can be very useful for vertical search deployments of nutch with very large filter sets.

A black-White Url pattern database can be created and used to filter urls until updating a crawldb. So the crawlDb contains only urls that passes the black white list. In case a url match a black url prefix it will not written to the crawlDb. In case a url match a white prefix it is written to the crawlDb.
In case a url does not match a white or black prefix it is also not written to the crawlDb.

Url filtering happens on a host level so a url only need to be filtered by all patterns for the same host.

Usage:
// inject prefix url patterns (a text file in a folder) that a url should not match
bin/nutch org.apache.nutch.crawl.bw.BWInjector bwdb ~/projects/negativeUrls/ -black
// injkect prefix url patterns that a url is allowed to match
bin/nutch org.apache.nutch.crawl.bw.BWInjector bwdb ~/projects/positiveUrls/ -white
// update a fetched segment into a database (only urls will be added to the db that pass the black white filter)
bin/nutch org.apache.nutch.crawl.bw.BWUpdateDb testCrawlDb bwdb segments/20060416181635/

Known Issues:
Hadoop does not allow to have different formats for one job, so some overhead format converting is required that currently slow down the processing.

Any comments are welcome!

> black- white list url filtering
> -------------------------------
>
>          Key: NUTCH-249
>          URL: http://issues.apache.org/jira/browse/NUTCH-249
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>      Fix For: 0.8-dev
>  Attachments: blackWhiteListV2.patch
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-249) black- white list url filtering

Hudson (Jira)
In reply to this post by Hudson (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-249?page=all ]

Stefan Groschupf updated NUTCH-249:
-----------------------------------

    Attachment: blackWhiteListV3.patch

A new patch that fix an bug where to less urls passed the filter.

> black- white list url filtering
> -------------------------------
>
>          Key: NUTCH-249
>          URL: http://issues.apache.org/jira/browse/NUTCH-249
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>      Fix For: 0.8-dev
>  Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-249) black- white list url filtering

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12376463 ]

Thomas Delnoij commented on NUTCH-249:
--------------------------------------

Sorry for my ignorance, but what is meant by a Terminus Suggestion?

> black- white list url filtering
> -------------------------------
>
>          Key: NUTCH-249
>          URL: http://issues.apache.org/jira/browse/NUTCH-249
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>      Fix For: 0.8-dev
>  Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-249) black- white list url filtering

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12376477 ]

Stefan Groschupf commented on NUTCH-249:
----------------------------------------

I mean the Class and method naming isn't very well.
Blacklist or blocklist? Whitelist or positivivelist?
Does this answer the question?

> black- white list url filtering
> -------------------------------
>
>          Key: NUTCH-249
>          URL: http://issues.apache.org/jira/browse/NUTCH-249
>      Project: Nutch
>         Type: Improvement

>   Components: fetcher
>     Versions: 0.8-dev
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>      Fix For: 0.8-dev
>  Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (NUTCH-249) black- white list url filtering

Thomas Delnoij-3
yes, thank you. What happens if the url matches both lists. There is not
guarantee that it won't match both lists is there?

Rgrds, Thomas



On 4/26/06, Stefan Groschupf (JIRA) <[hidden email]> wrote:

>
>     [
> http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12376477]
>
> Stefan Groschupf commented on NUTCH-249:
> ----------------------------------------
>
> I mean the Class and method naming isn't very well.
> Blacklist or blocklist? Whitelist or positivivelist?
> Does this answer the question?
>
> > black- white list url filtering
> > -------------------------------
> >
> >          Key: NUTCH-249
> >          URL: http://issues.apache.org/jira/browse/NUTCH-249
> >      Project: Nutch
> >         Type: Improvement
>
> >   Components: fetcher
> >     Versions: 0.8-dev
> >     Reporter: Stefan Groschupf
> >     Priority: Trivial
> >      Fix For: 0.8-dev
> >  Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
> >
> > Existing url filter mechanisms need to process each url against each
> filter pattern. For very large filter sets this may be does not scale very
> well.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
>    http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
>    http://www.atlassian.com/software/jira
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (NUTCH-249) black- white list url filtering

Stefan Groschupf-2
Blacklist hits have a higher priority.

Am 26.04.2006 um 14:22 schrieb TDLN:

> yes, thank you. What happens if the url matches both lists. There  
> is not
> guarantee that it won't match both lists is there?
>
> Rgrds, Thomas
>
>
>
> On 4/26/06, Stefan Groschupf (JIRA) <[hidden email]> wrote:
>>
>>     [
>> http://issues.apache.org/jira/browse/NUTCH-249?
>> page=comments#action_12376477]
>>
>> Stefan Groschupf commented on NUTCH-249:
>> ----------------------------------------
>>
>> I mean the Class and method naming isn't very well.
>> Blacklist or blocklist? Whitelist or positivivelist?
>> Does this answer the question?
>>
>>> black- white list url filtering
>>> -------------------------------
>>>
>>>          Key: NUTCH-249
>>>          URL: http://issues.apache.org/jira/browse/NUTCH-249
>>>      Project: Nutch
>>>         Type: Improvement
>>
>>>   Components: fetcher
>>>     Versions: 0.8-dev
>>>     Reporter: Stefan Groschupf
>>>     Priority: Trivial
>>>      Fix For: 0.8-dev
>>>  Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>>>
>>> Existing url filter mechanisms need to process each url against each
>> filter pattern. For very large filter sets this may be does not  
>> scale very
>> well.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the  
>> administrators:
>>    http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>>    http://www.atlassian.com/software/jira
>>
>>

---------------------------------------------------------------
company:        http://www.media-style.com
forum:        http://www.text-mining.org
blog:            http://www.find23.net


Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-249) black- white list url filtering

Hudson (Jira)
In reply to this post by Hudson (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-249?page=all ]

Sami Siren updated NUTCH-249:
-----------------------------

    Fix Version/s: 0.9-dev
                       (was: 0.8-dev)

> black- white list url filtering
> -------------------------------
>
>                 Key: NUTCH-249
>                 URL: http://issues.apache.org/jira/browse/NUTCH-249
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.8-dev
>            Reporter: Stefan Groschupf
>            Priority: Trivial
>             Fix For: 0.9-dev
>
>         Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-249) black- white list url filtering

Hudson (Jira)
In reply to this post by Hudson (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12432584 ]
           
Uros Gruber commented on NUTCH-249:
-----------------------------------

I'm trying to test this patch but I'm having build problems

compile-core:
    [javac] Compiling 2 source files to /usr/home/uros/nutch-wb/build/classes
    [javac] /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:261: createJob(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path) in org.apache.nutch.crawl.CrawlDb cannot be applied to (org.apache.hadoop.conf.Configuration,java.io.File)
    [javac]     JobConf updateJob = CrawlDb.createJob(getConf(), crawlDb);
    [javac]                                ^
    [javac] /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:267: install(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.fs.Path) in org.apache.nutch.crawl.CrawlDb cannot be applied to (org.apache.hadoop.mapred.JobConf,java.io.File)
    [javac]     CrawlDb.install(updateJob, crawlDb);
    [javac]            ^
    [javac] Note: /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java uses or overrides a deprecated API.



> black- white list url filtering
> -------------------------------
>
>                 Key: NUTCH-249
>                 URL: http://issues.apache.org/jira/browse/NUTCH-249
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.8
>            Reporter: Stefan Groschupf
>            Priority: Trivial
>             Fix For: 0.9.0
>
>         Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>
>
> Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (NUTCH-249) black- white list url filtering

Zaheed Haque
Hi

Lot of the patch/plugins in Jiira are not updated to reflect changes
in trunk. Probably the way to test it would be building this using
that specific revision of nutch.

cheers

On 9/5/06, Uros Gruber (JIRA) <[hidden email]> wrote:

>     [ http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12432584 ]
>
> Uros Gruber commented on NUTCH-249:
> -----------------------------------
>
> I'm trying to test this patch but I'm having build problems
>
> compile-core:
>     [javac] Compiling 2 source files to /usr/home/uros/nutch-wb/build/classes
>     [javac] /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:261: createJob(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path) in org.apache.nutch.crawl.CrawlDb cannot be applied to (org.apache.hadoop.conf.Configuration,java.io.File)
>     [javac]     JobConf updateJob = CrawlDb.createJob(getConf(), crawlDb);
>     [javac]                                ^
>     [javac] /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:267: install(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.fs.Path) in org.apache.nutch.crawl.CrawlDb cannot be applied to (org.apache.hadoop.mapred.JobConf,java.io.File)
>     [javac]     CrawlDb.install(updateJob, crawlDb);
>     [javac]            ^
>     [javac] Note: /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java uses or overrides a deprecated API.
>
>
>
> > black- white list url filtering
> > -------------------------------
> >
> >                 Key: NUTCH-249
> >                 URL: http://issues.apache.org/jira/browse/NUTCH-249
> >             Project: Nutch
> >          Issue Type: Improvement
> >          Components: fetcher
> >    Affects Versions: 0.8
> >            Reporter: Stefan Groschupf
> >            Priority: Trivial
> >             Fix For: 0.9.0
> >
> >         Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
> >
> >
> > Existing url filter mechanisms need to process each url against each filter pattern. For very large filter sets this may be does not scale very well.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (NUTCH-249) black- white list url filtering

Uroš Gruber-2
Zaheed Haque wrote:
> Hi
>
> Lot of the patch/plugins in Jiira are not updated to reflect changes
> in trunk. Probably the way to test it would be building this using
> that specific revision of nutch.
>
I'm aware of that. I just put a note because I see that this patch is
for 0.9.

regards,

Uros

> cheers
>
> On 9/5/06, Uros Gruber (JIRA) <[hidden email]> wrote:
>>     [
>> http://issues.apache.org/jira/browse/NUTCH-249?page=comments#action_12432584 
>> ]
>>
>> Uros Gruber commented on NUTCH-249:
>> -----------------------------------
>>
>> I'm trying to test this patch but I'm having build problems
>>
>> compile-core:
>>     [javac] Compiling 2 source files to
>> /usr/home/uros/nutch-wb/build/classes
>>     [javac]
>> /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:261:
>> createJob(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path)
>> in org.apache.nutch.crawl.CrawlDb cannot be applied to
>> (org.apache.hadoop.conf.Configuration,java.io.File)
>>     [javac]     JobConf updateJob = CrawlDb.createJob(getConf(),
>> crawlDb);
>>     [javac]                                ^
>>     [javac]
>> /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java:267:
>> install(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.fs.Path)
>> in org.apache.nutch.crawl.CrawlDb cannot be applied to
>> (org.apache.hadoop.mapred.JobConf,java.io.File)
>>     [javac]     CrawlDb.install(updateJob, crawlDb);
>>     [javac]            ^
>>     [javac] Note:
>> /usr/home/uros/nutch-wb/src/java/org/apache/nutch/crawl/bw/BWUpdateDb.java
>> uses or overrides a deprecated API.
>>
>>
>>
>> > black- white list url filtering
>> > -------------------------------
>> >
>> >                 Key: NUTCH-249
>> >                 URL: http://issues.apache.org/jira/browse/NUTCH-249
>> >             Project: Nutch
>> >          Issue Type: Improvement
>> >          Components: fetcher
>> >    Affects Versions: 0.8
>> >            Reporter: Stefan Groschupf
>> >            Priority: Trivial
>> >             Fix For: 0.9.0
>> >
>> >         Attachments: blackWhiteListV2.patch, blackWhiteListV3.patch
>> >
>> >
>> > Existing url filter mechanisms need to process each url against
>> each filter pattern. For very large filter sets this may be does not
>> scale very well.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the
>> administrators: http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>>
>>