[jira] Created: (NUTCH-242) Add optional -urlFiltering to updatedb

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
Add optional -urlFiltering to updatedb
--------------------------------------

         Key: NUTCH-242
         URL: http://issues.apache.org/jira/browse/NUTCH-242
     Project: Nutch
        Type: New Feature
    Versions: 0.8-dev    
    Reporter: Rod Taylor


Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.

Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/NUTCH-242?page=all ]

Rod Taylor updated NUTCH-242:
-----------------------------

    Attachment: nutch_crawldb_filtering.patch

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>          Key: NUTCH-242
>          URL: http://issues.apache.org/jira/browse/NUTCH-242
>      Project: Nutch
>         Type: New Feature
>     Versions: 0.8-dev
>     Reporter: Rod Taylor
>  Attachments: nutch_crawldb_filtering.patch
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/NUTCH-242?page=comments#action_12372581 ]

Doug Cutting commented on NUTCH-242:
------------------------------------

Shouldn't you use the returned value of the filter?  If so, then this should be done in a mapper, not in the reducer.

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>          Key: NUTCH-242
>          URL: http://issues.apache.org/jira/browse/NUTCH-242
>      Project: Nutch
>         Type: New Feature
>     Versions: 0.8-dev
>     Reporter: Rod Taylor
>  Attachments: nutch_crawldb_filtering.patch
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/NUTCH-242?page=comments#action_12372592 ]

Rod Taylor commented on NUTCH-242:
----------------------------------

"Shouldn't you use the returned value of the filter?"

I forgot about URL Normalization (focused on expunging the data only). I suppose normalization rules could change as well.

Okay. I will look at Injector.java and essentially run the same steps when urlFiltering is enabled for updateDb.

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>          Key: NUTCH-242
>          URL: http://issues.apache.org/jira/browse/NUTCH-242
>      Project: Nutch
>         Type: New Feature
>     Versions: 0.8-dev
>     Reporter: Rod Taylor
>  Attachments: nutch_crawldb_filtering.patch
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/NUTCH-242?page=all ]

Rod Taylor updated NUTCH-242:
-----------------------------

    Attachment: nutch_urlfilter.patch

How about this one instead? It creates a CrawlDbMapper class which does the filtering when requested.

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>          Key: NUTCH-242
>          URL: http://issues.apache.org/jira/browse/NUTCH-242
>      Project: Nutch
>         Type: New Feature
>     Versions: 0.8-dev
>     Reporter: Rod Taylor
>  Attachments: nutch_crawldb_filtering.patch, nutch_urlfilter.patch
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/NUTCH-242?page=comments#action_12372620 ]

Rod Taylor commented on NUTCH-242:
----------------------------------

Sorry. I cannot mark the first patch as being invalid and neglected to use a version number.

v2 is named nutch_urlfilter.patch

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>          Key: NUTCH-242
>          URL: http://issues.apache.org/jira/browse/NUTCH-242
>      Project: Nutch
>         Type: New Feature
>     Versions: 0.8-dev
>     Reporter: Rod Taylor
>  Attachments: nutch_crawldb_filtering.patch, nutch_urlfilter.patch
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Closed: (NUTCH-242) Add optional -urlFiltering to updatedb

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/NUTCH-242?page=all ]

Andrzej Bialecki  closed NUTCH-242.
-----------------------------------

    Resolution: Fixed

Fixed in rev. 438670, with modifications.

> Add optional -urlFiltering to updatedb
> --------------------------------------
>
>                 Key: NUTCH-242
>                 URL: http://issues.apache.org/jira/browse/NUTCH-242
>             Project: Nutch
>          Issue Type: New Feature
>    Affects Versions: 0.8
>            Reporter: Rod Taylor
>         Attachments: nutch_crawldb_filtering.patch, nutch_urlfilter.patch
>
>
> Allow filtering the URLs completely out of the database during an updatedb run. This allows the regex-urlfilter.xml rules to be changed and the non-matching entries to be expunged from the database.
> Merging the functionality with updatedb was done for efficiency reasons since this eats up CPU time only where a separate job would use IO and CPU time.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira