[jira] Created: (NUTCH-68) A tool to generate arbitrary fetchlists

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-68) A tool to generate arbitrary fetchlists

Steve Loughran (Jira)
A tool to generate arbitrary fetchlists
---------------------------------------

         Key: NUTCH-68
         URL: http://issues.apache.org/jira/browse/NUTCH-68
     Project: Nutch
        Type: New Feature
  Components: fetcher  
    Reporter: Andrzej Bialecki
 Assigned to: Andrzej Bialecki  
    Priority: Minor
 Attachments: FreeFetchlistTool.java

This is a tool to generate arbitrary fetchlists out of plain-text URL lists. I found it useful quite often, e.g. when I had to fetch certain specific pages without adding them to DB, or for testing purposes.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-68) A tool to generate arbitrary fetchlists

Steve Loughran (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-68?page=all ]

Andrzej Bialecki  updated NUTCH-68:
-----------------------------------

    Attachment: FreeFetchlistTool.java

> A tool to generate arbitrary fetchlists
> ---------------------------------------
>
>          Key: NUTCH-68
>          URL: http://issues.apache.org/jira/browse/NUTCH-68
>      Project: Nutch
>         Type: New Feature
>   Components: fetcher
>     Reporter: Andrzej Bialecki
>     Assignee: Andrzej Bialecki
>     Priority: Minor
>  Attachments: FreeFetchlistTool.java
>
> This is a tool to generate arbitrary fetchlists out of plain-text URL lists. I found it useful quite often, e.g. when I had to fetch certain specific pages without adding them to DB, or for testing purposes.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-68) A tool to generate arbitrary fetchlists

Steve Loughran (Jira)
In reply to this post by Steve Loughran (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-68?page=comments#action_12363403 ]

byron miller commented on NUTCH-68:
-----------------------------------

Works like a charm. I use this to do almost "live" fetches of submitted links.  How hard would it be to port this to .8?  it would almost be interesting to show how to do so on the wiki so people can have an idea how old stuff moves to mapreduce as i'm still wrapping my brain around the process :)

> A tool to generate arbitrary fetchlists
> ---------------------------------------
>
>          Key: NUTCH-68
>          URL: http://issues.apache.org/jira/browse/NUTCH-68
>      Project: Nutch
>         Type: New Feature
>   Components: fetcher
>     Reporter: Andrzej Bialecki
>     Assignee: Andrzej Bialecki
>     Priority: Minor
>  Attachments: FreeFetchlistTool.java
>
> This is a tool to generate arbitrary fetchlists out of plain-text URL lists. I found it useful quite often, e.g. when I had to fetch certain specific pages without adding them to DB, or for testing purposes.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira