[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12467931 ]

Scott Ganyo commented on NUTCH-258:
-----------------------------------

Chris,

I originally opened the issue... but unfortunately I can neither confirm nor deny that this is fixed as I'm no longer on the project that originally had the issue.  (And, in fact, they never allowed an upgrade to the latest version of Nutch/Hadoop anyway.)  So, close away if nobody else is having the issue!

Thanks!
Scott


> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>
>                 Key: NUTCH-258
>                 URL: https://issues.apache.org/jira/browse/NUTCH-258
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.8
>         Environment: All
>            Reporter: Scott Ganyo
>         Assigned To: Chris A. Mattmann
>            Priority: Critical
>             Fix For: 0.9.0
>
>         Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt, NUTCH-258.Mattmann.080406.patch.txt
>
>
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore.  This is from the run() method in Fetcher.java:
>     public void run() {
>       synchronized (Fetcher.this) {activeThreads++;} // count threads
>      
>       try {
>         UTF8 key = new UTF8();
>         CrawlDatum datum = new CrawlDatum();
>        
>         while (true) {
>           if (LogFormatter.hasLoggedSevere())     // something bad happened
>             break;                                // exit
>          
> Notice the last 2 lines.  This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service.  Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down.  (As it has already for me.)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.