[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] Commented: (NUTCH-258) Once Nutch logs a SEVERE log item, Nutch fails forevermore

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/NUTCH-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12467887 ]

Chris A. Mattmann commented on NUTCH-258:


 From recent conversations on the mailing list where Doug mentioned that this issue may now be rendered resolved by recent changes to Hadoop, I'm wondering if we can close this issue? It's currently listed as a "critical" priority bug, and there are currently 3 watchers of the issue. I've asked this several times recently in the last few months about whether people are still experiencing this issue. So, the question is, are they? If not, I'd like to close out the issue as I'm trying to get things organized here in JIRA, so that developers and contributors can have a good idea of what issues are out there, that really need some attention. With the recent lack of developer resources, I think closing out issues that are not reproduceable, issues that people are no longer experiencing, or issues resolved by recent changes in Hadoop/etc. is an important thing to do in this process.

 Thus, I'm opening this issue up to any objections for closing/resolving it. If I don't hear any objections in the next week, I will close this issue out.



> Once Nutch logs a SEVERE log item, Nutch fails forevermore
> ----------------------------------------------------------
>                 Key: NUTCH-258
>                 URL: https://issues.apache.org/jira/browse/NUTCH-258
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 0.8
>         Environment: All
>            Reporter: Scott Ganyo
>         Assigned To: Chris A. Mattmann
>            Priority: Critical
>             Fix For: 0.9.0
>         Attachments: dumbfix.patch, NUTCH-258.Mattmann.060906.patch.txt, NUTCH-258.Mattmann.080406.patch.txt
> Once a SEVERE log item is written, Nutch shuts down any fetching forevermore.  This is from the run() method in Fetcher.java:
>     public void run() {
>       synchronized (Fetcher.this) {activeThreads++;} // count threads
>       try {
>         UTF8 key = new UTF8();
>         CrawlDatum datum = new CrawlDatum();
>         while (true) {
>           if (LogFormatter.hasLoggedSevere())     // something bad happened
>             break;                                // exit
> Notice the last 2 lines.  This will prevent Nutch from ever Fetching again once this is hit as LogFormatter is storing this data as a static.
> (Also note that "LogFormatter.hasLoggedSevere()" is also checked in org.apache.nutch.net.URLFilterChecker and will disable this class as well.)
> This must be fixed or Nutch cannot be run as any kind of long-running service.  Furthermore, I believe it is a poor decision to rely on a logging event to determine the state of the application - this could have any number of side-effects that would be extremely difficult to track down.  (As it has already for me.)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.