Gracefull stop in the middle of a fetch phase ?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Gracefull stop in the middle of a fetch phase ?

MilleBii
Hi guys,

I'm in the middle of a very long fetch phase, too long actually. I would
like to stop it but not loose 5 days of fetching.
Is there anything I can do ?

--
-MilleBii-
Reply | Threaded
Open this post in threaded view
|

Re: Gracefull stop in the middle of a fetch phase ?

Doğacan Güney-3
On Thu, Jul 23, 2009 at 21:29, MilleBii<[hidden email]> wrote:
> Hi guys,
>
> I'm in the middle of a very long fetch phase, too long actually. I would
> like to stop it but not loose 5 days of fetching.
> Is there anything I can do ?
>

No, unfortunately nutch 1.0 does not have that feature. But we are
working on it :)

https://issues.apache.org/jira/browse/NUTCH-650

> --
> -MilleBii-
>



--
Doğacan Güney
Reply | Threaded
Open this post in threaded view
|

Re: Gracefull stop in the middle of a fetch phase ?

Alex McLintock
In reply to this post by MilleBii
I am not sure if it solves your problem but you might do something
like disconnect your machines from the internet - preferably by making
your dns server return "dont know that domain"

This will relatively quickly cause the remaining part of the fetch to fail.

Just a suggestion...

Alex

2009/7/23 MilleBii <[hidden email]>:
> Hi guys,
>
> I'm in the middle of a very long fetch phase, too long actually. I would
> like to stop it but not loose 5 days of fetching.
> Is there anything I can do ?
>
> --
> -MilleBii-
>
Reply | Threaded
Open this post in threaded view
|

Re: Gracefull stop in the middle of a fetch phase ?

Andrzej Białecki-2
Alex McLintock wrote:
> I am not sure if it solves your problem but you might do something
> like disconnect your machines from the internet - preferably by making
> your dns server return "dont know that domain"
>
> This will relatively quickly cause the remaining part of the fetch to fail.
>
> Just a suggestion...

I solved this once by implementing a check in Fetcher.run() for a marker
file on HDFS. If the presence of this file was detected, the
FetcherThreads would be stopped one by one (again, by setting a flag in
their run() methods to terminate the loop).

It's a hack but it works well.

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Reply | Threaded
Open this post in threaded view
|

Re: Gracefull stop in the middle of a fetch phase ?

Alex McLintock
2009/7/25 Andrzej Bialecki <[hidden email]>:

> I solved this once by implementing a check in Fetcher.run() for a marker
> file on HDFS. If the presence of this file was detected, the FetcherThreads
> would be stopped one by one (again, by setting a flag in their run() methods
> to terminate the loop).
>

Personally I would like to see such a feature in the core code.

Alex