Nodemanager crashing repeatedly

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Nodemanager crashing repeatedly

Gajanan Watkar
I am running Nutch-2.3.1 over Hadoop-2.5.2 and Hbase-1.2.3 with
integration to Solr-6.5.1. I have crawled over 10 million pages. But
while doing all this I am continuously facing two problems:

1. My Nodemanager is crashing repeatedly during different phases of
crawl. It crashes my linux session and forces logout with nodemanager
killed. I log-in again, restart NodeManger and the same failed crawl
phase runs to success. [Nodemanager log has nothing to report]

2. I am running all my crawl phases one by one without crawl script, as
with crawl script most of the time my jobs were exiting with
"WaitForjobCompletion" error at different stages of crawl. So, I
decided to go ahead with one by one method which prevented
"WaitForjobCompletion" to occure.

Any help will be highly appreciated. New to mailing-list, New to Nutch.

-Gajanan
Reply | Threaded
Open this post in threaded view
|

Re: Nodemanager crashing repeatedly

lewis john mcgibbney-2
Hi Gajanan,
Which OS are you running this on?
I would also suggest that if you want to use the 2.x codebase, you should
use the most recent from SCM e.g. check out master and change to 2.x branch.
Finally, for now at least, you didn't mention the phase at which the crawl
is failing. Can you provide this?

On Thu, Sep 6, 2018 at 8:58 AM <[hidden email]> wrote:

> From: Gajanan Watkar <[hidden email]>
> To: [hidden email]
> Cc:
> Bcc:
> Date: Wed, 05 Sep 2018 11:27:21 +0530
> Subject: Nodemanager crashing repeatedly
> I am running Nutch-2.3.1 over Hadoop-2.5.2 and Hbase-1.2.3 with
> integration to Solr-6.5.1. I have crawled over 10 million pages. But
> while doing all this I am continuously facing two problems:
>
> 1. My Nodemanager is crashing repeatedly during different phases of
> crawl. It crashes my linux session and forces logout with nodemanager
> killed. I log-in again, restart NodeManger and the same failed crawl
> phase runs to success. [Nodemanager log has nothing to report]
>
> 2. I am running all my crawl phases one by one without crawl script, as
> with crawl script most of the time my jobs were exiting with
> "WaitForjobCompletion" error at different stages of crawl. So, I
> decided to go ahead with one by one method which prevented
> "WaitForjobCompletion" to occure.
>
> Any help will be highly appreciated. New to mailing-list, New to Nutch.
>
> -Gajanan
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Nodemanager crashing repeatedly

Gajanan Watkar
Thanks Lewis,
I am running on Debian Stretch.
Its month old checkout that I am using.
Nodemanager crashes during different phases of crawl, i.e. sometimes during
generate, sometimes during fetch, sometimes during parse and sometime
during parse, updatedb, index and dedupe.
On some occasions it crashes immediately after completing the respective
crawl phase.
Note: It appears that my nodemanager, all other hadoop daemons and hbase
were using /tmp for local and temporary storage. Even though my /tmp was
having enough space, I configured temp and local directories for everything
including map reduce tasks on my /home partition. That seem to have
stabilizing effect. Needs more testing. Will report if it stabilizes.

-Gajanan




On Thu, Sep 6, 2018 at 10:31 PM lewis john mcgibbney <[hidden email]>
wrote:

> Hi Gajanan,
> Which OS are you running this on?
> I would also suggest that if you want to use the 2.x codebase, you should
> use the most recent from SCM e.g. check out master and change to 2.x
> branch.
> Finally, for now at least, you didn't mention the phase at which the crawl
> is failing. Can you provide this?
>
> On Thu, Sep 6, 2018 at 8:58 AM <[hidden email]> wrote:
>
> > From: Gajanan Watkar <[hidden email]>
> > To: [hidden email]
> > Cc:
> > Bcc:
> > Date: Wed, 05 Sep 2018 11:27:21 +0530
> > Subject: Nodemanager crashing repeatedly
> > I am running Nutch-2.3.1 over Hadoop-2.5.2 and Hbase-1.2.3 with
> > integration to Solr-6.5.1. I have crawled over 10 million pages. But
> > while doing all this I am continuously facing two problems:
> >
> > 1. My Nodemanager is crashing repeatedly during different phases of
> > crawl. It crashes my linux session and forces logout with nodemanager
> > killed. I log-in again, restart NodeManger and the same failed crawl
> > phase runs to success. [Nodemanager log has nothing to report]
> >
> > 2. I am running all my crawl phases one by one without crawl script, as
> > with crawl script most of the time my jobs were exiting with
> > "WaitForjobCompletion" error at different stages of crawl. So, I
> > decided to go ahead with one by one method which prevented
> > "WaitForjobCompletion" to occure.
> >
> > Any help will be highly appreciated. New to mailing-list, New to Nutch.
> >
> > -Gajanan
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Nodemanager crashing repeatedly

Gajanan Watkar
Hi Lewis,
It appears that my setup was infected. After studying ResourceManager logs
closely I found that lot of jobs were getting submitted to my cluster as
user "dr who". Moreover my crontab was listing 2 wget cron jobs I never
configured (Suspect it to be cryptocurrency miner) and one java app running
from /var/tmp/java. I Configured firewall, blocked port 8088, purged cron
(as it was coming back with every re-install) and removed java app from
/var/tmp/java. It seem to have stabilized my setup. For now it is working
fine. No more unexpected NodeManager Exits. Also applied patch for
MalformedURLException.

I am getting uneven region sizes, can you suggest me on pre-spliting
webpage table i.e. split points to be used and splitting policy and optimum
GC setup for regionserver for efficient Nutch crawling.

-Gajanan





On Sun, Sep 9, 2018 at 8:34 AM Gajanan Watkar <[hidden email]>
wrote:

> Thanks Lewis,
> I am running on Debian Stretch.
> Its month old checkout that I am using.
> Nodemanager crashes during different phases of crawl, i.e. sometimes
> during generate, sometimes during fetch, sometimes during parse and
> sometime during parse, updatedb, index and dedupe.
> On some occasions it crashes immediately after completing the respective
> crawl phase.
> Note: It appears that my nodemanager, all other hadoop daemons and hbase
> were using /tmp for local and temporary storage. Even though my /tmp was
> having enough space, I configured temp and local directories for everything
> including map reduce tasks on my /home partition. That seem to have
> stabilizing effect. Needs more testing. Will report if it stabilizes.
>
> -Gajanan
>
>
>
>
> On Thu, Sep 6, 2018 at 10:31 PM lewis john mcgibbney <[hidden email]>
> wrote:
>
>> Hi Gajanan,
>> Which OS are you running this on?
>> I would also suggest that if you want to use the 2.x codebase, you should
>> use the most recent from SCM e.g. check out master and change to 2.x
>> branch.
>> Finally, for now at least, you didn't mention the phase at which the crawl
>> is failing. Can you provide this?
>>
>> On Thu, Sep 6, 2018 at 8:58 AM <[hidden email]> wrote:
>>
>> > From: Gajanan Watkar <[hidden email]>
>> > To: [hidden email]
>> > Cc:
>> > Bcc:
>> > Date: Wed, 05 Sep 2018 11:27:21 +0530
>> > Subject: Nodemanager crashing repeatedly
>> > I am running Nutch-2.3.1 over Hadoop-2.5.2 and Hbase-1.2.3 with
>> > integration to Solr-6.5.1. I have crawled over 10 million pages. But
>> > while doing all this I am continuously facing two problems:
>> >
>> > 1. My Nodemanager is crashing repeatedly during different phases of
>> > crawl. It crashes my linux session and forces logout with nodemanager
>> > killed. I log-in again, restart NodeManger and the same failed crawl
>> > phase runs to success. [Nodemanager log has nothing to report]
>> >
>> > 2. I am running all my crawl phases one by one without crawl script, as
>> > with crawl script most of the time my jobs were exiting with
>> > "WaitForjobCompletion" error at different stages of crawl. So, I
>> > decided to go ahead with one by one method which prevented
>> > "WaitForjobCompletion" to occure.
>> >
>> > Any help will be highly appreciated. New to mailing-list, New to Nutch.
>> >
>> > -Gajanan
>> >
>> >
>>
>