Job failes during injection or generation

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Job failes during injection or generation

sourabhk
Hi,
I am trying to run multiple nutch instances in parallel. But some times
during the execution one of the job fails during injection with the
exception:-

java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:211)

Or at
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
at org.apache.nutch.crawl.Generator.generate(Generator.java:430)

Can somebody give some light on this that what can be the reason of job
failing due to IOException
regards
Sourabh
Reply | Threaded
Open this post in threaded view
|

Re: Job failes during injection or generation

Alex McLintock
Does it not say elsewhere in the stack trace? Can you show us more of the
logs?

On 29 November 2010 17:52, Sourabh Kasliwal <[hidden email]> wrote:

> Hi,
> I am trying to run multiple nutch instances in parallel. But some times
> during the execution one of the job fails during injection with the
> exception:-
>
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
>
> Or at
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
> at org.apache.nutch.crawl.Generator.generate(Generator.java:430)
>
> Can somebody give some light on this that what can be the reason of job
> failing due to IOException
> regards
> Sourabh
>
Reply | Threaded
Open this post in threaded view
|

Re: Job failes during injection or generation

sourabhk
Thanks for replying..
The stack trace that I have sent is at the lowest level...
There is no other location that stack trace points to....
regards
Sourabh

On Mon, Nov 29, 2010 at 11:41 PM, Alex McLintock
<[hidden email]>wrote:

> Does it not say elsewhere in the stack trace? Can you show us more of the
> logs?
>
> On 29 November 2010 17:52, Sourabh Kasliwal <[hidden email]>
> wrote:
>
> > Hi,
> > I am trying to run multiple nutch instances in parallel. But some times
> > during the execution one of the job fails during injection with the
> > exception:-
> >
> > java.io.IOException: Job failed!
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
> >
> > Or at
> > java.io.IOException: Job failed!
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
> > at org.apache.nutch.crawl.Generator.generate(Generator.java:430)
> >
> > Can somebody give some light on this that what can be the reason of job
> > failing due to IOException
> > regards
> > Sourabh
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Job failes during injection or generation

Jitendra
I also faced same problem when trying to run crawl jobs in parallel locally.
Even stack trace was also same.

@Sourabh, Did you find out any reason/solution for this ?

Thanks
Jitendra

On Tue, Nov 30, 2010 at 11:17 AM, Sourabh Kasliwal
<[hidden email]>wrote:

> Thanks for replying..
> The stack trace that I have sent is at the lowest level...
> There is no other location that stack trace points to....
> regards
> Sourabh
>
> On Mon, Nov 29, 2010 at 11:41 PM, Alex McLintock
> <[hidden email]>wrote:
>
> > Does it not say elsewhere in the stack trace? Can you show us more of the
> > logs?
> >
> > On 29 November 2010 17:52, Sourabh Kasliwal <[hidden email]>
> > wrote:
> >
> > > Hi,
> > > I am trying to run multiple nutch instances in parallel. But some times
> > > during the execution one of the job fails during injection with the
> > > exception:-
> > >
> > > java.io.IOException: Job failed!
> > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > > at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
> > >
> > > Or at
> > > java.io.IOException: Job failed!
> > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > > at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
> > > at org.apache.nutch.crawl.Generator.generate(Generator.java:430)
> > >
> > > Can somebody give some light on this that what can be the reason of job
> > > failing due to IOException
> > > regards
> > > Sourabh
> > >
> >
>



--
Thanks and regards

Jitendra Singh
Reply | Threaded
Open this post in threaded view
|

Re: Job failes during injection or generation

sourabhk
No... did not find any solution or reason...
regards
Sourabh

On Wed, Dec 1, 2010 at 12:40 AM, jitendra rajput <[hidden email]>wrote:

> I also faced same problem when trying to run crawl jobs in parallel
> locally.
> Even stack trace was also same.
>
> @Sourabh, Did you find out any reason/solution for this ?
>
> Thanks
> Jitendra
>
> On Tue, Nov 30, 2010 at 11:17 AM, Sourabh Kasliwal
> <[hidden email]>wrote:
>
> > Thanks for replying..
> > The stack trace that I have sent is at the lowest level...
> > There is no other location that stack trace points to....
> > regards
> > Sourabh
> >
> > On Mon, Nov 29, 2010 at 11:41 PM, Alex McLintock
> > <[hidden email]>wrote:
> >
> > > Does it not say elsewhere in the stack trace? Can you show us more of
> the
> > > logs?
> > >
> > > On 29 November 2010 17:52, Sourabh Kasliwal <[hidden email]>
> > > wrote:
> > >
> > > > Hi,
> > > > I am trying to run multiple nutch instances in parallel. But some
> times
> > > > during the execution one of the job fails during injection with the
> > > > exception:-
> > > >
> > > > java.io.IOException: Job failed!
> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > > > at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
> > > >
> > > > Or at
> > > > java.io.IOException: Job failed!
> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> > > > at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
> > > > at org.apache.nutch.crawl.Generator.generate(Generator.java:430)
> > > >
> > > > Can somebody give some light on this that what can be the reason of
> job
> > > > failing due to IOException
> > > > regards
> > > > Sourabh
> > > >
> > >
> >
>
>
>
> --
> Thanks and regards
>
> Jitendra Singh
>
Reply | Threaded
Open this post in threaded view
|

Re: Job failes during injection or generation

sourabhk
@Jitendra: An off topic question, while running parallel jobs, after some
time performance degrades a lot. It takes much more time to do any of the
nutch jobs. Looking at vmstat, it seems that the program starts
thrashing(most of the time is spent in swapping memory blocks). Did you face
any thing similar ??
regards
Sourabh

On Wed, Dec 1, 2010 at 1:32 PM, Sourabh Kasliwal <[hidden email]>wrote:

> No... did not find any solution or reason...
> regards
> Sourabh
>
>
> On Wed, Dec 1, 2010 at 12:40 AM, jitendra rajput <[hidden email]>wrote:
>
>> I also faced same problem when trying to run crawl jobs in parallel
>> locally.
>> Even stack trace was also same.
>>
>> @Sourabh, Did you find out any reason/solution for this ?
>>
>> Thanks
>> Jitendra
>>
>> On Tue, Nov 30, 2010 at 11:17 AM, Sourabh Kasliwal
>> <[hidden email]>wrote:
>>
>> > Thanks for replying..
>> > The stack trace that I have sent is at the lowest level...
>> > There is no other location that stack trace points to....
>> > regards
>> > Sourabh
>> >
>> > On Mon, Nov 29, 2010 at 11:41 PM, Alex McLintock
>> > <[hidden email]>wrote:
>> >
>> > > Does it not say elsewhere in the stack trace? Can you show us more of
>> the
>> > > logs?
>> > >
>> > > On 29 November 2010 17:52, Sourabh Kasliwal <[hidden email]>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > > I am trying to run multiple nutch instances in parallel. But some
>> times
>> > > > during the execution one of the job fails during injection with the
>> > > > exception:-
>> > > >
>> > > > java.io.IOException: Job failed!
>> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>> > > > at org.apache.nutch.crawl.Injector.inject(Injector.java:211)
>> > > >
>> > > > Or at
>> > > > java.io.IOException: Job failed!
>> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>> > > > at org.apache.nutch.crawl.Generator.generate(Generator.java:523)
>> > > > at org.apache.nutch.crawl.Generator.generate(Generator.java:430)
>> > > >
>> > > > Can somebody give some light on this that what can be the reason of
>> job
>> > > > failing due to IOException
>> > > > regards
>> > > > Sourabh
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Thanks and regards
>>
>> Jitendra Singh
>>
>
>