Cluster Summary

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Cluster Summary

Developer Developer
Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
setup is correct ?

I can start and stop using start-all and stop-all script with no errors.

But during crawl when i look at the status on
http://master:50030/jobtracker.jsp, the cluster summary shows only one node.
?

Any suggestions


 MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
Is there any document or wiki where describes how to read  Is thee any
document that explains the hadoop/mapreduce administration  outout ? It is
very daunting for a newbee to follow it.

Thanks !


On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <[hidden email]>
wrote:

> Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
> setup is correct ?
>
> I can start and stop using start-all and stop-all script with no errors.
>
> But during crawl when i look at the status on
> http://master:50030/jobtracker.jsp, the cluster summary shows only one
> node. ?
>
> Any suggestions
>
>
>  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
In reply to this post by Developer Developer
Any comments? I am not sure how i debug this problem.

On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <[hidden email]>
wrote:

> Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
> setup is correct ?
>
> I can start and stop using start-all and stop-all script with no errors.
>
> But during crawl when i look at the status on
> http://master:50030/jobtracker.jsp, the cluster summary shows only one
> node. ?
>
> Any suggestions
>
>
>  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
In reply to this post by Developer Developer
no comments yet ?

Did I ask any stupid question?


On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <[hidden email]>
wrote:

> Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
> setup is correct ?
>
> I can start and stop using start-all and stop-all script with no errors.
>
> But during crawl when i look at the status on
> http://master:50030/jobtracker.jsp, the cluster summary shows only one
> node. ?
>
> Any suggestions
>
>
>  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

John Mendenhall
> > Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
> > setup is correct ?
> >
> > I can start and stop using start-all and stop-all script with no errors.
> >
> > But during crawl when i look at the status on
> > http://master:50030/jobtracker.jsp, the cluster summary shows only one
> > node. ?
> >
> > Any suggestions
> >
> >
> >  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>

Did you see all nodes listed in the output of the start-all script?
It should list the different node types for each node when starting up.

JohnM

--
john mendenhall
[hidden email]
surf utopia
internet services
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
Yeah , I did see them in the output os start-all.sh and also when i stop
them stop-all.sh

Is that all ? I would be surprised if there is no other way to confirm the
distributiveness of Nutch/Hadoop.

The filesystem directory does not contain any content on the second node (
although it is formatted correctly).



On Thu, Mar 20, 2008 at 7:36 PM, John Mendenhall <[hidden email]>
wrote:

> > > Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that
> the
> > > setup is correct ?
> > >
> > > I can start and stop using start-all and stop-all script with no
> errors.
> > >
> > > But during crawl when i look at the status on
> > > http://master:50030/jobtracker.jsp, the cluster summary shows only one
> > > node. ?
> > >
> > > Any suggestions
> > >
> > >
> > >  MapsReducesTasks/NodeNodes 0241 <http://master:50030/machines.jsp<http://ascot1:50030/machines.jsp>
> >
>
> Did you see all nodes listed in the output of the start-all script?
> It should list the different node types for each node when starting up.
>
> JohnM
>
> --
> john mendenhall
> [hidden email]
> surf utopia
> internet services
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

gostanford
I have the same issue
Developer Developer wrote
Yeah , I did see them in the output os start-all.sh and also when i stop
them stop-all.sh

Is that all ? I would be surprised if there is no other way to confirm the
distributiveness of Nutch/Hadoop.

The filesystem directory does not contain any content on the second node (
although it is formatted correctly).



On Thu, Mar 20, 2008 at 7:36 PM, John Mendenhall <john@surfutopia.net>
wrote:

> > > Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that
> the
> > > setup is correct ?
> > >
> > > I can start and stop using start-all and stop-all script with no
> errors.
> > >
> > > But during crawl when i look at the status on
> > > http://master:50030/jobtracker.jsp, the cluster summary shows only one
> > > node. ?
> > >
> > > Any suggestions
> > >
> > >
> > >  MapsReducesTasks/NodeNodes 0241 <http://master:50030/machines.jsp<http://ascot1:50030/machines.jsp>
> >
>
> Did you see all nodes listed in the output of the start-all script?
> It should list the different node types for each node when starting up.
>
> JohnM
>
> --
> john mendenhall
> john@surfutopia.net
> surf utopia
> internet services
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
In reply to this post by Developer Developer
Hello Frens,


I debugged the problem and here is some more informaiton and question.

I think the slave node is not communicating with the master node through
continuous heart beat. I have set the <name>mapred.job.tracker</name> value
to <value>hostname:40070</value>. So , the slaves must communicate on 40070

On the Master Node the process that listens on 40071 looks like this.


lsof -i -n -P | grep 400
java 10766 nutch 34u IPv6 1460375 TCP 127.0.0.1:40070 (LISTEN)
java 10766 nutch 38u IPv6 1462135 TCP
127.0.0.1:40070->127.0.0.1:60165(ESTABLISHED)


Thanks means the java process on 40070 can only listen to localhost,
therefore all slave connections will be rejected.

What am I missing here ?





On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <[hidden email]>
wrote:

> Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
> setup is correct ?
>
> I can start and stop using start-all and stop-all script with no errors.
>
> But during crawl when i look at the status on
> http://master:50030/jobtracker.jsp, the cluster summary shows only one
> node. ?
>
> Any suggestions
>
>
>  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Boris Lau-2
Hi,

If you check your host file, is the hostname mapped to 127.0.0.1?
They might be the reason why it is listening only on a local port.
Try comment out that line.

Hope this helps.

Cheers
boris

On Tue, Mar 25, 2008 at 7:37 PM, Developer Developer
<[hidden email]> wrote:

> Hello Frens,
>
>
>  I debugged the problem and here is some more informaiton and question.
>
>  I think the slave node is not communicating with the master node through
>  continuous heart beat. I have set the <name>mapred.job.tracker</name> value
>  to <value>hostname:40070</value>. So , the slaves must communicate on 40070
>
>  On the Master Node the process that listens on 40071 looks like this.
>
>
>  lsof -i -n -P | grep 400
>  java 10766 nutch 34u IPv6 1460375 TCP 127.0.0.1:40070 (LISTEN)
>  java 10766 nutch 38u IPv6 1462135 TCP
>  127.0.0.1:40070->127.0.0.1:60165(ESTABLISHED)
>
>
>  Thanks means the java process on 40070 can only listen to localhost,
>  therefore all slave connections will be rejected.
>
>  What am I missing here ?
>
>
>
>
>
>
>  On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <[hidden email]>
>  wrote:
>
>
>
> > Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that the
>  > setup is correct ?
>  >
>  > I can start and stop using start-all and stop-all script with no errors.
>  >
>  > But during crawl when i look at the status on
>  > http://master:50030/jobtracker.jsp, the cluster summary shows only one
>  > node. ?
>  >
>  > Any suggestions
>  >
>  >
>  >  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
>  >
>  >
>  >
>
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Summary

Developer Developer
Boris, Thank you very much. It worked :)




On Tue, Mar 25, 2008 at 3:46 PM, Boris Lau <[hidden email]> wrote:

> Hi,
>
> If you check your host file, is the hostname mapped to 127.0.0.1?
> They might be the reason why it is listening only on a local port.
> Try comment out that line.
>
> Hope this helps.
>
> Cheers
> boris
>
> On Tue, Mar 25, 2008 at 7:37 PM, Developer Developer
> <[hidden email]> wrote:
> > Hello Frens,
> >
> >
> >  I debugged the problem and here is some more informaiton and question.
> >
> >  I think the slave node is not communicating with the master node
> through
> >  continuous heart beat. I have set the <name>mapred.job.tracker</name>
> value
> >  to <value>hostname:40070</value>. So , the slaves must communicate on
> 40070
> >
> >  On the Master Node the process that listens on 40071 looks like this.
> >
> >
> >  lsof -i -n -P | grep 400
> >  java 10766 nutch 34u IPv6 1460375 TCP 127.0.0.1:40070 (LISTEN)
> >  java 10766 nutch 38u IPv6 1462135 TCP
> >  127.0.0.1:40070->127.0.0.1:60165(ESTABLISHED)
> >
> >
> >  Thanks means the java process on 40070 can only listen to localhost,
> >  therefore all slave connections will be rejected.
> >
> >  What am I missing here ?
> >
> >
> >
> >
> >
> >
> >  On Wed, Mar 19, 2008 at 3:53 PM, Developer Developer <
> [hidden email]>
> >  wrote:
> >
> >
> >
> > > Hello I have setup nutch/hadoop on 2 nodes. How can i make sure that
> the
> >  > setup is correct ?
> >  >
> >  > I can start and stop using start-all and stop-all script with no
> errors.
> >  >
> >  > But during crawl when i look at the status on
> >  > http://master:50030/jobtracker.jsp, the cluster summary shows only
> one
> >  > node. ?
> >  >
> >  > Any suggestions
> >  >
> >  >
> >  >  MapsReducesTasks/NodeNodes 0241 <http://ascot1:50030/machines.jsp>
> >  >
> >  >
> >  >
> >
>