Verify cluster setup works

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Verify cluster setup works

Mike Wenzel

I just set up my first Hadoop cluster. My cluster looks like:


Node1 – NameNode + ResourceManager

Node2 – SecondaryNameNode

Node3 – DataNode (+NodeManager)

Node4 – DataNode (+NodeManager)

Node5 – DataNode (+NodeManager)


Doing java’s jps command on all machines looks good.

My hadoop-hduser-namenode-node1.log (and same on other machines) are looking fine too. I got a single warning ( only got one directory).

I started a mapreduce-example job wordcount (twice) using a 550 MB apache-log. Job ran, output seems fine.


My questions:


- My Web-UI http://namenode:8088/ shows 3 Active Nodes – Node State Running but I couldn’t see any changes at all while my jobs were run.

Java Output after starting the job:

> 16/07/05 13:27:31 INFO mapreduce.Job: The url to track the job: http://localhost:8080/


- Also Java output said ‘split into 5 blocks’. So I’d like to see where blocks getting stored to be sure that my cluster is working fine. Can I somehow check where blocks are getting stored and if it replication works just like it should in theory.


- My Web-UI http://datanode:50075 only shows: ‘DataNode on Datanode:50075’. I cannot click on overview or get any helpful information here. When I got my first hands on Hadoop I used a single-node-setup. There, this page had tons of information about the node-health/status/jobs and I was able to browse the HDFS. I’m missing this here, and I don’t get why.


I appreciate every kind of help in advice.

Have a good day everyone.

Best Regards,