Lucene › Hadoop Lucene › Hadoop lucene-users

Help - can't start namenode after disk full error

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Ryan LeCompte
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

Help - can't start namenode after disk full error

Ryan LeCompte
Hey guys,

Really trying to get our namenode back up and running after a full disk
error last night. I've freed up a lot of space, however the NameNode still
fails to startup:

2011-06-12 10:26:09,042 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-06-12 10:26:09,083 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 614919
2011-06-12 10:26:22,293 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 17
2011-06-12 10:26:22,300 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 102029859 loaded in 13 seconds.
2011-06-12 10:26:22,510 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:468)
at java.lang.Short.parseShort(Short.java:120)
at java.lang.Short.parseShort(Short.java:78)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.java:1269)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


We currently have our config setup as follows:

 <property>
    <name>dfs.name.dir</name>

<value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/name,/data4/hadoop/dfs/name</value>
  </property>

I've looked in each of those directories for an image/edits.new file, but
only the "edits" files exist.

Can anyone please guide me on the next step here to get this back up and
running?

Thanks!

Ryan
Zhong, Sheng
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

RE: Help - can't start namenode after disk full error

Zhong, Sheng
Could you backup your edits file, try $ printf "\xff\xff\xff\xee\xff" >
edits, and start HDFS? It should work. - Andy

-----Original Message-----
From: Ryan LeCompte [mailto:[hidden email]]
Sent: Sunday, June 12, 2011 9:29 AM
To: [hidden email]
Subject: Help - can't start namenode after disk full error

Hey guys,

Really trying to get our namenode back up and running after a full disk
error last night. I've freed up a lot of space, however the NameNode
still fails to startup:

2011-06-12 10:26:09,042 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-06-12 10:26:09,083 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 614919
2011-06-12 10:26:22,293 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 17
2011-06-12 10:26:22,300 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 102029859 loaded in 13 seconds.
2011-06-12 10:26:22,510 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
at java.lang.Integer.parseInt(Integer.java:468)
at java.lang.Short.parseShort(Short.java:120)
at java.lang.Short.parseShort(Short.java:78)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.jav
a:1269)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


We currently have our config setup as follows:

 <property>
    <name>dfs.name.dir</name>

<value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/n
ame,/data4/hadoop/dfs/name</value>
  </property>

I've looked in each of those directories for an image/edits.new file,
but only the "edits" files exist.

Can anyone please guide me on the next step here to get this back up and
running?

Thanks!

Ryan

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
Zhong, Sheng
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

RE: Help - can't start namenode after disk full error

Zhong, Sheng
Only apply it to /dfs/name/current/edits file...

-----Original Message-----
From: Zhong, Andy [mailto:[hidden email]]
Sent: Sunday, June 12, 2011 9:43 AM
To: [hidden email]
Subject: RE: Help - can't start namenode after disk full error

Could you backup your edits file, try $ printf "\xff\xff\xff\xee\xff" >
edits, and start HDFS? It should work. - Andy

-----Original Message-----
From: Ryan LeCompte [mailto:[hidden email]]
Sent: Sunday, June 12, 2011 9:29 AM
To: [hidden email]
Subject: Help - can't start namenode after disk full error

Hey guys,

Really trying to get our namenode back up and running after a full disk
error last night. I've freed up a lot of space, however the NameNode
still fails to startup:

2011-06-12 10:26:09,042 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-06-12 10:26:09,083 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 614919
2011-06-12 10:26:22,293 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 17
2011-06-12 10:26:22,300 INFO
org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 102029859 loaded in 13 seconds.
2011-06-12 10:26:22,510 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
at java.lang.Integer.parseInt(Integer.java:468)
at java.lang.Short.parseShort(Short.java:120)
at java.lang.Short.parseShort(Short.java:78)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.jav
a:1269)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:550)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


We currently have our config setup as follows:

 <property>
    <name>dfs.name.dir</name>

<value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/n
ame,/data4/hadoop/dfs/name</value>
  </property>

I've looked in each of those directories for an image/edits.new file,
but only the "edits" files exist.

Can anyone please guide me on the next step here to get this back up and
running?

Thanks!

Ryan

This message, including any attachments, is the property of Sears
Holdings Corporation and/or one of its subsidiaries. It is confidential
and may contain proprietary or legally privileged information. If you
are not the intended recipient, please delete it without reading the
contents. Thank you.
Ryan LeCompte
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

Re: Help - can't start namenode after disk full error

Ryan LeCompte
That worked, thanks!

On Sun, Jun 12, 2011 at 10:47 AM, Zhong, Andy <[hidden email]>wrote:

> Only apply it to /dfs/name/current/edits file...
>
> -----Original Message-----
> From: Zhong, Andy [mailto:[hidden email]]
> Sent: Sunday, June 12, 2011 9:43 AM
> To: [hidden email]
> Subject: RE: Help - can't start namenode after disk full error
>
> Could you backup your edits file, try $ printf "\xff\xff\xff\xee\xff" >
> edits, and start HDFS? It should work. - Andy
>
> -----Original Message-----
> From: Ryan LeCompte [mailto:[hidden email]]
> Sent: Sunday, June 12, 2011 9:29 AM
> To: [hidden email]
> Subject: Help - can't start namenode after disk full error
>
> Hey guys,
>
> Really trying to get our namenode back up and running after a full disk
> error last night. I've freed up a lot of space, however the NameNode
> still fails to startup:
>
> 2011-06-12 10:26:09,042 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2011-06-12 10:26:09,083 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Number of files = 614919
> 2011-06-12 10:26:22,293 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Number of files under construction = 17
> 2011-06-12 10:26:22,300 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 102029859 loaded in 13 seconds.
> 2011-06-12 10:26:22,510 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
> at java.lang.Integer.parseInt(Integer.java:468)
> at java.lang.Short.parseShort(Short.java:120)
> at java.lang.Short.parseShort(Short.java:78)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.jav
> a:1269)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
>
> We currently have our config setup as follows:
>
>  <property>
>    <name>dfs.name.dir</name>
>
> <value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/n
> ame,/data4/hadoop/dfs/name</value>
>  </property>
>
> I've looked in each of those directories for an image/edits.new file,
> but only the "edits" files exist.
>
> Can anyone please guide me on the next step here to get this back up and
> running?
>
> Thanks!
>
> Ryan
>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential
> and may contain proprietary or legally privileged information. If you
> are not the intended recipient, please delete it without reading the
> contents. Thank you.
>
Zhong, Sheng
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

Re: Help - can't start namenode after disk full error

Zhong, Sheng
My pleasure!

----- Original Message -----
From: Ryan LeCompte [mailto:[hidden email]]
Sent: Sunday, June 12, 2011 10:59 AM
To: [hidden email] <[hidden email]>
Subject: Re: Help - can't start namenode after disk full error

That worked, thanks!

On Sun, Jun 12, 2011 at 10:47 AM, Zhong, Andy <[hidden email]>wrote:

> Only apply it to /dfs/name/current/edits file...
>
> -----Original Message-----
> From: Zhong, Andy [mailto:[hidden email]]
> Sent: Sunday, June 12, 2011 9:43 AM
> To: [hidden email]
> Subject: RE: Help - can't start namenode after disk full error
>
> Could you backup your edits file, try $ printf "\xff\xff\xff\xee\xff" >
> edits, and start HDFS? It should work. - Andy
>
> -----Original Message-----
> From: Ryan LeCompte [mailto:[hidden email]]
> Sent: Sunday, June 12, 2011 9:29 AM
> To: [hidden email]
> Subject: Help - can't start namenode after disk full error
>
> Hey guys,
>
> Really trying to get our namenode back up and running after a full disk
> error last night. I've freed up a lot of space, however the NameNode
> still fails to startup:
>
> 2011-06-12 10:26:09,042 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2011-06-12 10:26:09,083 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Number of files = 614919
> 2011-06-12 10:26:22,293 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Number of files under construction = 17
> 2011-06-12 10:26:22,300 INFO
> org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 102029859 loaded in 13 seconds.
> 2011-06-12 10:26:22,510 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
> at java.lang.Integer.parseInt(Integer.java:468)
> at java.lang.Short.parseShort(Short.java:120)
> at java.lang.Short.parseShort(Short.java:78)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.readShort(FSEditLog.jav
> a:1269)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:550)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
>
> We currently have our config setup as follows:
>
>  <property>
>    <name>dfs.name.dir</name>
>
> <value>/data1/hadoop/dfs/name,/data2/hadoop/dfs/name,/data3/hadoop/dfs/n
> ame,/data4/hadoop/dfs/name</value>
>  </property>
>
> I've looked in each of those directories for an image/edits.new file,
> but only the "edits" files exist.
>
> Can anyone please guide me on the next step here to get this back up and
> running?
>
> Thanks!
>
> Ryan
>
> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential
> and may contain proprietary or legally privileged information. If you
> are not the intended recipient, please delete it without reading the
> contents. Thank you.
>
Loading...
Powered by Nabble See how NAML generates this page