wrong remaining space reported by Data Nodes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

wrong remaining space reported by Data Nodes

Ophir Etzion
Hi,

I have a cluster with about 200 Data Nodes. 
when doing `hdfs dfsadmin -report` all nodes report less remaining space then exists on the disks (as seen by 'df -h' on the data node machine).
it is less by about 2TB in every machine.
when restarting a DataNode it calculates the remaining space well again and we get the 2TB back as remaining space.

any idea why this difference in the remaining space happens and how I can solve it? (something other than doing a rolling restart on the nodes every time this happens)

Thank you,
Ophir


Reply | Threaded
Open this post in threaded view
|

RE: wrong remaining space reported by Data Nodes

Vinayakumar B-4

Hi

 

You might be hitting https://issues.apache.org/jira/browse/HDFS-9530,.

 

This will arrive soon in coming 2.7.3 Release. J

 

-Vinay

 

From: Ophir Etzion [mailto:[hidden email]]
Sent: 22 July 2016 19:03
To: [hidden email]
Subject: wrong remaining space reported by Data Nodes

 

Hi,

I have a cluster with about 200 Data Nodes. 
when doing `hdfs dfsadmin -report` all nodes report less remaining space then exists on the disks (as seen by 'df -h' on the data node machine).
it is less by about 2TB in every machine.
when restarting a DataNode it calculates the remaining space well again and we get the 2TB back as remaining space.

any idea why this difference in the remaining space happens and how I can solve it? (something other than doing a rolling restart on the nodes every time this happens)

Thank you,
Ophir

 

Reply | Threaded
Open this post in threaded view
|

Re: wrong remaining space reported by Data Nodes

Ophir Etzion
Thank you very much for response!

Do you know if its just reporting badly in that command or will hdfs not have that space available if it needs it?

Thank you,
Ophir

On Fri, Jul 22, 2016 at 9:35 AM, Vinayakumar B <[hidden email]> wrote:

Hi

 

You might be hitting https://issues.apache.org/jira/browse/HDFS-9530,.

 

This will arrive soon in coming 2.7.3 Release. J

 

-Vinay

 

From: Ophir Etzion [mailto:[hidden email]]
Sent: 22 July 2016 19:03
To: [hidden email]
Subject: wrong remaining space reported by Data Nodes

 

Hi,

I have a cluster with about 200 Data Nodes. 
when doing `hdfs dfsadmin -report` all nodes report less remaining space then exists on the disks (as seen by 'df -h' on the data node machine).
it is less by about 2TB in every machine.
when restarting a DataNode it calculates the remaining space well again and we get the 2TB back as remaining space.

any idea why this difference in the remaining space happens and how I can solve it? (something other than doing a rolling restart on the nodes every time this happens)

Thank you,
Ophir