About Archival Storage

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

About Archival Storage

kevin-5
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?
Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

Rakesh Radhakrishnan-2
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

kevin-5
Thanks a lot Rakesh. 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?


Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

Rakesh Radhakrishnan-2
>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 
While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

>>>It use algorithm like LRU、LFU ?
It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

Regards,
Rakesh


On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:
Thanks a lot Rakesh. 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?



Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

kevin-5
Thanks again. "automatically" what I mean is the hdfs mover knows the hot data have come to cold , I don't need to tell it what exactly files/dirs need to be move now ? 
Of course I should tell it what files/dirs need to monitoring.

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 
While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

>>>It use algorithm like LRU、LFU ?
It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

Regards,
Rakesh


On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:
Thanks a lot Rakesh. 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?




Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

Rakesh Radhakrishnan-2
Based on storage policy the data from hot storage will be moved to cold storage. The storage policy defines the number of replicas to be located on each storage type. It is possible to change the storage policy on a directory(for example: HOT to COLD) and then invoke 'Mover tool' on that directory to make the policy effective. One can set/change the storage policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>". After setting the new policy, you need to run the tool, then it identifies the replicas to be moved based on the storage policy information, and schedules the movement between source and destination data nodes to satisfy the policy. Internally, the tool is comparing the 'storage type' of a block in order to fulfill the 'storage policy' requirement. 

Probably you can refer https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html to know more about storage types, storage policies and hdfs commands. Hope this helps.

Rakesh

On Wed, Jul 20, 2016 at 10:30 AM, kevin <[hidden email]> wrote:
Thanks again. "automatically" what I mean is the hdfs mover knows the hot data have come to cold , I don't need to tell it what exactly files/dirs need to be move now ? 
Of course I should tell it what files/dirs need to monitoring.

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 
While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

>>>It use algorithm like LRU、LFU ?
It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

Regards,
Rakesh


On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:
Thanks a lot Rakesh. 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?





Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

kevin-5
Ok,thanks.It means that I need to decide which data is hot and which time it is cold, then change it storage policy and tell  'Mover tool'  to move it.

2016-07-20 14:29 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
Based on storage policy the data from hot storage will be moved to cold storage. The storage policy defines the number of replicas to be located on each storage type. It is possible to change the storage policy on a directory(for example: HOT to COLD) and then invoke 'Mover tool' on that directory to make the policy effective. One can set/change the storage policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>". After setting the new policy, you need to run the tool, then it identifies the replicas to be moved based on the storage policy information, and schedules the movement between source and destination data nodes to satisfy the policy. Internally, the tool is comparing the 'storage type' of a block in order to fulfill the 'storage policy' requirement. 

Probably you can refer https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html to know more about storage types, storage policies and hdfs commands. Hope this helps.

Rakesh

On Wed, Jul 20, 2016 at 10:30 AM, kevin <[hidden email]> wrote:
Thanks again. "automatically" what I mean is the hdfs mover knows the hot data have come to cold , I don't need to tell it what exactly files/dirs need to be move now ? 
Of course I should tell it what files/dirs need to monitoring.

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 
While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

>>>It use algorithm like LRU、LFU ?
It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

Regards,
Rakesh


On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:
Thanks a lot Rakesh. 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRU、LFU ?

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:
>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

Regards,
Rakesh

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:
I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?






Reply | Threaded
Open this post in threaded view
|

RE: About Archival Storage

Zheng, Kai

There was some related discussion about this in https://issues.apache.org/jira/browse/HDFS-7343 where files and blocks temperature will be measured across the cluster and used to determine when and how to set storage polices accordingly.

 

The effort was suspended somehow because the contributors are working on HDFS erasure coding feature. It will be revived soon and feedback are welcome!

 

Regards,

Kai

 

From: kevin [mailto:[hidden email]]
Sent: Wednesday, July 20, 2016 2:48 PM
To: Rakesh Radhakrishnan <[hidden email]>
Cc: user.hadoop <[hidden email]>
Subject: Re: About Archival Storage

 

Ok,thanks.It means that I need to decide which data is hot and which time it is cold, then change it storage policy and tell  'Mover tool'  to move it.

 

2016-07-20 14:29 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

Based on storage policy the data from hot storage will be moved to cold storage. The storage policy defines the number of replicas to be located on each storage type. It is possible to change the storage policy on a directory(for example: HOT to COLD) and then invoke 'Mover tool' on that directory to make the policy effective. One can set/change the storage policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>". After setting the new policy, you need to run the tool, then it identifies the replicas to be moved based on the storage policy information, and schedules the movement between source and destination data nodes to satisfy the policy. Internally, the tool is comparing the 'storage type' of a block in order to fulfill the 'storage policy' requirement. 

 

Probably you can refer https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html to know more about storage types, storage policies and hdfs commands. Hope this helps.

 

Rakesh

 

On Wed, Jul 20, 2016 at 10:30 AM, kevin <[hidden email]> wrote:

Thanks again. "automatically" what I mean is the hdfs mover knows the hot data have come to cold , I don't need to tell it what exactly files/dirs need to be move now ? 

Of course I should tell it what files/dirs need to monitoring.

 

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 

While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

 

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

 

>>>It use algorithm like LRULFU ?

It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

 

Regards,

Rakesh

 

 

On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:

Thanks a lot Rakesh. 

 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRULFU ?

 

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

 

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

 

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

 

Regards,

Rakesh

 

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:

I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

 

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

 

 

 

 

 

 

Reply | Threaded
Open this post in threaded view
|

Re: About Archival Storage

kevin-5
Thanks a lot.

2016-07-20 15:20 GMT+08:00 Zheng, Kai <[hidden email]>:

There was some related discussion about this in https://issues.apache.org/jira/browse/HDFS-7343 where files and blocks temperature will be measured across the cluster and used to determine when and how to set storage polices accordingly.

 

The effort was suspended somehow because the contributors are working on HDFS erasure coding feature. It will be revived soon and feedback are welcome!

 

Regards,

Kai

 

From: kevin [mailto:[hidden email]]
Sent: Wednesday, July 20, 2016 2:48 PM
To: Rakesh Radhakrishnan <[hidden email]>
Cc: user.hadoop <[hidden email]>
Subject: Re: About Archival Storage

 

Ok,thanks.It means that I need to decide which data is hot and which time it is cold, then change it storage policy and tell  'Mover tool'  to move it.

 

2016-07-20 14:29 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

Based on storage policy the data from hot storage will be moved to cold storage. The storage policy defines the number of replicas to be located on each storage type. It is possible to change the storage policy on a directory(for example: HOT to COLD) and then invoke 'Mover tool' on that directory to make the policy effective. One can set/change the storage policy via HDFSCommand, "hdfs storagepolicies -setStoragePolicy -path <path> -policy <policy>". After setting the new policy, you need to run the tool, then it identifies the replicas to be moved based on the storage policy information, and schedules the movement between source and destination data nodes to satisfy the policy. Internally, the tool is comparing the 'storage type' of a block in order to fulfill the 'storage policy' requirement. 

 

Probably you can refer https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html to know more about storage types, storage policies and hdfs commands. Hope this helps.

 

Rakesh

 

On Wed, Jul 20, 2016 at 10:30 AM, kevin <[hidden email]> wrote:

Thanks again. "automatically" what I mean is the hdfs mover knows the hot data have come to cold , I don't need to tell it what exactly files/dirs need to be move now ? 

Of course I should tell it what files/dirs need to monitoring.

 

2016-07-20 12:35 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

>>>I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? 

While running the tool, it reads the argument and get the separated list of hdfs files/dirs to migrate. Then it periodically scans these files in HDFS to check if the block placement satisfies the storage policy, if not satisfied it moves the replicas to a different storage type in order to fulfill the storage policy requirement. This cycle continues until it hits an error or no blocks to move etc. Could you please tell me, what do you meant by "automatically" ? FYI, HDFS-10285 is proposing an idea to introduce a daemon thread in Namenode to track the storage movements set by APIs from clients. This Daemon thread named as StoragePolicySatisfier(SPS) serves something similar to ReplicationMonitor. If interested you can read the https://goo.gl/NA5EY0 proposal/idea and welcome feedback.

 

Sleep time between each cycle is, ('dfs.heartbeat.interval' * 2000) + ('dfs.namenode.replication.interval' * 1000) milliseconds;

 

>>>It use algorithm like LRULFU ?

It will simply iterating over the lists in the order of files/dirs given to this tool as an argument. afaik, its just maintains the order mentioned by the user. 

 

Regards,

Rakesh

 

 

On Wed, Jul 20, 2016 at 7:05 AM, kevin <[hidden email]> wrote:

Thanks a lot Rakesh. 

 

I have another question is , hdfs mover (A New Data Migration Tool ) know when to move data from hot to cold  automatically ? It use algorithm like LRULFU ?

 

2016-07-19 19:55 GMT+08:00 Rakesh Radhakrishnan <[hidden email]>:

>>>>Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?

 

The idea of Lazy_Persist policy is, while writing blocks, one replica will be placed in memory first and then it is lazily persisted into DISK. It doesn't means that, you are not allowed to configure dfs.replication > 1. If 'dfs.replication' is configured > 1 then the first replica will be placed in RAM_DISK and all the other replicas (n-1) will be written to the DISK. Here the (n-1) replicas will have the overhead of pipeline replication over the network and the DISK write latency on the write hot path. So you will not get better performance results.

 

IIUC, for getting memory latency benefits, it is recommended to use replication=1. In this way, applications should be able to perform single replica writes to a local DN with low latency. HDFS will store block data in memory and lazily save it to disk avoiding incurring disk write latency on the hot path. By writing to local memory we can also avoid checksum computation on the hot path.

 

Regards,

Rakesh

 

On Tue, Jul 19, 2016 at 3:25 PM, kevin <[hidden email]> wrote:

I don't quite understand :"Note that the Lazy_Persist policy is useful only for single replica blocks. For blocks with more than one replicas, all the replicas will be written to DISK since writing only one of the replicas to RAM_DISK does not improve the overall performance."  

 

Is that mean I should config dfs.replication with 1 ?  if more than one I should not use Lazy_Persist  policies ?