Securely Transfering Data on Container Launch

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Securely Transfering Data on Container Launch

Tim Likarish

I had a question about writing a Yarn application.  Is it possible to transfer data from the yarn client to the application master and each subsequent container?  Basically I want to be able to start my application with some private data on an edge node and have the data copied to the AM and any container it launches using a secure method.

One thing we're considering is using LocalResource, which seems to deviate for security reasons if the LocalResourceVisibility is set to private and the NodeManager is set to secure mode. The article also mentions: "Each PublicLocalizer uses credentials passed as part of ContainerLaunchContext to securely copy the resources from the remote file-system." [1]  However, I'm wondering how secure is this method of transferring data? Can I trust that any data I send marked as private is guaranteed only to be accessible by the user of my application?

Any other recommendations for how I might accomplish this goal?  

At the moment, the data is being manually copied to each node and protected with posix permissions, but would like to improve this process.