[DISCUSS] official docker image(s) for hadoop

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] official docker image(s) for hadoop

Marton, Elek

TL;DR: I propose to create official hadoop images and upload them to the
dockerhub.

GOAL/SCOPE: I would like improve the existing documentation with
easy-to-use docker based recipes to start hadoop clusters with various
configuration.

The images also could be used to test experimental features. For example
ozone could be tested easily with these compose file and configuration:

https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml

I would like to create separated example compose files for federation,
ha, metrics usage, etc. to make it easier to try out and understand the
features.

CONTEXT: There is an existing Jira
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images
(multiple types, in a flexible way). If no objections, I will create a
separated issue to create simplified docker images for rapid prototyping
and investigating new features. And register the branch to the dockerhub
to create the images automatically.

MY BACKGROUND: I am working with docker based hadoop/spark clusters
quite a while and run them succesfully in different environments
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is
available from here: https://github.com/flokkr but they could handle
more complex use cases (eg. instrumenting java processes with btrace, or
read/reload configuration from consul).
  And IMHO in the official hadoop documentation it’s better to suggest
to use official apache docker images and not external ones (which could
be changed).

Please let me know if you have any comments.

Marton

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] official docker image(s) for hadoop

Bharat Viswanadham
+1 (non-binding)
It would be really nice to have Docker to try different features of Hadoop (like HA, Federation Enabled, Erasure coding…), which will helpful for both developers and users.


Thanks,
Bharat


On 9/13/17, 11:31 AM, "Eric Badger" <[hidden email]> wrote:

    +1 definitely think an official Hadoop docker image (possibly 1 per major
    or minor release) would be a positive both for contributors and for users
    of Hadoop.
   
    Eric
   
    On Wed, Sep 13, 2017 at 1:19 PM, Wangda Tan <[hidden email]> wrote:
   
    > +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
    > super helpful!
    >
    > Thanks,
    > Wangda
    >
    > On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
    > [hidden email]> wrote:
    >
    > > Marton, thank you for working on this. I think Official Docker images for
    > > Hadoop would be very useful for a lot of reasons. I think that it is
    > better
    > > to have a coordinated effort with production ready base images with
    > > dependent images for prototyping. Does anyone else have an opinion about
    > > this?
    > >
    > > Thank you,
    > > Miklos
    > >
    > > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek <[hidden email]> wrote:
    > >
    > > >
    > > > TL;DR: I propose to create official hadoop images and upload them to
    > the
    > > > dockerhub.
    > > >
    > > > GOAL/SCOPE: I would like improve the existing documentation with
    > > > easy-to-use docker based recipes to start hadoop clusters with various
    > > > configuration.
    > > >
    > > > The images also could be used to test experimental features. For
    > example
    > > > ozone could be tested easily with these compose file and configuration:
    > > >
    > > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
    > > >
    > > > Or even the configuration could be included in the compose file:
    > > >
    > > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
    > > > ker-compose.yaml
    > > >
    > > > I would like to create separated example compose files for federation,
    > > ha,
    > > > metrics usage, etc. to make it easier to try out and understand the
    > > > features.
    > > >
    > > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
    > > > /browse/HADOOP-13397
    > > > But it’s about a tool to generate production quality docker images
    > > > (multiple types, in a flexible way). If no objections, I will create a
    > > > separated issue to create simplified docker images for rapid
    > prototyping
    > > > and investigating new features. And register the branch to the
    > dockerhub
    > > to
    > > > create the images automatically.
    > > >
    > > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
    > quite
    > > > a while and run them succesfully in different environments (kubernetes,
    > > > docker-swarm, nomad-based scheduling, etc.) My work is available from
    > > here:
    > > > https://github.com/flokkr but they could handle more complex use cases
    > > > (eg. instrumenting java processes with btrace, or read/reload
    > > configuration
    > > > from consul).
    > > >  And IMHO in the official hadoop documentation it’s better to suggest
    > to
    > > > use official apache docker images and not external ones (which could be
    > > > changed).
    > > >
    > > > Please let me know if you have any comments.
    > > >
    > > > Marton
    > > >
    > > > ---------------------------------------------------------------------
    > > > To unsubscribe, e-mail: [hidden email]
    > > > For additional commands, e-mail: [hidden email]
    > > >
    > > >
    > >
    >
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]