Where's official Docker image for Hadoop?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Where's official Docker image for Hadoop?

Klaus Ma

Hi team,


Does anyone know where's official docker images? If not, I'd like to contribute the Dockefile for it.

BTW, do we have official docker hub account for Hadoop?


If any suggestion, please let me know.


----

Da (Klaus), Ma (马达), PMP®| Software Architect
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | [hidden email] | http://k82.me

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Roman Shaposhnik-3
On Mon, Jul 18, 2016 at 5:34 PM, Klaus Ma <[hidden email]> wrote:
> Hi team,
>
>
> Does anyone know where's official docker images? If not, I'd like to
> contribute the Dockefile for it.

I am just curious, what's your use case?

Also, you may want to look at the following "prior art" in the area
of Hadoop/Docker:
    http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
    https://github.com/trifacta/floating-elephants

> BTW, do we have official docker hub account for Hadoop?

It was suggested that it may be useful for ASF to have an official account,
but nothing materialized AFAIK.

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Klaus Ma

I'd like to deploy YARN by Kubernetes.


I built docker images with Apache Hadoop, and I'd like to contribute it into hadoop source if not. It'll be great if Hadoop have an official place for those images.


----

Da (Klaus), Ma (马达), PMP®| Software Architect
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | [hidden email] | http://k82.me


From: [hidden email] <[hidden email]> on behalf of Roman Shaposhnik <[hidden email]>
Sent: Tuesday, July 19, 2016 12:57:40 AM
To: Klaus Ma
Cc: [hidden email]
Subject: Re: Where's official Docker image for Hadoop?
 
On Mon, Jul 18, 2016 at 5:34 PM, Klaus Ma <[hidden email]> wrote:
> Hi team,
>
>
> Does anyone know where's official docker images? If not, I'd like to
> contribute the Dockefile for it.

I am just curious, what's your use case?

Also, you may want to look at the following "prior art" in the area
of Hadoop/Docker:
    http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
    https://github.com/trifacta/floating-elephants

> BTW, do we have official docker hub account for Hadoop?

It was suggested that it may be useful for ASF to have an official account,
but nothing materialized AFAIK.

Thanks,
Roman.
Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Deepak Vohra-2
In reply to this post by Klaus Ma
The   cloudera/quickstart is the Docker image for Hadoop.

https://hub.docker.com/r/cloudera/quickstart/

Also refer,
http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html
http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/


--------------------------------------------
On Mon, 7/18/16, Roman Shaposhnik <[hidden email]> wrote:

 Subject: Re: Where's official Docker image for Hadoop?
 To: "Klaus Ma" <[hidden email]>
 Cc: "[hidden email]" <[hidden email]>
 Received: Monday, July 18, 2016, 5:57 PM
 
 On Mon, Jul 18, 2016 at 5:34 PM,
 Klaus Ma <[hidden email]>
 wrote:
 > Hi team,
 >
 >
 > Does anyone know where's official docker images? If
 not, I'd like to
 > contribute the Dockefile for it.
 
 I am just curious, what's your use case?
 
 Also, you may want to look at the following "prior art" in
 the area
 of Hadoop/Docker:
     http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
     https://github.com/trifacta/floating-elephants
 
 > BTW, do we have official docker hub account for
 Hadoop?
 
 It was suggested that it may be useful for ASF to have an
 official account,
 but nothing materialized AFAIK.
 
 Thanks,
 Roman.
 
 ---------------------------------------------------------------------
 To unsubscribe, e-mail: [hidden email]
 For additional commands, e-mail: [hidden email]
 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Deepak Vohra-2
In reply to this post by Klaus Ma
A custom implementation would have to be developed using some container orchestration service such as Kubernetes. Create a cluster of Pods (container sets) with different daemons running in different Pods and scale the Pods. For example, start ResourceManager on one instance and NodeManager on multiple instances in the cluster. The following blog runs a cluster with NodeManager on multiple instances. Kubernetes cluster manager could be added.

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm

The Docker image used is https://hub.docker.com/r/sequenceiq/hadoop-docker/

But any Docker image could be used.


--------------------------------------------
On Mon, 7/18/16, Klaus Ma <[hidden email]> wrote:

 Subject: Re: Where's official Docker image for Hadoop?
 To: "Roman Shaposhnik" <[hidden email]>, "Deepak Vohra" <[hidden email]>
 Cc: "[hidden email]" <[hidden email]>
 Received: Monday, July 18, 2016, 7:00 PM
 
 #yiv9491280873
 #yiv9491280873 -- .yiv9491280873EmailQuote
 {margin-left:1pt;padding-left:4pt;border-left:#800000 2px
 solid;}#yiv9491280873
 
 #yiv9491280873 #yiv9491280873 --
 p
  {margin-top:0;margin-bottom:0;}
 #yiv9491280873
 
 
 Thanks for your info.
 
 
 
 It seems all daemon are running in one container in
 cloudera/quickstart; I'd like to run resourcemanager,
 nodemanager in different containter, so I can scale
 nodemanager out.
 
 
 
 ----
 
 
 
 
 
 
 
 Da (Klaus), Ma (马达), PMP®|
 Software Architect
 
 Platform DCOS Development &
 Support, STG, IBM GCG
 
 +86-10-8245 4084 |
 [hidden email] | http://k82.me
 
 
 
 
 
 From: Deepak Vohra
 <[hidden email]>
 
 Sent: Tuesday, July 19, 2016 1:44:34 AM
 
 To: Roman Shaposhnik
 
 Cc: [hidden email]
 
 Subject: Re: Where's official Docker image for
 Hadoop?
  
 
 
 
 The   cloudera/quickstart
 is the Docker image for Hadoop.
 
 
 
 https://hub.docker.com/r/cloudera/quickstart/
 
 
 
 Also refer,
 
 http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html
 
 http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
 
 
 
 
 
 --------------------------------------------
 
 On Mon, 7/18/16, Roman Shaposhnik
 <[hidden email]> wrote:
 
 
 
  Subject: Re: Where's official Docker image for
 Hadoop?
 
  To: "Klaus Ma"
 <[hidden email]>
 
  Cc: "[hidden email]"
 <[hidden email]>
 
  Received: Monday, July 18, 2016, 5:57 PM
 
  
 
  On Mon, Jul 18, 2016 at 5:34 PM,
 
  Klaus Ma <[hidden email]>
 
  wrote:
 
  > Hi team,
 
  >
 
  >
 
  > Does anyone know where's official docker images?
 If
 
  not, I'd like to
 
  > contribute the Dockefile for it.
 
  
 
  I am just curious, what's your use case?
 
  
 
  Also, you may want to look at the following "prior
 art" in
 
  the area
 
  of Hadoop/Docker:
 
     
 http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
 
      https://github.com/trifacta/floating-elephants
 
  
 
  > BTW, do we have official docker hub account for
 
  Hadoop?
 
  
 
  It was suggested that it may be useful for ASF to have
 an
 
  official account,
 
  but nothing materialized AFAIK.
 
  
 
  Thanks,
 
  Roman.
 
  
 
  ---------------------------------------------------------------------
 
  To unsubscribe, e-mail:
 [hidden email]
 
  For additional commands, e-mail:
 [hidden email]
 
  
 
 
 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Tsuyoshi Ozawa-3
Hi Klaus,

Thanks for telling us the request. Currently, the official docker
image of Apache Hadoop is not available as Roman mentioned. I will
raise this request as discussion.

Thanks,
- Tsuyoshi

On Tue, Jul 19, 2016 at 12:29 PM, Deepak Vohra
<[hidden email]> wrote:

> A custom implementation would have to be developed using some container orchestration service such as Kubernetes. Create a cluster of Pods (container sets) with different daemons running in different Pods and scale the Pods. For example, start ResourceManager on one instance and NodeManager on multiple instances in the cluster. The following blog runs a cluster with NodeManager on multiple instances. Kubernetes cluster manager could be added.
>
> http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm
>
> The Docker image used is https://hub.docker.com/r/sequenceiq/hadoop-docker/
>
> But any Docker image could be used.
>
>
> --------------------------------------------
> On Mon, 7/18/16, Klaus Ma <[hidden email]> wrote:
>
>  Subject: Re: Where's official Docker image for Hadoop?
>  To: "Roman Shaposhnik" <[hidden email]>, "Deepak Vohra" <[hidden email]>
>  Cc: "[hidden email]" <[hidden email]>
>  Received: Monday, July 18, 2016, 7:00 PM
>
>  #yiv9491280873
>  #yiv9491280873 -- .yiv9491280873EmailQuote
>  {margin-left:1pt;padding-left:4pt;border-left:#800000 2px
>  solid;}#yiv9491280873
>
>  #yiv9491280873 #yiv9491280873 --
>  p
>         {margin-top:0;margin-bottom:0;}
>  #yiv9491280873
>
>
>  Thanks for your info.
>
>
>
>  It seems all daemon are running in one container in
>  cloudera/quickstart; I'd like to run resourcemanager,
>  nodemanager in different containter, so I can scale
>  nodemanager out.
>
>
>
>  ----
>
>
>
>
>
>
>
>  Da (Klaus), Ma (马达), PMP®|
>  Software Architect
>
>  Platform DCOS Development &
>  Support, STG, IBM GCG
>
>  +86-10-8245 4084 |
>  [hidden email] | http://k82.me
>
>
>
>
>
>  From: Deepak Vohra
>  <[hidden email]>
>
>  Sent: Tuesday, July 19, 2016 1:44:34 AM
>
>  To: Roman Shaposhnik
>
>  Cc: [hidden email]
>
>  Subject: Re: Where's official Docker image for
>  Hadoop?
>
>
>
>
>  The   cloudera/quickstart
>  is the Docker image for Hadoop.
>
>
>
>  https://hub.docker.com/r/cloudera/quickstart/
>
>
>
>  Also refer,
>
>  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html
>
>  http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
>
>
>
>
>
>  --------------------------------------------
>
>  On Mon, 7/18/16, Roman Shaposhnik
>  <[hidden email]> wrote:
>
>
>
>   Subject: Re: Where's official Docker image for
>  Hadoop?
>
>   To: "Klaus Ma"
>  <[hidden email]>
>
>   Cc: "[hidden email]"
>  <[hidden email]>
>
>   Received: Monday, July 18, 2016, 5:57 PM
>
>
>
>   On Mon, Jul 18, 2016 at 5:34 PM,
>
>   Klaus Ma <[hidden email]>
>
>   wrote:
>
>   > Hi team,
>
>   >
>
>   >
>
>   > Does anyone know where's official docker images?
>  If
>
>   not, I'd like to
>
>   > contribute the Dockefile for it.
>
>
>
>   I am just curious, what's your use case?
>
>
>
>   Also, you may want to look at the following "prior
>  art" in
>
>   the area
>
>   of Hadoop/Docker:
>
>
>  http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
>
>       https://github.com/trifacta/floating-elephants
>
>
>
>   > BTW, do we have official docker hub account for
>
>   Hadoop?
>
>
>
>   It was suggested that it may be useful for ASF to have
>  an
>
>   official account,
>
>   but nothing materialized AFAIK.
>
>
>
>   Thanks,
>
>   Roman.
>
>
>
>   ---------------------------------------------------------------------
>
>   To unsubscribe, e-mail:
>  [hidden email]
>
>   For additional commands, e-mail:
>  [hidden email]
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Klaus Ma
Hi Tsuyoshi,

I have a set of dockerfile at https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn to for Apache YARN/HDFS; and I’d like to contribute it to upstream if possible.

Would you also keep me in the loop if any discussion?

Thanks
Klaus


On Jul 19, 2016, at 15:32, Tsuyoshi Ozawa <[hidden email]> wrote:

Hi Klaus,

Thanks for telling us the request. Currently, the official docker
image of Apache Hadoop is not available as Roman mentioned. I will
raise this request as discussion.

Thanks,
- Tsuyoshi

On Tue, Jul 19, 2016 at 12:29 PM, Deepak Vohra
<[hidden email]> wrote:
A custom implementation would have to be developed using some container orchestration service such as Kubernetes. Create a cluster of Pods (container sets) with different daemons running in different Pods and scale the Pods. For example, start ResourceManager on one instance and NodeManager on multiple instances in the cluster. The following blog runs a cluster with NodeManager on multiple instances. Kubernetes cluster manager could be added.

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm

The Docker image used is https://hub.docker.com/r/sequenceiq/hadoop-docker/

But any Docker image could be used.


--------------------------------------------
On Mon, 7/18/16, Klaus Ma <[hidden email]> wrote:

Subject: Re: Where's official Docker image for Hadoop?
To: "Roman Shaposhnik" <[hidden email]>, "Deepak Vohra" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Received: Monday, July 18, 2016, 7:00 PM

#yiv9491280873
#yiv9491280873 -- .yiv9491280873EmailQuote
{margin-left:1pt;padding-left:4pt;border-left:#800000 2px
solid;}#yiv9491280873

#yiv9491280873 #yiv9491280873 --
p
       {margin-top:0;margin-bottom:0;}
#yiv9491280873


Thanks for your info.



It seems all daemon are running in one container in
cloudera/quickstart; I'd like to run resourcemanager,
nodemanager in different containter, so I can scale
nodemanager out.



----







Da (Klaus), Ma (马达), PMP®|
Software Architect

Platform DCOS Development &
Support, STG, IBM GCG

+86-10-8245 4084 |
[hidden email] | http://k82.me





From: Deepak Vohra
<[hidden email]>

Sent: Tuesday, July 19, 2016 1:44:34 AM

To: Roman Shaposhnik

Cc: [hidden email]

Subject: Re: Where's official Docker image for
Hadoop?




The   cloudera/quickstart
is the Docker image for Hadoop.



https://hub.docker.com/r/cloudera/quickstart/



Also refer,

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html

http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/





--------------------------------------------

On Mon, 7/18/16, Roman Shaposhnik
<[hidden email]> wrote:



 Subject: Re: Where's official Docker image for
Hadoop?

 To: "Klaus Ma"
<[hidden email]>

 Cc: "[hidden email]"
<[hidden email]>

 Received: Monday, July 18, 2016, 5:57 PM



 On Mon, Jul 18, 2016 at 5:34 PM,

 Klaus Ma <[hidden email]>

 wrote:

Hi team,





Does anyone know where's official docker images?
If

 not, I'd like to

contribute the Dockefile for it.



 I am just curious, what's your use case?



 Also, you may want to look at the following "prior
art" in

 the area

 of Hadoop/Docker:


http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop

     https://github.com/trifacta/floating-elephants



BTW, do we have official docker hub account for

 Hadoop?



 It was suggested that it may be useful for ASF to have
an

 official account,

 but nothing materialized AFAIK.



 Thanks,

 Roman.



 ---------------------------------------------------------------------

 To unsubscribe, e-mail:
[hidden email]

 For additional commands, e-mail:
[hidden email]






---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Deepak Vohra-2
In reply to this post by Klaus Ma
What is meant by official Hadoop image? Hadoop has several distributions such as HortonWorks, Cloudera and MapR and they do provide a Docker image.


1. Official image from Cloudera is the quickstart image.
https://hub.docker.com/r/cloudera/quickstart/

2.  From HortonWorks sequenceiq
https://hub.docker.com/r/sequenceiq/hadoop-docker/

3. MapR provides the mapr-sandbox-base
https://hub.docker.com/r/maprtech/mapr-sandbox-base/
--------------------------------------------
On Tue, 7/19/16, Klaus Ma <[hidden email]> wrote:

 Subject: Re: Where's official Docker image for Hadoop?
 To: "Tsuyoshi Ozawa" <[hidden email]>
 Cc: "Deepak Vohra" <[hidden email]>, "Klaus Ma" <[hidden email]>, "[hidden email]" <[hidden email]>
 Received: Tuesday, July 19, 2016, 5:49 AM
 
 
 Hi Tsuyoshi,
 
 
 
 I have a set of dockerfile at
 https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn to
 for Apache YARN/HDFS; and I’d like to contribute it to
 upstream if possible.
 
 
 
 Would you also keep me in the
 loop if any discussion?
 
 
 
 Thanks
 Klaus
 
 
 
 
 
 
 
 On
 Jul 19, 2016, at 15:32, Tsuyoshi Ozawa <[hidden email]>
 wrote:
 
 
 
 Hi Klaus,
 
 
 
 Thanks for telling us the request. Currently, the official
 docker
 
 image of Apache Hadoop is not available as Roman mentioned.
 I will
 
 raise this request as discussion.
 
 
 
 Thanks,
 
 - Tsuyoshi
 
 
 
 On Tue, Jul 19, 2016 at 12:29 PM, Deepak Vohra
 
 <[hidden email]>
 wrote:
 
 A custom
 implementation would have to be developed using some
 container orchestration service such as Kubernetes. Create a
 cluster of Pods (container sets) with different daemons
 running in different Pods and scale the Pods.
  For example, start ResourceManager on one instance and
 NodeManager on multiple instances in the cluster. The
 following blog runs a cluster with NodeManager on multiple
 instances. Kubernetes cluster manager could be added.
 
 
 
 http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm
 
 
 
 The Docker image used is
 https://hub.docker.com/r/sequenceiq/hadoop-docker/
 
 
 
 But any Docker image could be used.
 
 
 
 
 
 --------------------------------------------
 
 On Mon, 7/18/16, Klaus Ma <[hidden email]>
 wrote:
 
 
 
 Subject: Re: Where's official Docker image for
 Hadoop?
 
 To: "Roman Shaposhnik"
 <[hidden email]>, "Deepak Vohra"
 <[hidden email]>
 
 Cc: "[hidden email]"
 <[hidden email]>
 
 Received: Monday, July 18, 2016, 7:00 PM
 
 
 
 #yiv9491280873
 
 #yiv9491280873 -- .yiv9491280873EmailQuote
 
 {margin-left:1pt;padding-left:4pt;border-left:#800000 2px
 
 solid;}#yiv9491280873
 
 
 
 #yiv9491280873 #yiv9491280873 --
 
 p
 
        {margin-top:0;margin-bottom:0;}
 
 #yiv9491280873
 
 
 
 
 
 Thanks for your info.
 
 
 
 
 
 
 
 It seems all daemon are running in one container in
 
 cloudera/quickstart; I'd like to run resourcemanager,
 
 nodemanager in different containter, so I can scale
 
 nodemanager out.
 
 
 
 
 
 
 
 ----
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Da (Klaus), Ma (马达), PMP®|
 
 Software Architect
 
 
 
 Platform DCOS Development &
 
 Support, STG, IBM GCG
 
 
 
 +86-10-8245 4084 |
 
 [hidden email] | http://k82.me
 
 
 
 
 
 
 
 
 
 
 
 From: Deepak Vohra
 
 <[hidden email]>
 
 
 
 Sent: Tuesday, July 19, 2016 1:44:34 AM
 
 
 
 To: Roman Shaposhnik
 
 
 
 Cc: [hidden email]
 
 
 
 Subject: Re: Where's official Docker image for
 
 Hadoop?
 
 
 
 
 
 
 
 
 
 The   cloudera/quickstart
 
 is the Docker image for Hadoop.
 
 
 
 
 
 
 
 https://hub.docker.com/r/cloudera/quickstart/
 
 
 
 
 
 
 
 Also refer,
 
 
 
 http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html
 
 
 
 http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
 
 
 
 
 
 
 
 
 
 
 
 --------------------------------------------
 
 
 
 On Mon, 7/18/16, Roman Shaposhnik
 
 <[hidden email]> wrote:
 
 
 
 
 
 
 
  Subject: Re: Where's official Docker image for
 
 Hadoop?
 
 
 
  To: "Klaus Ma"
 
 <[hidden email]>
 
 
 
  Cc: "[hidden email]"
 
 <[hidden email]>
 
 
 
  Received: Monday, July 18, 2016, 5:57 PM
 
 
 
 
 
 
 
  On Mon, Jul 18, 2016 at 5:34 PM,
 
 
 
  Klaus Ma <[hidden email]>
 
 
 
  wrote:
 
 
 
 Hi team,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Does anyone
 know where's official docker images?
 
 
 If
 
 
 
  not, I'd like to
 
 
 
 contribute the
 Dockefile for it.
 
 
 
 
 
 
 
 
  I am just curious, what's your use case?
 
 
 
 
 
 
 
  Also, you may want to look at the following "prior
 
 art" in
 
 
 
  the area
 
 
 
  of Hadoop/Docker:
 
 
 
 
 
 http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
 
 
 
      https://github.com/trifacta/floating-elephants
 
 
 
 
 
 
 
 BTW, do we
 have official docker hub account for
 
 
 
 
  Hadoop?
 
 
 
 
 
 
 
  It was suggested that it may be useful for ASF to have
 
 an
 
 
 
  official account,
 
 
 
  but nothing materialized AFAIK.
 
 
 
 
 
 
 
  Thanks,
 
 
 
  Roman.
 
 
 
 
 
 
 
  ---------------------------------------------------------------------
 
 
 
  To unsubscribe, e-mail:
 
 [hidden email]
 
 
 
  For additional commands, e-mail:
 
 [hidden email]
 
 
 
 
 
 
 
 
 
 
 
 
 
 ---------------------------------------------------------------------
 
 To unsubscribe, e-mail:
 [hidden email]
 
 For additional commands, e-mail:
 [hidden email]
 
 
 
 
 
 
 
 
 
 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Klaus Ma
I means community version; those docker image are provided by vendors.

——
Da (Klaus) Ma (马达), PMP® | Software  Architect
IBM Spectrum, STG, IBM GCG
+86-10-8245 4084 | [hidden email] | http://k82.me

On Jul 19, 2016, at 21:33, Deepak Vohra <[hidden email]> wrote:

What is meant by official Hadoop image? Hadoop has several distributions such as HortonWorks, Cloudera and MapR and they do provide a Docker image.


1. Official image from Cloudera is the quickstart image.
https://hub.docker.com/r/cloudera/quickstart/

2.  From HortonWorks sequenceiq
https://hub.docker.com/r/sequenceiq/hadoop-docker/

3. MapR provides the mapr-sandbox-base
https://hub.docker.com/r/maprtech/mapr-sandbox-base/
--------------------------------------------
On Tue, 7/19/16, Klaus Ma <[hidden email]> wrote:

Subject: Re: Where's official Docker image for Hadoop?
To: "Tsuyoshi Ozawa" <[hidden email]>
Cc: "Deepak Vohra" <[hidden email]>, "Klaus Ma" <[hidden email]>, "[hidden email]" <[hidden email]>
Received: Tuesday, July 19, 2016, 5:49 AM


Hi Tsuyoshi,



I have a set of dockerfile at
https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn to
for Apache YARN/HDFS; and I’d like to contribute it to
upstream if possible.



Would you also keep me in the
loop if any discussion?



Thanks
Klaus







On
Jul 19, 2016, at 15:32, Tsuyoshi Ozawa <[hidden email]>
wrote:



Hi Klaus,



Thanks for telling us the request. Currently, the official
docker

image of Apache Hadoop is not available as Roman mentioned.
I will

raise this request as discussion.



Thanks,

- Tsuyoshi



On Tue, Jul 19, 2016 at 12:29 PM, Deepak Vohra

<[hidden email]>
wrote:

A custom
implementation would have to be developed using some
container orchestration service such as Kubernetes. Create a
cluster of Pods (container sets) with different daemons
running in different Pods and scale the Pods.
 For example, start ResourceManager on one instance and
NodeManager on multiple instances in the cluster. The
following blog runs a cluster with NodeManager on multiple
instances. Kubernetes cluster manager could be added.



http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm



The Docker image used is
https://hub.docker.com/r/sequenceiq/hadoop-docker/



But any Docker image could be used.





--------------------------------------------

On Mon, 7/18/16, Klaus Ma <[hidden email]>
wrote:



Subject: Re: Where's official Docker image for
Hadoop?

To: "Roman Shaposhnik"
<[hidden email]>, "Deepak Vohra"
<[hidden email]>

Cc: "[hidden email]"
<[hidden email]>

Received: Monday, July 18, 2016, 7:00 PM



#yiv9491280873

#yiv9491280873 -- .yiv9491280873EmailQuote

{margin-left:1pt;padding-left:4pt;border-left:#800000 2px

solid;}#yiv9491280873



#yiv9491280873 #yiv9491280873 --

p

       {margin-top:0;margin-bottom:0;}

#yiv9491280873





Thanks for your info.







It seems all daemon are running in one container in

cloudera/quickstart; I'd like to run resourcemanager,

nodemanager in different containter, so I can scale

nodemanager out.







----















Da (Klaus), Ma (马达), PMP®|

Software Architect



Platform DCOS Development &

Support, STG, IBM GCG



+86-10-8245 4084 |

[hidden email] | http://k82.me











From: Deepak Vohra

<[hidden email]>



Sent: Tuesday, July 19, 2016 1:44:34 AM



To: Roman Shaposhnik



Cc: [hidden email]



Subject: Re: Where's official Docker image for

Hadoop?









The   cloudera/quickstart

is the Docker image for Hadoop.







https://hub.docker.com/r/cloudera/quickstart/







Also refer,



http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html



http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/











--------------------------------------------



On Mon, 7/18/16, Roman Shaposhnik

<[hidden email]> wrote:







 Subject: Re: Where's official Docker image for

Hadoop?



 To: "Klaus Ma"

<[hidden email]>



 Cc: "[hidden email]"

<[hidden email]>



 Received: Monday, July 18, 2016, 5:57 PM







 On Mon, Jul 18, 2016 at 5:34 PM,



 Klaus Ma <[hidden email]>



 wrote:



Hi team,














Does anyone
know where's official docker images?


If



 not, I'd like to



contribute the
Dockefile for it.








 I am just curious, what's your use case?







 Also, you may want to look at the following "prior

art" in



 the area



 of Hadoop/Docker:





http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop



     https://github.com/trifacta/floating-elephants







BTW, do we
have official docker hub account for




 Hadoop?







 It was suggested that it may be useful for ASF to have

an



 official account,



 but nothing materialized AFAIK.







 Thanks,



 Roman.







 ---------------------------------------------------------------------



 To unsubscribe, e-mail:

[hidden email]



 For additional commands, e-mail:

[hidden email]













---------------------------------------------------------------------

To unsubscribe, e-mail:
[hidden email]

For additional commands, e-mail:
[hidden email]











Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Deepak Vohra-2
In reply to this post by Klaus Ma

Even the Hadoop documentation refers to the HortonWorks Docker image sequenceiq/hadoop-docker.
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

Apache Hadoop develops the Hadoop software, not related technologies such as Docker image. But a Docker image could be developed using a Dockerfile that downloads and installs a Apache Hadoop distribution.
--------------------------------------------
On Tue, 7/19/16, Klaus Ma <[hidden email]> wrote:

 Subject: Re: Where's official Docker image for Hadoop?
 To: "Deepak Vohra" <[hidden email]>
 Cc: "[hidden email]" <[hidden email]>
 Received: Tuesday, July 19, 2016, 6:40 AM
 
 
 I means community version; those docker image are provided
 by vendors.
 
 
 
 
 
 
 
 ——
 
 Da (Klaus) Ma (马达), PMP® | Software  Architect
 
 IBM Spectrum, STG, IBM GCG
 
 +86-10-8245 4084 | [hidden email] | http://k82.me
 
 
 
 
 
 
 
 On Jul 19, 2016, at 21:33, Deepak
 Vohra <[hidden email]>
 wrote:
 
 
 
 What is meant by official Hadoop
 image? Hadoop has several distributions such as HortonWorks,
 Cloudera and MapR and they do provide a Docker image.
 
 
 
 
 
 1. Official image from Cloudera is the quickstart image.
 
 https://hub.docker.com/r/cloudera/quickstart/
 
 
 
 2.  From HortonWorks sequenceiq
 
 https://hub.docker.com/r/sequenceiq/hadoop-docker/
 
 
 
 3. MapR provides the mapr-sandbox-base
 
 https://hub.docker.com/r/maprtech/mapr-sandbox-base/
 
 --------------------------------------------
 
 On Tue, 7/19/16, Klaus Ma <[hidden email]>
 wrote:
 
 
 
 Subject: Re: Where's official Docker image for
 Hadoop?
 
 To: "Tsuyoshi Ozawa" <[hidden email]>
 
 Cc: "Deepak Vohra" <[hidden email]>,
 "Klaus Ma" <[hidden email]>,
 "[hidden email]"
 <[hidden email]>
 
 Received: Tuesday, July 19, 2016, 5:49 AM
 
 
 
 
 
 Hi Tsuyoshi,
 
 
 
 
 
 
 
 I have a set of dockerfile at
 
 https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn to
 
 for Apache YARN/HDFS; and I’d like to contribute it to
 
 upstream if possible.
 
 
 
 
 
 
 
 Would you also keep me in the
 
 loop if any discussion?
 
 
 
 
 
 
 
 Thanks
 
 Klaus
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 On
 
 Jul 19, 2016, at 15:32, Tsuyoshi Ozawa
 <[hidden email]>
 
 wrote:
 
 
 
 
 
 
 
 Hi Klaus,
 
 
 
 
 
 
 
 Thanks for telling us the request. Currently, the
 official
 
 docker
 
 
 
 image of Apache Hadoop is not available as Roman
 mentioned.
 
 I will
 
 
 
 raise this request as discussion.
 
 
 
 
 
 
 
 Thanks,
 
 
 
 - Tsuyoshi
 
 
 
 
 
 
 
 On Tue, Jul 19, 2016 at 12:29 PM, Deepak Vohra
 
 
 
 <[hidden email]>
 
 wrote:
 
 
 
 A custom
 
 implementation would have to be developed using some
 
 container orchestration service such as Kubernetes. Create
 a
 
 cluster of Pods (container sets) with different daemons
 
 running in different Pods and scale the Pods.
 
  For example, start ResourceManager on one instance and
 
 NodeManager on multiple instances in the cluster. The
 
 following blog runs a cluster with NodeManager on
 multiple
 
 instances. Kubernetes cluster manager could be added.
 
 
 
 
 
 
 
 http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_configuringthedockerinstanceandhdfstransparency.htm
 
 
 
 
 
 
 
 The Docker image used is
 
 https://hub.docker.com/r/sequenceiq/hadoop-docker/
 
 
 
 
 
 
 
 But any Docker image could be used.
 
 
 
 
 
 
 
 
 
 
 
 --------------------------------------------
 
 
 
 On Mon, 7/18/16, Klaus Ma <[hidden email]>
 
 wrote:
 
 
 
 
 
 
 
 Subject: Re: Where's official Docker image for
 
 Hadoop?
 
 
 
 To: "Roman Shaposhnik"
 
 <[hidden email]>, "Deepak Vohra"
 
 <[hidden email]>
 
 
 
 Cc: "[hidden email]"
 
 <[hidden email]>
 
 
 
 Received: Monday, July 18, 2016, 7:00 PM
 
 
 
 
 
 
 
 #yiv9491280873
 
 
 
 #yiv9491280873 -- .yiv9491280873EmailQuote
 
 
 
 {margin-left:1pt;padding-left:4pt;border-left:#800000 2px
 
 
 
 solid;}#yiv9491280873
 
 
 
 
 
 
 
 #yiv9491280873 #yiv9491280873 --
 
 
 
 p
 
 
 
        {margin-top:0;margin-bottom:0;}
 
 
 
 #yiv9491280873
 
 
 
 
 
 
 
 
 
 
 
 Thanks for your info.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 It seems all daemon are running in one container in
 
 
 
 cloudera/quickstart; I'd like to run resourcemanager,
 
 
 
 nodemanager in different containter, so I can scale
 
 
 
 nodemanager out.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 ----
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Da (Klaus), Ma (马达), PMP®|
 
 
 
 Software Architect
 
 
 
 
 
 
 
 Platform DCOS Development &
 
 
 
 Support, STG, IBM GCG
 
 
 
 
 
 
 
 +86-10-8245 4084 |
 
 
 
 [hidden email] | http://k82.me
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 From: Deepak Vohra
 
 
 
 <[hidden email]>
 
 
 
 
 
 
 
 Sent: Tuesday, July 19, 2016 1:44:34 AM
 
 
 
 
 
 
 
 To: Roman Shaposhnik
 
 
 
 
 
 
 
 Cc: [hidden email]
 
 
 
 
 
 
 
 Subject: Re: Where's official Docker image for
 
 
 
 Hadoop?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 The   cloudera/quickstart
 
 
 
 is the Docker image for Hadoop.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 https://hub.docker.com/r/cloudera/quickstart/
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Also refer,
 
 
 
 
 
 
 
 http://www.cloudera.com/documentation/enterprise/5-6-x/topics/quickstart_docker_container.html
 
 
 
 
 
 
 
 http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 --------------------------------------------
 
 
 
 
 
 
 
 On Mon, 7/18/16, Roman Shaposhnik
 
 
 
 <[hidden email]> wrote:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  Subject: Re: Where's official Docker image for
 
 
 
 Hadoop?
 
 
 
 
 
 
 
  To: "Klaus Ma"
 
 
 
 <[hidden email]>
 
 
 
 
 
 
 
  Cc: "[hidden email]"
 
 
 
 <[hidden email]>
 
 
 
 
 
 
 
  Received: Monday, July 18, 2016, 5:57 PM
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  On Mon, Jul 18, 2016 at 5:34 PM,
 
 
 
 
 
 
 
  Klaus Ma <[hidden email]>
 
 
 
 
 
 
 
  wrote:
 
 
 
 
 
 
 
 Hi team,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Does anyone
 
 know where's official docker images?
 
 
 
 
 
 If
 
 
 
 
 
 
 
  not, I'd like to
 
 
 
 
 
 
 
 contribute the
 
 Dockefile for it.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  I am just curious, what's your use case?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  Also, you may want to look at the following "prior
 
 
 
 art" in
 
 
 
 
 
 
 
  the area
 
 
 
 
 
 
 
  of Hadoop/Docker:
 
 
 
 
 
 
 
 
 
 
 
 http://www.slideshare.net/saintya/trend-micro-big-data-platform-and-apache-bigtop
 
 
 
 
 
 
 
      https://github.com/trifacta/floating-elephants
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 BTW, do we
 
 have official docker hub account for
 
 
 
 
 
 
 
 
 
  Hadoop?
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  It was suggested that it may be useful for ASF to have
 
 
 
 an
 
 
 
 
 
 
 
  official account,
 
 
 
 
 
 
 
  but nothing materialized AFAIK.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  Thanks,
 
 
 
 
 
 
 
  Roman.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  ---------------------------------------------------------------------
 
 
 
 
 
 
 
  To unsubscribe, e-mail:
 
 
 
 [hidden email]
 
 
 
 
 
 
 
  For additional commands, e-mail:
 
 
 
 [hidden email]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 ---------------------------------------------------------------------
 
 
 
 To unsubscribe, e-mail:
 
 [hidden email]
 
 
 
 For additional commands, e-mail:
 
 [hidden email]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Klaus Ma
HI Deepak,

This image still need to manually configure which did not meet the requirement. And I’d suggest Hadoop community provide a set of Dockerfile as example instead of vendor.

And where’s the dockerfile in source code? Here’s the output of 2.7.2.

Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ pwd
/Users/klaus/Workspace/hadoop-2.7.2-src
Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ find . | grep Dockerfile
Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$

If any comments, please let me know.

——
Da (Klaus) Ma (马达), PMP® | Software  Architect
IBM Spectrum, STG, IBM GCG
+86-10-8245 4084 | [hidden email] | http://k82.me

On Jul 19, 2016, at 21:55, Deepak Vohra <[hidden email]> wrote:

Apache Hadoop develops the Hadoop software, not related technologies such as Docker image. But a Docker image could be developed using a Dockerfile that downloads and installs a Apache Hadoop distribution.

Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Ravi Prakash-3
Would something like this be useful as a starting point? https://github.com/apache/hadoop/tree/trunk/dev-support/docker (this is checked into apache/trunk)

The DockerContainerExecutor was an alpha feature that didn't really get much traction and is not what you think it is. (If configured on the cluster, it enables users to launch yarn applications that spawn docker containers for tasks).

On Tue, Jul 19, 2016 at 5:05 PM, Klaus Ma <[hidden email]> wrote:
HI Deepak,

This image still need to manually configure which did not meet the requirement. And I’d suggest Hadoop community provide a set of Dockerfile as example instead of vendor.

And where’s the dockerfile in source code? Here’s the output of 2.7.2.

Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ pwd
/Users/klaus/Workspace/hadoop-2.7.2-src
Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ find . | grep Dockerfile
Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$

If any comments, please let me know.

——
Da (Klaus) Ma (马达), PMP® | Software  Architect
IBM Spectrum, STG, IBM GCG
<a href="tel:%2B86-10-8245%204084" value="+861082454084" target="_blank">+86-10-8245 4084 | [hidden email] | http://k82.me

On Jul 19, 2016, at 21:55, Deepak Vohra <[hidden email]> wrote:

Apache Hadoop develops the Hadoop software, not related technologies such as Docker image. But a Docker image could be developed using a Dockerfile that downloads and installs a Apache Hadoop distribution.


Reply | Threaded
Open this post in threaded view
|

Re: Where's official Docker image for Hadoop?

Sean Busbey
downstream users should rely on published releases and not things that
are still in source control.

Klaus, thank you for the feature suggestion. Would you mind getting
your request lodged in the project's JIRA? I think we'll have a better
idea on acceptability once you have a description and a patch to
review.

https://issues.apache.org/jira/browse/HADOOP

If you're unsure about the details still and would like to discuss
what preferences (if any) the project has, then an email to the
common-dev@hadoop mailing list would be a good starting point.

http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/

On Wed, Jul 20, 2016 at 2:07 PM, Ravi Prakash <[hidden email]> wrote:

> Would something like this be useful as a starting point?
> https://github.com/apache/hadoop/tree/trunk/dev-support/docker (this is
> checked into apache/trunk)
>
> The DockerContainerExecutor was an alpha feature that didn't really get much
> traction and is not what you think it is. (If configured on the cluster, it
> enables users to launch yarn applications that spawn docker containers for
> tasks).
>
> On Tue, Jul 19, 2016 at 5:05 PM, Klaus Ma <[hidden email]> wrote:
>>
>> HI Deepak,
>>
>> This image still need to manually configure which did not meet the
>> requirement. And I’d suggest Hadoop community provide a set of Dockerfile as
>> example instead of vendor.
>>
>> And where’s the dockerfile in source code? Here’s the output of 2.7.2.
>>
>> Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ pwd
>> /Users/klaus/Workspace/hadoop-2.7.2-src
>> Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$ find . | grep Dockerfile
>> Klauss-MacBook-Pro:hadoop-2.7.2-src klaus$
>>
>> If any comments, please let me know.
>>
>> ——
>> Da (Klaus) Ma (马达), PMP® | Software  Architect
>> IBM Spectrum, STG, IBM GCG
>> +86-10-8245 4084 | [hidden email] | http://k82.me
>>
>> On Jul 19, 2016, at 21:55, Deepak Vohra <[hidden email]> wrote:
>>
>> Apache Hadoop develops the Hadoop software, not related technologies such
>> as Docker image. But a Docker image could be developed using a Dockerfile
>> that downloads and installs a Apache Hadoop distribution.
>>
>>
>



--
busbey

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]