Re: Re: Any thoughts making Submarine a separate Apache project?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Re: Any thoughts making Submarine a separate Apache project?

dashuiguailuyun@gmail.com
+1 ,Good idea, we are very much looking forward to it.



[hidden email]
 
From: Szilard Nemeth
Date: 2019-07-17 14:55
To: runlin zhang
CC: Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea.
As Hadoop repository has already grown huge and contains many projects, I
think in general it's a good idea to separate projects in the early phase.
 
 
On Wed, Jul 17, 2019, 08:50 runlin zhang <[hidden email]> wrote:
 

> +1 ,That will be great !
>
> > 在 2019年7月10日,下午3:34,Xun Liu <[hidden email]> 写道:
> >
> > Hi all,
> >
> > This is Xun Liu contributing to the Submarine project for deep learning
> > workloads running with big data workloads together on Hadoop clusters.
> >
> > There are a bunch of integrations of Submarine to other projects are
> > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> step
> > of Submarine is going to integrate with more projects like Apache Arrow,
> > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > cases like model serving, notebook management, advanced training
> > optimizations (like auto parameter tuning, memory cache optimizations for
> > large datasets for training, etc.), and make it run on other platforms
> like
> > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> project
> > to Apache so we can put Submarine and TonY together to the same codebase
> > (Page #30.
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > ).
> >
> > This expands the scope of the original Submarine project in exciting new
> > ways. Toward that end, would it make sense to create a separate Submarine
> > project at Apache? This can make faster adoption of Submarine, and allow
> > Submarine to grow to a full-blown machine learning platform.
> >
> > There will be lots of technical details to work out, but any initial
> > thoughts on this?
> >
> > Best Regards,
> > Xun Liu
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: Any thoughts making Submarine a separate Apache project?

dashuiguailuyun@gmail.com
+1.Submarine is already in use at our company(贝壳找房) and is performing. Looking forward to the next step to provide more features



[hidden email]
 
From: Oliver Hu
Date: 2019-07-19 07:50
To: Jeff Zhang
CC: sid yu; Xun Liu; Hadoop Common; yarn-dev; Hdfs-dev; mapreduce-dev; submarine-dev
Subject: Re: Any thoughts making Submarine a separate Apache project?
+1 (non-binding). Make Submarine a separate project would make it easier to
integrate with other components in the ML pipeline and expand cross
platform.
 
On Thu, Jul 18, 2019 at 2:48 AM Jeff Zhang <[hidden email]> wrote:
 

> +1, This is Jeff Zhang from Zeppelin community.
> Thanks Xun for bring this up. Submarine has been integrated into Zeppelin
> several months ago, and I already see some early adoption of that in China.
> AI is fast growing area, I believe moving into a separate project would be
> helpful for Submarine to catch up with the new trend of AI and release more
> new features quickly than before.
>
>
>
> sid yu <[hidden email]> 于2019年7月18日周四 下午2:06写道:
>
> > +1  We are look forward to it. The idea is great.
> >
> > > On Jul 10, 2019, at 3:34 PM, Xun Liu <[hidden email]> wrote:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> --
> Best Regards
>
> Jeff Zhang
>
Reply | Threaded
Open this post in threaded view
|

Re: Any thoughts making Submarine a separate Apache project?

Xun Liu
In reply to this post by dashuiguailuyun@gmail.com
@adam.antal

The submarine development team has completed the following preparations:
1. Established a temporary test repository on Github.
2. Change the package name of hadoop submarine from org.hadoop.submarine to
org.submarine
3. Combine the Linkedin/TonY code into the Hadoop submarine module;
4. On the Github docked travis-ci system, all test cases have been tested;
5. Several Hadoop submarine users completed the system test using the code
in this repository.

赵欣 <[hidden email]> 于2019年7月22日周一 上午9:38写道:

> Hi
>
> I am a teacher at Southeast University (https://www.seu.edu.cn/). We are
> a major in electrical engineering. Our teaching teams and students use
> bigoop submarine for big data analysis and automation control of electrical
> equipment.
>
> Many thanks to the hadoop community for providing us with machine learning
> tools like submarine.
>
> I wish hadoop submarine is getting better and better.
>
>
> ==============================
> 赵欣
> 东南大学电气工程学院
>
> -----------------------------------------------------
>
> Zhao XIN
>
> School of Electrical Engineering
>
> ==============================
> 2019-07-18
>
>
> *From:* Xun Liu <[hidden email]>
> *Date:* 2019-07-18 09:46
> *To:* xinzhao <[hidden email]>
> *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> project?
>
>
> ---------- Forwarded message ---------
> 发件人: [hidden email] <[hidden email]>
> Date: 2019年7月17日周三 下午3:17
> Subject: Re: Re: Any thoughts making Submarine a separate Apache project?
> To: Szilard Nemeth <[hidden email]>, runlin zhang <
> [hidden email]>
> Cc: Xun Liu <[hidden email]>, common-dev <[hidden email]>,
> yarn-dev <[hidden email]>, hdfs-dev <
> [hidden email]>, mapreduce-dev <
> [hidden email]>, submarine-dev <
> [hidden email]>
>
>
> +1 ,Good idea, we are very much looking forward to it.
>
> ------------------------------
> [hidden email]
>
>
> *From:* Szilard Nemeth <[hidden email]>
> *Date:* 2019-07-17 14:55
> *To:* runlin zhang <[hidden email]>
> *CC:* Xun Liu <[hidden email]>; Hadoop Common
> <[hidden email]>; yarn-dev <[hidden email]>;
> Hdfs-dev <[hidden email]>; mapreduce-dev
> <[hidden email]>; submarine-dev
> <[hidden email]>
> *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> +1, this is a very great idea.
> As Hadoop repository has already grown huge and contains many projects, I
> think in general it's a good idea to separate projects in the early phase.
>
>
> On Wed, Jul 17, 2019, 08:50 runlin zhang <[hidden email]> wrote:
>
> > +1 ,That will be great !
> >
> > > 在 2019年7月10日,下午3:34,Xun Liu <[hidden email]> 写道:
> > >
> > > Hi all,
> > >
> > > This is Xun Liu contributing to the Submarine project for deep learning
> > > workloads running with big data workloads together on Hadoop clusters.
> > >
> > > There are a bunch of integrations of Submarine to other projects are
> > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next
> > step
> > > of Submarine is going to integrate with more projects like Apache
> Arrow,
> > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use
> > > cases like model serving, notebook management, advanced training
> > > optimizations (like auto parameter tuning, memory cache optimizations
> for
> > > large datasets for training, etc.), and make it run on other platforms
> > like
> > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > project
> > > to Apache so we can put Submarine and TonY together to the same
> codebase
> > > (Page #30.
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > ).
> > >
> > > This expands the scope of the original Submarine project in exciting
> new
> > > ways. Toward that end, would it make sense to create a separate
> Submarine
> > > project at Apache? This can make faster adoption of Submarine, and
> allow
> > > Submarine to grow to a full-blown machine learning platform.
> > >
> > > There will be lots of technical details to work out, but any initial
> > > thoughts on this?
> > >
> > > Best Regards,
> > > Xun Liu
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>