Kubernetes for Data Science: meet Kubeflow

栏目: IT技术 · 发布时间: 4年前

内容简介:Data science has exploded as a practice in the past decade and has become an undisputed driver of innovation.The forcing factors behind the rising interest in Machine Learning, a not so new concept, have consolidated and created anDeep Learning enabling fa

Deep Learning is set to thrive

Data science has exploded as a practice in the past decade and has become an undisputed driver of innovation.

The forcing factors behind the rising interest in Machine Learning, a not so new concept, have consolidated and created an unparalleled capacity for Deep Learning , a subset of Artificial Neural Networks with many “ hidden layers ”, to thrive in the years to come.

Deep Learning enabling factors:

  • Computational capacity increased exponentially ( GPGPUs and TPUs)
  • Hardware availability at low cost (Public & Hybrid Clouds, efficient Data Centers)
  • Data availability (Publicly accessible data, low-cost widespread IoT devices)
  • Open Source community (Libraries – TensorFlow , PyTorch ; Competitions – Kaggle )

But it still faces many challenges

The common pathway for a data scientist is to start by writing a model on a Jupyter notebook using Python and amazing Open Source libraries such as TensorFlow , Keras or PyTorch . When starting out, we tend to be focused on the end result of the model, but, there is a lot more.

While trying to bring the model to the hands of users or to edge devices, things get more complicated. In fact, developing the model itself is only a fairly small portion of the effort required to train, deploy and manage an AI project.

There is just a lot of background work to be done:

Kubernetes for Data Science: meet Kubeflow
Area = effort

The typical Machine Learning workflow can look like this:

Kubernetes for Data Science: meet Kubeflow

With these different stages, having diverse requirements, the challenges that arise are threefold:

  1. Composability – The workflow from data ingestion to model serving, monitoring and logging, includes many components spread across multiple systems making it hard to manage, secure and maintain.
  2. Portability – At different stages of the ML process, computation requirements change, and so does the hardware in which your software is running – Laptop, on-prem training rig, public cloud.
  3. Scalability – Computation requirements for AI projects are very dynamic, a training phase is resource intensive, while inference phase is lightweight and speedy, hence, having elasticity at the infrastructure level is compulsory.

The word that best defines these needs is MLOps .

Kubernetes for Data Science: meet Kubeflow

Kubernetes can help!

Kubernetes (a.k.a. K8s) is an open source system to automate deployment, scaling, and management of containerized applications widely used in the world of DevOps .

For Data Scientists with the above mentioned challenges, this means they can package each step of the process as a container, making it system agnostic (portable) and composable (i.e. modular building blocks), and have Kubernetes handle the deployment and management at scale.

«Why not simply use the great powers of Kubernetes?»

The only problem is, we need to become experts in:

– Kubernetes service endpoints

– Immutable deployments

– Persistent volumes

– GPGPU passthough

– Drivers & the GPL

– Containers

– Cloud APIs

– Packaging

– DevOps

– Scaling

– (…)

Kubernetes for Data Science: meet Kubeflow

Meet Kubeflow

Kubeflow makes deployments of Machine Learning workflows on Kubernetes Simple, Portable and Scalable .

Kubeflow is the machine learning toolkit for Kubernetes. It extends Kubernetes ability to run independent and configurable steps, with machine learning specific frameworks and libraries.

And, it is all Open Source!

Run it on your workstation, on-premises training rig, or in any hybrid or public cloud, in a new or already running Kubernetes deployment. Within Kubeflow you will find all the open source tools and frameworks you need:

Kubernetes for Data Science: meet Kubeflow

To know more, visitubuntu.com/kubeflow, or install Kubeflow by following the tutorial Deploy Kubeflow on Ubuntu, Windows and MacOS .

In upcoming posts, we will dive deeper in the technologies that make Kubeflow, and how you can leverage them to enhance your Data Science capabilities. Subscribe to our Cloud and Server newsletter to up to date.


以上所述就是小编给大家介绍的《Kubernetes for Data Science: meet Kubeflow》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

大学算法教程

大学算法教程

约翰森堡 / 清华大学 / 2007-6 / 69.80元

本书是美国德保罗大学DePaul University教授R.Johnsonbaugh等人长期从事算法课程教学经验的结晶,是一本关于算法基础知识和基本方法的教科书。内容包括:算法必备的数学基础、数据结构和描述算法的语言与记号;常用算法的设计分析及其正确性证明;NP和NP完全问题的特征及其近似处理方法。 全书含300多个生动有趣的算法实际示例和1450多道习题,从经典方法到最新成果,层层剖析......一起来看看 《大学算法教程》 这本书的介绍吧!

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试

RGB HSV 转换
RGB HSV 转换

RGB HSV 互转工具

HEX HSV 转换工具
HEX HSV 转换工具

HEX HSV 互换工具