1. A platform for the Complete Machine Learning Lifecycle
Corey Zumar
March 27th, 2019

2. Outline Overview of ML development challenges
How MLflow tackles these challenges
MLflow components
Demo
How to get started
- Quick overview of MLflow.
- Some of the concepts were covered in Matei's keynote talk on Machine Leaning and AI.

3. Machine Learning Development is Complex

4. ML Lifecycle Data Prep Raw Data Training Deploy Tuning Scale Tuning
Delta μ λ θ
Tuning
Scale
Data Prep
Model Exchange μ λ θ
Tuning
Raw Data
Training
Scale
Scale
Deploy
Governance
Scale
0. ML dev process: raw data -> prep (ETL and feature computation) -> training -> deployment
aspects to ML dev cycle that makes it complex
Various tools. Best tool for each problem. Multiple tools. Comparison to tradition s/w development.
Various steps tuning and optimization: play with longer windows for feature, remember learning rate…
Scale: use all the data to improve results
Larger teams, isolation => model and data format for exchange. Changes in data storage format -> trickle in ETL changes -> …. Large teams, friction.
Governance of models and data: various models in testing. Audit an older model – observe data, retune it???

5. Custom ML Platforms Can we provide similar benefits in an open manner?
Facebook FBLearner, Uber Michelangelo, Google TFX
Standardize the data prep / training / deploy loop: if you work with the platform, you get these!
Limited to a few algorithms or frameworks
Tied to one company’s infrastructure
Can we provide similar benefits in an open manner?
How are the other big players in Machine Learning scaling out? Invested in platform team
POSITIVES : simplification by standardization. A set of tools for data prep, training and deploy.
NEGATIVE : limited models. Inflexibility of algorithms. Tied to company’s infra. Hard to open it to other users or open source these systems.

6. Introducing Open machine learning platform
Works with any ML library & language
Runs the same way anywhere (e.g. any cloud)
Designed to be useful for 1 or person orgs
Mlflow brings about an innovation – as an open ML platform.
Work with any ML library/algorithm, language of choice. Not boxed into limited set of tools
- Design philosophy: API first, modular.
Submit runs in the same way anywhere. Local machine, any type of cloud
Reduce complexity and learning curve. Make it easy for 1 person to use it and scalable to 100K of user in an org.

7. MLflow Components Tracking Projects Models
Standard packaging format for reproducible ML runs
Folder of code + data files with a “MLproject” description file
MLflow Components
Tracking
Record and query experiments: code, configs, results, …etc
Projects
Packaging format for reproducible runs on any platform
Models
General model format that supports diverse deployment tools

8. Key Concepts in Tracking
Parameters: key-value inputs to your code
Metrics: numeric values (can update over time)
Artifacts: arbitrary files, including data and models
Source: training code that ran
Version: version of the training code
Tags and Notes: any additional information

9. MLflow Tracking Tracking Server Tracking APIs (REST, Python, Java, R)UI
API
Tracking APIs
(REST, Python, Java, R)

10. Record and query experiments: code, configs, results, …etc
Standard packaging format for reproducible ML runs
Folder of code + data files with a “MLproject” description file
MLflow Tracking
Tracking
Record and query experiments: code, configs, results, …etc
import mlflow
with mlflow.start_run():
mlflow.log_param("layers", layers)
mlflow.log_param("alpha", alpha)
# train model
mlflow.log_metric("mse", model.mse())
mlflow.log_artifact("plot", model.plot(test_df))
mlflow.tensorflow.log_model(model)

11. MLflow backend stores Entity Store Artifact Repository
FileStore (local filesystem)
SQLStore (via SQLAlchemy)
REST Store
Artifact Repository
S3 backed store
Azure Blob storage
Google Cloud storage
DBFS artifact repo

12. Demo Goal: Classify hand-drawn digits
Instrument Keras training code with MLflow tracking APIs
Run training code as an MLflow Project
Deploy an MLflow Model for real-time serving

13. MLflow Projects Motivation
Diverse set of training tools
Result:
ML code is difficult to productionize.
Diverse set of environments

14. MLflow Projects Project Spec Local Execution Remote Execution Code
Config
Remote Execution
Dependencies
Data
Project: specify code & dependencies
Soundtrack: Kubernetes, or EC2, or Databricks & logos

15. MLflow Projects Packaging format for reproducible ML runs
Any code folder or GitHub repository
Optional MLproject file with project configuration
Defines dependencies for reproducibility
Conda (+ R, Docker, …) dependencies can be specified in MLproject
Reproducible in (almost) any environment
Execution API for running projects
CLI / Python / R / Java
Supports local and remote execution

16. Example MLflow Project
my_project/ ├── MLproject │ │ │ │ │ ├── conda.yaml ├── main.py └── model.py
conda_env: conda.yaml entry_points: main: parameters: training_data: path lambda: {type: float, default: 0.1} command: python main.py {training_data}
{lambda}
$ mlflow run git://<my_project>

17. Demo Goal: Classify hand-drawn digits
Instrument Keras training code with MLflow tracking APIs
Run training code as an MLflow Project
Deploy an MLflow Model for real-time serving

18. MLflow Models Motivation
ML Frameworks
Inference Code
Batch & Stream Scoring
Serving Tools

19. MLflow Models ML Frameworks Inference Code Model Format
Flavor 1
Flavor 2
Batch & Stream Scoring
Standard for ML models
ML Frameworks
Serving Tools

20. MLflow Models Packaging format for ML Models
Any directory with MLmodel file
Defines dependencies for reproducibility
Conda environment can be specified in MLmodel configuration
Model creation utilities
Save models from any framework in MLflow format
Deployment APIs
CLI / Python / R / Java

21. Example MLflow Model mlflow.tensorflow.log_model(...)
run_id: efd4c4bbd time_created: T12:34 flavors: tensorflow: saved_model_dir: estimator signature_def_key: predict python_function: loader_module:
mlflow.tensorflow
my_model/ ├── MLmodel │ │ │ │ │ └ estimator/ ├─ saved_model.pb └─ variables/
Usable with Tensorflow
tools / APIs
Usable with any Python
tool
mlflow.tensorflow.log_model(...)

22. Model Flavors Example PyTorch Train a model mlflow.pytorch.log_model()
predict = mlflow.pyfunc.load_pyfunc(…)
predict(input_dataframe)
Flavor 1: Pyfunc
Model
Format
Flavor 2: PyTorch
model = mlflow.pytorch.load_model(…)
with torch.no_grad():
model(input_tensor)

23. Model Flavors Example predict = mlflow.pyfunc.load_pyfunc(…)
predict(input_dataframe)

24. Demo Goal: Classify hand-drawn digits
Instrument Keras training code with MLflow tracking APIs
Run training code as an MLflow Project
Deploy an MLflow Model for real-time serving

25. Get started with MLflow
pip install mlflow to get started
Find docs & examples at mlflow.org
tinyurl.com/mlflow-slack

26. 0.9.0 Release MLflow 0.9.0 was released this week! Major features:
Tracking server supports SQL via SQLAlchemy
Pluggable tracking server backends
Docker environments for Projects
Custom python models

27. Ongoing MLflow Roadmap
UI scalability improvements (1.0)
X-coordinate logging for metrics & batched logging (1.0)
Fluent API for Java and Scala (1.0)
Packaging projects with build steps (1.0+)
Better environment isolation when loading models (1.0)
Improved model schemas (1.0) ✔

28. Thank you!