What is MLOps?

Machine learning applications have become the de facto solution to many problems in our daily lives. Building these applications is a little different from building standard software for several reasons. Some of these differences include: You need to constantly monitor the performance of models because they may degrade with time. Bringing a machine learning model […]
Jun 23rd 2021
read

Share this post

What is MLOps?
Jun 23rd 2021
read

Share this post

What is MLOps?

What is MLOps?

Derrick Mwiti

Data Scientist @ Layer

Machine learning applications have become the de facto solution to many problems in our daily lives. Building these applications is a little different from building standard software for several reasons. Some of these differences include:

  • You need to constantly monitor the performance of models because they may degrade with time.
  • Bringing a machine learning model to productions involves many people, including data scientists, data engineers, and business people.
  • Deploying and operating machine learning models at scale is a big challenge.

For a machine learning project to be successful, a set of practices, tools, and techniques need to be considered. These practices fall under a broad term known as MLOps. In this article, let’s take a look at various aspects of machine learning operations.

What is MLOps?

Building a machine learning model involves creating the model, training it, tuning, and deploying it. This process should be:

  • Scalable
  • Collaborative
  • Reproducible

For instance, it would be regrettable to build an excellent model but not reproduce the results in a production environment. The set of principles, tools, and techniques that ensure building machine learning models is scalable, collaborative, and reproducible are referred to as MLOps. In the world of software engineering, these practices are referred to as DevOps.

word image 9

Inspired by this paper.

DevOps vs. MLOps vs. Data Ops

DevOps are a set of principles that ensure that there is continuous delivery of high-quality software. In the machine learning realm, these practices are referred to as MLOps. DataOps involve a set of rules that ensure that high-quality data is available to analyze and train machine learning models. DataOps can be thought of as tightly integrating with MLOps.

word image 10

What problems does MLOps solve?

The set of tools and techniques defined in MLOps are geared towards making the lives of data scientists and machine learning practitioners easier. Let’s take a look at some of the problems that MLOps solves.

Versioning

Versioning is a common practice in software engineering where tools such as Git and GitHub are used to version code. Apart from versioning code, other things need to be versioned in machine learning. These items include:

  • Data used in model training
  • Model artifacts

Versioning models and data ensures that machine learning experiments are reproducible.

Monitoring model performance

Models placed in production can degrade over time. This is caused by differences in training data and testing data. This is usually referred to as data drift. By monitoring the performance of a model, these issues can be identified and addressed quickly.

Feature generation

Creating features can be a time-consuming and compute-intensive task. Applying proper MLOps techniques ensures that features that are generated once can be reused as many times as needed. This frees up the data scientist to focus on designing and testing the model.

What skills do you need for MLOps?

MLOps is quite a broad field and requires quite several skills. Fortunately, you are not expected to have all of these skills. Specializing in a couple of areas makes more sense. However, here are the skills needed by an MLOps team to deliver a machine learning project successfully:

  • Ability to articulate the business problem and the objectives
  • Collect the data needed to solve the identified problem
  • Prepare and process the data so that it is acceptable by machine learning models
  • Create features that are important to the problem in question
  • Build and train machine learning models
  • Develop a pipeline for ingesting data, generating features, training, and evaluating the model
  • Deploy the model so that the actual users can use it. This can also be part of the above pipeline
  • Monitor how the model performs in the real world

With those basics out of the way, let’s now take a look at the main components of MLOps.

Parts of MLOps

Despite the field being quite broad, a couple of parts come together to make it one piece. In this section, we’ll explore those parts.

word image 11

Feature store

Also referred to as a feature factory, it stores the features used in training a machine learning model. It is a critical part of MLOps because it ensures that there is no duplicity in creating features. If necessary, features can also be fetched and used for building other models or for general analysis. Features are also versioned while in the feature store, ensuring that one can revert to a particular feature version that resulted in a better model.

Data versioning

Apart from versioning features, the entire dataset used to create a certain model can also be versioned. Versioning data ensures that there is reproducibility in the process of creating models. It is also essential during auditing since it makes it easier to identify the datasets used to develop various models.

ML metadata store

To get rid of the magic involved in creating machine learning models, one has to log everything. Logging is critical for reproducibility. Some of the essential items to log include:

  • The seed used in splitting the data. This ensures that you are using the same split when creating a training and testing set
  • The random state used to initialize the model. The random state affects the reproducibility of model training
  • Model metrics
  • Hyperparameters
  • Learning curves
  • Training code and configuration files
  • Code used to generate features
  • Hardware logs

Storing model metadata is vital for various reasons:

  • Building dashboard with different models
  • Enabling the searchability of models based on hyperparameters

Model versioning

Versioning models is important because it enables switching between models in real-time. Apart from that, multiple models can be served to users at the same time to monitor performance. For instance, once a new model is available, it can be served to a few users to ensure that it performs as expected before rolling it out to everyone. Versioning is also critical from a compliance, governance, and historical point of view.

Model registry

Once a model has been trained, it is stored in a model registry. Every model in the registry will have a version for reasons already mentioned above. Each model should also be coupled with its:

  • Hyperparameters
  • Metrics
  • Feature version used to create the model
  • Dataset version used in training the model

..to mention a few

The model metadata mentioned above is important for:

  • Compliance with regulations
  • Management of the models
  • Identifying the endpoints of models in production

Model artifacts will usually be saved automatically depending on the MLOps tool you are using. You can also instruct the tool to save the best model checkpoints and upload them to the registry.

Model serving

Once a model is in the registry it can be deployed and served to users. Serving a model means creating endpoints that can be used to run inference on the model. The model artifact can also be downloaded and packaged with an application. However, deploying API endpoints makes it easier to use the model in various applications. That said, there is a case for packaging the model in applications such as mobile applications to reduce the inference latency.

Model monitoring

Once machine learning models have been deployed, they have to be monitored for model drift and production skew. Model drift occurs when the statistical differences between the training data and inference data change in unexpected ways. The performance of the model thus degrades. You can catch these problems early by monitoring the statistical properties of the training and prediction data.

Production skew occurs when the served model performs dismally compared to the offline model. This can be caused by bugs during the training process, serving bugs, and discrepancies in training and inference data.

Model drift and production skew should be monitored to ensure that the model is behaving as expected.

Model retraining

Machine learning models can be retrained for two main reasons:

  • To improve the performance of the model
  • When new training data becomes available

Your machine learning pipeline should detect the availability of new data or the dismal performance of the model and trigger retraining of the model. The system should also detect and deprecate models that would not benefit from retraining.

CI/CD

Continuous integration and continuous deployment in machine learning ensure that high-quality models are created and deployed often. Continuous delivery ensures that code is frequently merged in a central repository where automated builds and tests are implemented. In machine learning, this would involve not only testing the code but also the resulting models. It also entails packaging the models in readiness for use by actual users.

Continuous delivery involves automatically deploying code changes to a staging or production environment. In a machine learning pipeline, this would involve deploying a model to test and or production servers. Frequent deployments are significant because they ensure that code and models are tested vigorously and often before moving them to production.

How to implement MLOps

You can create a system to implement the items we have mentioned above. Alternatively, you can use a machine learning pipeline orchestration platform that will make your workflow easier. Let’s discover some of the best tools that you can use for an ML pipeline orchestration.

MLOps solutions

The choice of a machine learning orchestration tool will depend on a couple of factors, including:

  • The skills of your team
  • Your budget
  • Whether you want to automate part of your pipeline or the entire pipeline
  • The ease of integrating the new tool

Just to mention a few.

Let’s now look at some of the best tools you can use to orchestrate your machine learning pipeline.

Layer

Layer is a Declarative MLOps platform that empowers data science teams to implement end-to-end machine learning with minimal effort. Layer is a managed solution and can be used without the hassle involved in configuring and setting up servers.

The first step is to install the Layer SDK.

$ pip install layer-sdk

Let’s create a model using the famous titanic dataset. You can take a look at this dataset on a Jupyter Notebook using Layer. In order to do that, you’ll need to import Layer and then log in. After that, you can use the `get_dataset` function to access the data. Here is the complete code snippet that does that.

import layer

layer.login()

df = layer.get_dataset("titanic")

After that, you can convert the dataset into a Pandas DataFrame and display the first few records.

df = df.to_pandas()

df.head()

word image 12

Now that you have seen how the dataset looks, you can move to your favorite Python editor to start a Layer project. A Layer project usually contains YAML files that define the models and features in either Python, or SQL. At the moment Layer allows you to define your features in SQL,and Python. Models are defined in Python.

Creating a Layer project

You can create a Layer project by cloning the empty Layer repository.

$ layer clone https://github.com/layerml/empty.git

Alternatively, you can also a clone project that has been provided by Layer in the examples repo. After cloning the repo, you can change into one of the example projects, for example the Titanic Survival Project.

$ layer clone https://github.com/layerml/examples.git

Here is the folder structure of that the Titanic Survival Project.

.
├── .layer
├── data
│   ├── passenger_features	                # feature definitions
│   │   ├── ageband.sql				# Age Band of the passenger
│   │   ├── embarked.sql  			# Embarked or not
│   │   ├── fareband.sql			# Fare Band of the passenger
│   │   ├── is_alone.sql			# Is Passenger travelling alone
│   │   ├── sex.sql				# Sex of the passenger
│   │   ├── survived.sql 			# Survived or not
│   │   ├── title.sql				# Title of the passenger
│   │   └── dataset.yaml				# Declares the metadata of the features above
│   └── titanic_data
│       └── dataset.yaml				# Declares where our source `titanic` dataset is
├── models
│   └── survival_model
│       ├── model.yaml				# Training directives of our model
│       ├── model.py				# Source code of the `Survival` model
│       └── requirements.txt		        # Environment config file
└── README.md

The `.layer` folder contains a `project.yml` file that is the main configuration of the project. This file defines the project’s metadata such as the project name.

apiVersion: 1

# Name of your project

name: "Titanic Survival Project"

The data folder contains a YAML file that links the titanic dataset to this project. In this case, the titanic dataset has already been uploaded to Snowflake and connected to Layer. If you were using a different dataset, you’d have to do the integration yourself. The YAML file below means that here is a table on our Snowflake instance known as `titanic` and we would like to access it on our project using the `titanic` name.

# Titanic Survival Project Example
#
# In this file, we will integrate a source table from the Layer Demo
# Snowflake database. This source table is the Titanic dataset from
# Kaggle (https://www.kaggle.com/c/titanic/data)

apiVersion: 1

# Unique name of this dataset which will be used in this project to
# refer to this dataset
name: "titanic"
type: source

materialization:
target: layer-public-datasets
table_name: "titanic"

The name of the model will be `titanic_survival_model_pred`The `models` folder contains the implementation of the model defined in Python. This folder also hosts a YAML file that stores the model metadata, such as its name. This file also defines other model configurations such as hyperparameter tuning. For instance, the file below means that:

  • The model will be defined in a file known as `model.py`
  • The libraries required by the model will be found in a file called `requirements.txt`.
apiVersion: 1

# Name and description of our model

name: "titanic_survival_model_pred"

description: "Titanic survival predictor"

training:
name: survival_model_training
description: "Titanic Survival Model Training"
# The source model definition file with a `train_model` method
entrypoint: model.py
# File includes the required python libraries with their correct versions
environment: requirements.txt

Layer treats any folder with a `model.yml` file as the model ML project.

Feature engineering with Layer

In Layer, features are defined in the data folder. You can create a folder for the features or a folder for each feature or not. In this case, let’s walk through the creation of the features in Python. The project we downloaded has the features in SQL, so we will be converting those to Pythion features. The features are defined in a folder called `features_from_passengers`.

Now you have to define the attributes of the `featureset` in the `dataset.yaml`. Layer treats any folder with this file as a dataset project. A `Featureset` is a group of features. In this YAML file you need to define:

  • The name of the `Featureset`
  • The description of the `Featureset`
  • The individual features and requirements files for any libraries that are needed to compute each feature
  • Setting the `materializations` to target `layer` refers back to the dataset you defined earlier
  • The `source` defines the file where the feature will be defined. This can be a Python or SQL file.

The features defined using a `Featureset` is versioned and stored in Layer’s Data Catalog. You’ll see this in Layer’s UI later.

apiVersion: 1

type: featureset

name: "features_from_passengers"
description: "Passenger features from Titanic Dataset"

features:
- name: age
description: "Age based on the class"
source: ageband.py
environment: requirements.txt
- name: embarked
description: "The embark status of the passenger. (S=0, C=1, Q=2)"
source: embarked.py
environment: requirements.txt
- name: fare
description: "Fare"
source: fareband.py
environment: requirements.txt
- name: sex
description: "Sex vector (0=Female, 1=Male)"
source: sex.py
environment: requirements.txt
- name: survived
description: "Whether the passenger survived or not."
source: survived.py
environment: requirements.txt

materializations:
- target: layer-public-datasets

Now create the individual features as defined in the YAML file above. Let’s start with the age feature. In Layer, Python features are created by implementing the `build_feature` function. The function can be used to load entities from Layer’s Data and Model Catalog. In this case, you have to load the `titanic` dataset.

After loading the dataset, you can convert it to a Pandas DataFrame and proceed as you are used to. In this case, let’s clean the age column based on the passenger’s class. Notice, that the passenger’s ID is also returned with the new DataFrame. This is important because this ID will be used to merge all the features later.

import pandas as pd
from typing import Any
from layer import Dataset


def build_feature(sdf: Dataset("titanic")) -> Any:
    df = sdf.to_pandas()
    feature_data = df[["PASSENGERID", "AGE","PCLASS"]]
    age = feature_data[['AGE', 'PCLASS']].apply(clean_age, axis=1)
    feature_data['AGE'] = pd.DataFrame(age)
    return feature_data[["PASSENGERID","AGE"]]

Here’s the function for cleaning the age for reference.

def clean_age(data):
    age = data[0]
    pclass = data[1]
    if pd.isnull(age):
        if pclass == 1:
            return 37
        elif pclass == 2:
            return 29
        else:
            return 24
    else:
        return age

Let’s do something similar with the `embarked` feature. Since Pandas is being used here, you will have to define it in the `requirements.text` file.

 

import pandas as pd
from typing import Any
from layer import Dataset


def clean_embark(embark):
    result = 0
    if embark == "S":
        result = 0
    elif embark == "C":
        result = 1
    elif embark == "Q":
        result == 2
    else:
        result = -1
    return result


def build_feature(sdf: Dataset("titanic")) -> Any:
    df = sdf.to_pandas()
    feature_data = df[["PASSENGERID", "EMBARKED"]]
    embark = feature_data['EMBARKED'].apply(clean_embark)
    feature_data['EMBARKED'] = pd.DataFrame(embark)

    return feature_data[["PASSENGERID","EMBARKED"]]

Next, let’s encode the sex feature with integers. You can also use a label encoder instead of writing your own function.

import pandas as pd
from typing import Any
from layer import Dataset


def clean_sex(sex):
    result = 0
    if sex == "female":
        result = 0
    elif sex == "male":
        result = 1
    return result


def build_feature(sdf: Dataset("titanic")) -> Any:
    df = sdf.to_pandas()
    feature_data = df[["PASSENGERID", "SEX"]]
    sex = df['SEX'].apply(clean_sex)
    feature_data['SEX'] = pd.DataFrame(sex)

    return feature_data[["PASSENGERID","SEX"]]

from typing import AnyFor the fare, let’s just return the fare since it’s numeric. Alternatively, you can fail to create the feature and obtain it during the model training stage. However, this may be a good approach if you plan to perform further transformations on the feature in the future.

from layer import Dataset


def build_feature(sdf: Dataset("titanic")) -> Any:
    df = sdf.to_pandas()
    fare = df[['PASSENGERID','FARE']]
    return fare

The final item needed here is the target variable i.e if the passenger survived or not. Let’s define that next.

import pandas as pd

from typing import Any

from layer import Dataset

def build_feature(sdf: Dataset("titanic")) -> Any:
    df = sdf.to_pandas()
    survived = df[['PASSENGERID', 'SURVIVED']]
return survived

Now that the features are ready, the next item is to use them to train a machine learning model.

Model training with Layer

In Layer, models are defined in a directory containing a `model.yml` file. Models are defined in a Python file implementing the `train_model` function. The function accepts a Layer `Datasetet` and the `FeatureSet` defined above as arguments. Models are trained according to the configuration found in the `model.yml` file.

Let’s now use the features generated above to train a Random Forest model. In the end, this function has to return a model.

The `train.log_parameter` can be used to log various items in the training process. For instance, you can log:

  • The size of the test set
  • The size of the training set
  • Different model parameters
  • The training and testing loss and accuracy
test_size = 0.2

train.log_parameter("test_size", test_size)

train.log_parameter("n_estimators", estimators)

train.log_metric("accuracy", acc)

The input and output of the training function can also be registered to Layer. Layer will use this to compute data drift.

train.register_input(X_train)
train.register_output(y_train)

All logged items will be available on Layer’s web UI.

Here’s the complete code for training this titanic classification model.

"""Titanic Survival Project Example

This file demonstrates how we can develop and train our model by using the
`passenger_features` we've developed earlier. Every ML model project
should have a definition file like this one.

"""
from typing import Any
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from layer import Featureset, Train


def train_model(train: Train, pf: Featureset("features_from_passengers")) -> Any:
    """Model train function

    This function is a reserved function and will be called by Layer
    when we want this model to be trained along with the parameters.
    Just like the `passenger_features` featureset, you can add more
    parameters to this method to request artifacts (datasets,
    featuresets or models) from Layer.

    Args:
        train (layer.Train): Represents the current train of the model, passed by
            Layer when the training of the model starts.
        pf (layer.Featureset): Layer will return a Featureset object,
            an interface to access the features inside the
            `passenger_features`

    Returns:
       model: Trained model object

    """
    df = pf.to_pandas()

    X = df.drop(["PASSENGERID", "SURVIVED"], axis=1)
    y = df["SURVIVED"]

    # Split dataset into training set and test set
    test_size = 0.2
    train.log_parameter("test_size", test_size)
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        test_size=test_size)

    # Here we register input & output of the train. Layer will use
    # this registers to extract the signature of the model and calculate
    # the drift
    train.register_input(X_train)
    train.register_output(y_train)

    # We will use `RandomForestClassifier` for this task with fixed parameters
    estimators = 100
    random_forest = RandomForestClassifier(n_estimators=estimators)

    # We can log parameters of this train. Later we can compare
    # parameters of different versions of this model in the Layer
    # Web interface
    train.log_parameter("n_estimators", estimators)

    # We fit our model with the train and the label data
    random_forest.fit(X_train, y_train)

    # Let's calculate the accuracy of our model
    y_pred = random_forest.predict(X_test)
    acc = accuracy_score(y_test, y_pred)

    # Just like we logged parameters above, we can log metrics as well to be
    # compared later in Layer Web interface
    train.log_metric("accuracy", acc)

    # We return the model
    return random_forest
Layer 0.8.1 using https://beta.layer.co
Loading the project under /Users/derrickmwiti/PycharmProjects/Layer-videos/examples/titanic-python-features
Found 1 dataset, 1 featureset and 1 model
📔 Session logs at /Users/derrickmwiti/.layer/logs/20210914T120431-session-c3c50ffe-ffe9-47da-b15e-c5b9ffb7fbd8.log
💾 Starting…
🔵 Pipeline run id: cc40aac7-46a6-43e6-aa01-0835d68d1350
2021-09-14 12:04:34 | dataset titanic ━━━━━━━━━━━━━━━━━━━━━━ DONE [372ms]
2021-09-14 12:04:53 | featureset features_from_passengers ━━━━━━━━━━━━━━━━━━━━━━ DONE [50664ms]
https://beta.layer.co/features/9436818a-142b-47a9-8e27-ab44a6dbae22
2021-09-14 12:05:44 | model titanic_survival_model_pred ━━━━━━━━━━━━━━━━━━━━━━ DONE [20318ms]
https://beta.layer.co/models/504b61f6-2759-4b0c-b0df-cf3b7c389a3c/trains/5a616c7e-1265-4ded-94b4-b73352f2c71e

Once you are done with the model definition, the next step is to run the project. While in the projects folder, execute `layer run` on the terminal.

Accessing the Data Catalog

Now that the project has run successfully click on the first link on the terminal to view the Data Catalog. The Data Catalog hosts the features generated by Layer.

word image 15

The Profile tab shows a snapshot of the generated features.

word image 16

These features can be reused in your organization to build different models or to re-train the current model. This reduces the amount of work needed to create the same features afresh. Each `Featureset` is automatically versioned so that you can easily switch between different feature versions. Auto versioning also ensures that you build reproducible data pipelines.

Viewing the Model Catalog

Clicking the second link on the CLI will take you to Layer’s Model Catalog. This a central storage for your ML models. Layer automatically stores and versions every model. The models are immutable.

word image 17

All the items that you logged while training the model can be found under the Attributes tab.

word image 18

Deploying the model and running inference

Layer allows you to deploy your trained model at the click of a button. Click the Deploy button to start the process of deploying the model.

word image 19

After deploying is complete, the model will be ready for inference.

word image 20

Copy the link next to the deploy button to start running inferences on this model immediately. Here’s how you would run inference using curl.

curl --header "Content-Type: application/json; format=pandas-records" --request POST --data '[{"FARE":4.8, "EMBARKED":0, "AGE":34, "sex":1}]' https://65760cd4-ca8f-4eda-a60e-360fc1448447.inferences.development.layer.co/invocations

The same can be done using Postman.

word image 21

In this case, you’ll get the response as [0], indicating that a passenger with these attributes wouldn’t have survived on the titanic.

Apart from Layer, there are other tools that you can use to orchestrate your machine learning pipelines. Let’s discover a couple more:

  • MLflow is an open-source platform for managing a machine learning life cycle. The platform can be used for ML experiment tracking, deployment as well as a central model registry.
  • Sacred is an open-source library that can be used to organize, log and reproduce machine learning experiments. It doesn’t ship with a web UI. Omniboard is a popular front-end library for Sacred.
  • ModelDB is an open-source tool for versioning models, storing model metadata, and managing machine learning experiments. It can be used for making ML pipelines reproducible as well as displaying model performance dashboards.

MLOps best practices

MLOps is a relatively new field; however, some best practices will lead to the success of your machine learning orchestration process when adhered to. Let’s mention some of them:

  • Use tools that are collaborative. This makes it easier for everyone on the team to access code, data, and information about the project, for example, the generated features. It also makes it easier to raise and track issues.
  • Start with a simple model. Starting with a simple model gives you adequate time to ensure that the infrastructure is right. A complex model means that you have to debug a complicated model and optimize the infrastructure it will run on.
  • Just launch. Don’t spend months on end building and deploying the machine learning model. It is better to launch the model as soon as possible to start testing it on actual users. You can serve the model to a small number of users to start getting initial feedback. That feedback can be used to iterate the model and infrastructure as necessary.
  • Perform automated regression tests. This is crucial to ensure that new code doesn’t introduce bugs in existing code. Code that fails the tests is not merged into the main source code. Regression testing ensures that new code doesn’t break existing functions.
  • Automate model deployments. This ensures that new models that pass certain tests become automatically available to users. It also frees up engineers from the manual process of packaging models for production. The process involves automated packing of models together with their dependencies and delivering them to production or staging environments. The models should be monitored constantly and automatically rolled back when they perform dismally.
  • Attach predictions to model versions and data. This makes it easier to track every prediction to a specific model and data. This is important for traceability, reproducibility, and compliance reasons. Logging predictions with data and model versions also makes it easy to debug models in the event of unexpected behavior.
  • Measure training and serving skew. Machine learning models may not always perform as expected when served for use with unseen data. As a result, it is crucial to measure the difference between performance on training and unseen data. If the difference is not acceptable by your standards, you will have to implement a way to alleviate that, for instance, reworking the features.
  • Implement shadow production. This involves using production data to make predictions on a model. The predictions are, however, not to used to make real-world decisions. They are compared with the decisions made by the existing system of making decisions even if that system is manual. When the decisions made by the model are acceptable, it can be promoted to making real decisions.
  • Hyperparameter tuning optimization. Selecting and searching for the best model parameters by hand can be a pain in the neck. Automating the process will hasten your machine learning experimentation process. The results of hyperparameter optimization can then be compared, and the best algorithms and parameters were chosen for production.

You have learned a lot from this article. Let’s summarize it in the table below.

A summary overview of MLOps

Item Description
What is MLOps? MLOps encompass a set of best practices that ensure that a machine learning project is completed successfully. These practices include automated testing of ML code, data validation, and automated deployment and rollback of ML models.
How is MLOps different from DevOps? DevOps are standards to ensure the success of a software project. In the machine learning space, there are several other items that one needs to be concerned about. For instance, a machine learning model’s performance can degrade over time. This could be a result of changes in data or errors during deployment. As a result of that MLOps is also concerned with ensuring that an ML model performs optimally at any point in time. It also ensures that the model is retrained in the event of dismal performance.
Which problems does MLOps Solve? MLOps practices are geared at solving the following ML problems:

  • Versioning of data and models to ensure reproducibility and traceability
  • Tracking the performance of the model to ensure that dismal model performance is addressed immediately
  • Generating and version, storing features so that they can be reused in the future
Which are the most crucial elements of an MLOps system? An end-to-end MLOps system should have the following elements:

  • A Featurestore where all the generated features are stored
  • Ability to version features so that each model is linked to its features
  • An ML metadata store to log all model metadata such as test and training size, model hyperparameters, hardware metrics, code used to generate the features, etc
  • A model registry for automatically storing trained models and serving them
  • Ability to version all models to enable easy switching from one model to another
  • Monitor the model in production to tell if it’s performing as required
Why should you use an end-to-end MLOps platform? Many solutions solve different parts of the machine learning life cycle. To ensure that you don’t get bogged down with multiple tools, you must select a solution covering every aspect of the ML lifecycle. An end-to-end solution should allow you to:

  • Perform machine learning experimentation
  • Generate features easily
  • Store and version the features
  • Manage and version multiple machine learning models
  • Serve and run inferences on the trained models
  • Monitor the machine learning models in production for concept and data drift
  • Retraining dismally performing models
  • Rollback to a previous model version if the current version performs poorly

Final thoughts

This article has been a primer into the world of machine learning operations, popularly known as MLOps. We have covered various aspects of MLOps as well as a couple of best practices. We have also looked at some tools that you can use to automate the MLOps process. More specifically, you have learned:

  • What is MLOps
  • The difference between MLOps and DevOps
  • Problems that MLOps solve
  • Skills needed to operate in the MLOps space
  • Various components of MLOps
  • End-to-end MLOps solutions

Just to mention a few.

Resources

Rules of ML by Google

Engineering best practices for machine learning

Introducing MLOps

Titanic example source code

Jun 23rd 2021
read

Share this post

Try Layer for free

Get started with Layers Beta

Start Free