---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## Model Deployment

Keith VanderLinden<br>
Calvin University

::: {.notes}
- In this module, we'll moving the running of our model from the notebook to a Web service, where others can use it.
:::

## Deployment of ML Systems

Most production ML systems:

- Include more than just one model.
- Operate in dynamic, evolving environments.
- Update their models regularly.
- Operate at scale.

We&rsquo;ll implement limited variants of these realities.

::: {.notes}
- These are the realistic counter-parts to C. Huyen's myths of ML deployment.
- We'll implement C. Huyen's myths, keeping in mind that they are myths.
    - Include more than just one or two models.
        - Uber has thousands of deployed models.
        - We'll do one or two models, potentially maintaining versions of each.
    - Operate in dynamic, evolving environments.
        - Uber's models must adapt to changing customer and traffic patterns.
        - The TagIfAI domain isn't that dynamic; SLO was; your project may be as well. But we won't be working on the models long enough to see them evolve very much.
    - Update their models regularly.
        - AWS deploys every 11.7 seconds.
        - We won't be updating our models that regularly.
    - Operate at scale.
        - Google's ML models serve billions of users.
        - We won't operate at scale.
:::

## Prediction Modes

The basic prediction modes are:

- *Batch* prediction
- *Online* prediction
    - using *batch features*
    - using *streaming features*

There are many variations and combinations.

::: {.notes}
- From the UX perspective, we distinguish:
    - Batch (off-line) systems serve predictions from pre-computed results (q.v., DMLS Figure 7-4).
    - Online (real-time) systems serve predictions from live computation.
- From the feature engineering perspective, we distinguish:
    - Batch features come from pre-computed, stored data (q.v., DMLS Figure 7-5).
    - Streaming features come from live data, which only makes sense in online systems (q.v., DMLS Figure 7-6).
- Batch & streaming features can be used in combination, but we'll be implementing an online system with a pre-computed model that uses only streaming features (a single text to be classified).
- Interestingly, C. Huyen suggests that compute power will eventually make online systems the default.
:::

## Batch vs Streaming Features

![<small>DMLS Figure 7-8</small>](images/dmls-figure7-8.png){width=38%}

::: {.notes}
- We discussed this in the context of feature engineering. Batch and streaming features must be engineered in the same way (see the "should be equal" line).
- The feature inputs to the model could be a combination of live and batched data; the diagram doesn't show that. That is, the diagram shows what we'll be doing for TagIfAI, minus the streaming.
:::

## Computation Architecture

Model computation can be done on either:

- Cloud
- Edge

The first is the easiest and most common.

::: {.notes}
- ML computation can be done on the cloud or on the edge.
    - Cloud computing is centralized on the server. It's:
        - easier.
        - more expensive in terms of shared server compute resources.
    - Edge computing is distributed to the clients. It's:
        - more challenging because it's distributed to heterogeneous clients.
        - cheaper in terms of server compute resources.
        - can be run off-line.
        - quicker in terms of UX.
        - more private for users.
- We'll do ours on the cloud (CodeSpaces).
- One interesting edge example is running models in the web browser (e.g., using WASM).
:::

