---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## Model Monitoring

Keith VanderLinden<br>
Calvin University

::: {.notes}
- This module addresses monitoring and maintaining ML systems in production.
- It comprises two, tightly-coupled DMLS chapters:
    - Chapter 8 focused on passive monitoring, i.e., logging and dash-boarding
        stuff so that we can monitor and, if necessary, fix things.
    - Chapter 9 focuses on proactive monitoring, i.e., configuring the system to do
        monitoring/probing automatically. This cutting-edge stuff that we'll
        address at the conceptual level only.
:::

## ML System Failures

ML systems fail if:

- The software system fails to operate as expected.
- The ML system fails to perform as expected.

The first criterion is shared by all software systems; the second is unique to ML systems.

::: {.notes}
- System operation failures
    - Types:
        - hardware and software failures
        - crashes and downtime
    - These are: common to all software systems; easier to detect (e.g., 404 errors on a REST API).
- ML performance failures
    - Types:
        - Poorly configured/tuned models (q.v., other ML courses)
        - Training vs inference pipeline mismatches (aka, training-serving skew; q.v., module 12)
        - Degenerate feedback loops (i.e., when the ML system changes the environment it's trying to model; e.g., suggesting songs makes them more popular)
        - Data distribution shifts (see below)
    - These are: unique to ML systems; harder to detect.
- N.b., traditional software has data changes as well, but they tend to be structural rather than distributional. E.g.:
    - Traditional: WorkDay changed its course schedule CSV output format, which resulted in a system crash during I/O.
    - ML system: Course schedules slowly started including more 8am and online courses, which resulted in models failing to accurately predict course time scheduling distributions.
- We'll focus on the latter.
:::

## Data Distribution Shifts

We distinguish these distributions:

- *Source*
- *Target*

Rarely are these distributions:

- *Identical*
- *Stationary*

Detecting and addressing distribution shifts are crucial for maintaining ML system performance.

::: {.notes}
- The problem
    - Distributions (p. 237)
        - *Source* distribution: the distribution of the training data.
        - *Target* distribution: the distribution of the live inference data.
    - Properties (p. 230 - earlier in the chapter)
        - *Identical*: the source and target distributions are the same (e.g., *training-serving skew*) (cf. identically distributed)
        - *Stationary*: the target distribution doesn't change over time.
    - Distribution shifts (p. 237-242 - all interesting, but not in the assigned readings)
        - Concept drift; Covariate shift; Label shift
        - Feature change; label schema change
- The solution
    - Detecting DDSs (p.242-247 - not in the readings) - We'll:
        - Focus on general monitoring rather than on the mathematics of these shifts.
        - Log performance metrics.
    - Addressing DDSs (p. 248-250 - in the readings) - One could:
        - Just train on huge models. This has become common (e.g., GPT-type LLMs), but we won't (can't?) do it.
        - Adapt models without new labels. This is pretty theoretical, so we'll skip it.
        - Retrain using labeled target data. We'll say more about this later, in the Chapter 9 materials.
:::

## Monitoring ML Systems

If our systems are *observable*, we can *monitor* key metrics on:

- Raw inputs
- Features
- Predictions
- Accuracy

The latter metrics are easier to monitor and to interpret.

::: {.notes}
- In order to detect data distribution shifts, we need to monitor our ML systems.
- Terms
    - *Observability*: to configure the system in a way that allows monitoring (e.g., exposing the ML system's internal operations)
    - *Monitoring*: to log/watch selected metrics for help in discovering/debugging issues (e.g., logging the ML system's external performance)
    - [OPTIONAL] C. Huyen makes a big deal out of this distinction and of the rising importance of observability. It's good that we're discovering the value of observability, but this idea has been around for a long time.
- Artifacts to log/monitor
    - Raw inputs are usually monitored by data engineers rather than ML engineers.
    - Features can be monitored by tracking shifts in descriptive statistics (e.g., mean, max, min, var, ...).
    - Predictions can also be monitored using descriptive statistics (e.g., distributions of predicted values).
    - Accuracy metrics can be monitored if we can get feedback (e.g., user thumbs up/down). We're going to focus on this one.
:::

## Monitoring Tools

There are three main types of monitoring tools:

- Logs
- Dashboards
- Alerts

DMLS focuses on monitoring from the user&rsquo;s perspective.

::: {.notes}
- *Logs* of key operating values can help us debug/monitor the ML system.
    - "If it moves, we track it." (e.g.: BIT from Rockwell GPS; GE Aerospace logs enough flight data to build a 3D simulation of the flight/crash.)
    - Some companies use ML to analyze logs.
- *Dashboards* can make it easier to see what's going on, if we're careful what we display.
- *Alerts* can be configured to alert the right people about the most critical issues.
- We'll focus on logs and a REST-based dashboard.
:::

## Continual Learning

*Continual Learning* establishes an infrastructure for retraining models in production. We distinguish:

- *Stateful* retraining
- *Stateless* retraining

It can help address data distribution shifts and the *cold start* problem, but is challenged by data collection and model management.

::: {.notes}
- How often should we retrain our models?
    - In Chapter 7, myth #3 C. Huyen said that "How often should I retrain my model" is the wrong question; the right question is "How often can I retrain my model?" (p. 196)
    - That's not really the whole story. Probably, we should retrain more often than we do, but deciding how often is context dependent (p. 265).
    - In this chapter, C. Huyen discusses how to construct our infrastructure so that we can retrain more often if necessary.
- *Continual* learning (n.b., *continual" learning != "continuous" learning)
    - Has two basic approaches:
        - *Stateful retraining* is fine-tuning by continuing to train the existing model.
        - *Stateless retraining* is retraining from scratch. We'll focus on this (much simpler) one.
    - Potentially helps address:
        - Data distribution shifts.
        - The *cold start* problem, in which new users need good predictions based on little or no data.
    - Has challenges:
        - Getting data, i.e., you need crowd-sourced, natural, or automatic labeling.
        - Deciding when a new model is better than the old one. E.g., MS Tay showed how dangerous this could be.
        - *Catastrophic forgetting* is when the model forgets old data as it learns new data.
        - *Model drift* is when the model's performance degrades over time.
- C. Huyen suggests phased adoption (as usual):
    1. Manual, stateless retraining
    2. Automated, stateless retraining
    3. Automated, stateful retraining (on a fixed schedule)
    4. Continual learning (#3 on a flexible schedule)
- We won't be implementing this.
:::

## Test in Production

*Test in Production* is a practice of proactively testing models in production. We distinguish between:

- *Shadow deployment*
- *A/B testing*
- *Canary testing*

It&rsquo;s commonly used in large ML systems.

::: {.notes}
- Test in production (active) is "complementary" to monitoring (passive) (p. 263).
- C. Huyen suggests these techniques to reduce risk.
    - *Shadow deployment* involves running the new model in parallel with the old model and comparing their performance. It's safe but expensive.
    - *A/B testing* retains the two shadow versions of the model, but splits the traffic between them. It's cheaper but harder to do right.
    - *Canary testing* is a more advanced version of A/B testing that involves rolling out a new model to a small subset of users and gradually increasing the rollout.
- We won't be implementing this either.
:::
