---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## System Monitoring Demos

Keith VanderLinden<br>
Calvin University

::: {.notes}
- Integrate logging using:
    - Python logging for the full pipeline and the REST service.
    - MLFlow logging for the model training.
- Add a new REST API endpoint that serves up a measure of the model's performance.
    (q.v., <https://madewithml.com/courses/mlops/monitoring/#performance>)
- TODO?
    - Tag new data via REST feedback on predictions.
    - Implement retraining in the pipeline.
:::

## Our Monitoring Features

- System Logging in:
    - MLFlow
    - Python
- Model Evaluation

::: {.notes}
1. Demo MLFlow and Python logging.
    a. Demo MLFlow logging (review).
        - Walk through `makefile`. Added MLFLOW variables.
        - Walk through `src/train_model.py`.
        - Walk through `src.utilities.log_model_performance`.
        - Run `make mlflow`.
        - Show the TagIfAI model in the MLFlow dashboard.
        - MLFlow focuses on the models; Logging covers everything else.
    b. Demo basic logging.
        - Delete the current `logs/` sub-directory.
        - Walk through `src/utilities.py`.
            - Added the `logger` object.
        - Run `python src/hello_logger.py`.
            - Show the printed output (on the CLI).
            - Show the logged output (in `logs/info.log`).
    c. Demo logging for TagIfAI.
        - Walk through `makefile`.
            - Added the `logs/` sub-directory.
            - Made it a condition of model-building.
        - Walk through `src/train.py` & `src/api.py`.
            - Show the log commands.
        - Run `make`.
            - Show the live logging.
2. Demo the evaluation endpoint end the REST API, which serves as a
        sort of "manual dashboard" monitor for the accuracy of the model.
    a. Walk through the code.
        - Added the `/evaluate?since` endpoint to: `src/api.py`, which requires:
            - `utilities.py`: Added train-until date to `load-data`.
            - `train_model.py`: Used the `train-until` date to ensure that we don't mix train and test data.
            - `Makefile`: Set a `since` date.
        - This "simulates" a live data-collection process in which the model was:
            - Trained on Oct 1, 2010.
            - Testing late on the 15 new documents collected and tagged since that date.
    b. Run the system locally, demonstrating:
        - The new endpoint on the random model.
        - Specifying `since` or leaving it unspecified.
3. Demo the full pipeline using data562-tagifai running on Codespaces.
    a. Full pipeline logging. Get image of logfile output.
    b. Evaluation endpoint, called from local machine. Get image of CLI output.

References:

- <https://madewithml.com/courses/mlops/logging/>.
:::

## Snapshot - Logging

*See the live demo.*

![](images/demo-logging.png)

::: {.notes}
Give snapshot of the logger.
:::

## Snapshot - Monitoring

*See the live demo.*

![](images/demo-monitoring.png)

::: {.notes}
Give snapshot of the logger.
:::
