System Monitoring Demos

Keith VanderLinden
Calvin University

• Integrate logging using:
  • Python logging for the full pipeline and the REST service.
  • MLFlow logging for the model training.
• Add a new REST API endpoint that serves up a measure of the model’s performance. (q.v., https://madewithml.com/courses/mlops/monitoring/#performance)
• TODO?
  • Tag new data via REST feedback on predictions.
  • Implement retraining in the pipeline.

Our Monitoring Features

• System Logging in:
  • MLFlow
  • Python
• Model Evaluation

1. Demo MLFlow and Python logging.
   1. Demo MLFlow logging (review).
      • Walk through makefile. Added MLFLOW variables.
      • Walk through src/train_model.py.
      • Walk through src.utilities.log_model_performance.
      • Run make mlflow.
      • Show the TagIfAI model in the MLFlow dashboard.
      • MLFlow focuses on the models; Logging covers everything else.
   2. Demo basic logging.
      • Delete the current logs/ sub-directory.
      • Walk through src/utilities.py.
        • Added the logger object.
      • Run python src/hello_logger.py.
        • Show the printed output (on the CLI).
        • Show the logged output (in logs/info.log).
   3. Demo logging for TagIfAI.
      • Walk through makefile.
        • Added the logs/ sub-directory.
        • Made it a condition of model-building.
      • Walk through src/train.py & src/api.py.
        • Show the log commands.
      • Run make.
        • Show the live logging.
2. Demo the evaluation endpoint end the REST API, which serves as a sort of “manual dashboard” monitor for the accuracy of the model.
   1. Walk through the code.
      • Added the /evaluate?since endpoint to: src/api.py, which requires:
        • utilities.py: Added train-until date to load-data.
        • train_model.py: Used the train-until date to ensure that we don’t mix train and test data.
        • Makefile: Set a since date.
      • This “simulates” a live data-collection process in which the model was:
        • Trained on Oct 1, 2010.
        • Testing late on the 15 new documents collected and tagged since that date.
   2. Run the system locally, demonstrating:
      • The new endpoint on the random model.
      • Specifying since or leaving it unspecified.
3. Demo the full pipeline using data562-tagifai running on Codespaces.
   1. Full pipeline logging. Get image of logfile output.
   2. Evaluation endpoint, called from local machine. Get image of CLI output.

References:

• https://madewithml.com/courses/mlops/logging/.

Snapshot - Logging

See the live demo.

Give snapshot of the logger.

Snapshot - Monitoring

See the live demo.

Give snapshot of the logger.