Model Deployment

Keith VanderLinden
Calvin University

• In this module, we’ll moving the running of our model from the notebook to a Web service, where others can use it.

Deployment of ML Systems

Most production ML systems:

• Include more than just one model.
• Operate in dynamic, evolving environments.
• Update their models regularly.
• Operate at scale.

We’ll implement limited variants of these realities.

• These are the realistic counter-parts to C. Huyen’s myths of ML deployment.
• We’ll implement C. Huyen’s myths, keeping in mind that they are myths.
  • Include more than just one or two models.
    • Uber has thousands of deployed models.
    • We’ll do one or two models, potentially maintaining versions of each.
  • Operate in dynamic, evolving environments.
    • Uber’s models must adapt to changing customer and traffic patterns.
    • The TagIfAI domain isn’t that dynamic; SLO was; your project may be as well. But we won’t be working on the models long enough to see them evolve very much.
  • Update their models regularly.
    • AWS deploys every 11.7 seconds.
    • We won’t be updating our models that regularly.
  • Operate at scale.
    • Google’s ML models serve billions of users.
    • We won’t operate at scale.

Prediction Modes

The basic prediction modes are:

• Batch prediction
• Online prediction
  • using batch features
  • using streaming features

There are many variations and combinations.

• From the UX perspective, we distinguish:
  • Batch (off-line) systems serve predictions from pre-computed results (q.v., DMLS Figure 7-4).
  • Online (real-time) systems serve predictions from live computation.
• From the feature engineering perspective, we distinguish:
  • Batch features come from pre-computed, stored data (q.v., DMLS Figure 7-5).
  • Streaming features come from live data, which only makes sense in online systems (q.v., DMLS Figure 7-6).
• Batch & streaming features can be used in combination, but we’ll be implementing an online system with a pre-computed model that uses only streaming features (a single text to be classified).
• Interestingly, C. Huyen suggests that compute power will eventually make online systems the default.

Batch vs Streaming Features

DMLS Figure 7-8

• We discussed this in the context of feature engineering. Batch and streaming features must be engineered in the same way (see the “should be equal” line).
• The feature inputs to the model could be a combination of live and batched data; the diagram doesn’t show that. That is, the diagram shows what we’ll be doing for TagIfAI, minus the streaming.

Computation Architecture

Model computation can be done on either:

• Cloud
• Edge

The first is the easiest and most common.

• ML computation can be done on the cloud or on the edge.
  • Cloud computing is centralized on the server. It’s:
    • easier.
    • more expensive in terms of shared server compute resources.
  • Edge computing is distributed to the clients. It’s:
    • more challenging because it’s distributed to heterogeneous clients.
    • cheaper in terms of server compute resources.
    • can be run off-line.
    • quicker in terms of UX.
    • more private for users.
• We’ll do ours on the cloud (CodeSpaces).
• One interesting edge example is running models in the web browser (e.g., using WASM).