ML Systems

Key objectives

After this course, I will be able to:

375:

APIs and systems
- I can create a computational notebook that includes code, execution results, section headings, and formatted textual explanations.
- I can write code that loads data, preprocesses it, and feeds it to a supervised ML model using a sklearn-style fit-predict API.
- I can select appropriate loss functions and metrics for a given task (and thus choose an appropriate model type/structure). Specifically, I can distinguish between regression and classification problems, even if classification targets happen to be encoded as numbers.
- I can integrate an ML model into a larger application.
Experimentation and Evaluation
- I can design, run, and analyze empirical experiments to quantify the impact of hyperparameter changes on model performance.
- I can make and interpret plots of relevant evaluation metrics.
- I can identify hyperparameters that can be adjusted to improve the performance of a model.

376:

I can solve basic language and reasoning tasks using commercially deployed LLM APIs.
I can apply and critically analyze evaluation strategies for generative models.
I can describe the overall process of training a state-of-the-art dialogue LLM such as Llama or OLMo.
I can analyze the computational requirements of training and inference of generative AI systems.

Learning Sequence

I’ve trained a classifier (using code already provided)

basic image clf notebook

I’ve called a LLM API

llm library

I’ve designed a prompt for an LLM

I’ve used a high-level API to run a neural network model

I’ve generated:

An array of class probabilities
A chatbot response
A next-token prediction
An image

I’ve computed the similarity between two embeddings and compared that with another pairwise similarity.

I’ve used the sklearn (fit-predict) API

I’ve evaluated the performance of a regression model and a classification model using appropriate metrics.

Contents

Feature Extractors and Linear Classifiers

Goals

We can explain the importance of good features for neural network models.
We have started to get a sense of how a nonlinearity can be useful for feature extraction.

Warm-Up: ReLU features

We’ll play with this model briefly:

ReLU Regression Interactive (name: u05n00-relu.ipynb; show preview, open in Colab)

For more, try out Figure 3.3a in the Understanding Deep Learning interactive figures

Feature Extractors

Image Classification: Losses and Feature Extraction (name: u05n1-img-classifier-feature-extractor.ipynb; show preview, open in Colab)

Exploring open-weights models on Hugging Face

Today we’re going to try out some of the open-weights models that are available on Hugging Face.

Objectives:

I’ve used a high-level API to run a neural network model.
I’ve generated an array of class probabilities.
I’ve computed the similarity between two embeddings and compared that with another pairwise similarity.

Background

Hugging Face is a company that provides a platform for sharing and using pre-trained models. They provide:

A model hub where you can find and download models.
A hosting service called Spaces where people can share their models.
Python libraries for working with models, including:
- transformers for working with models.
- datasets for working with datasets.

Explore

Let’s start by playing with a few Spaces, then we’ll try out some models.

Spaces

Go to the Spaces page. Try out two or three different Spaces:

One that’s highlighted or trending
One that is designed for a task that you might be interested in exploring for a project
One that is designed for a task that you’re not familiar with

A few that I tried out in February 2025:

RL
- Interactive DeepRL Demo - a Hugging Face Space by flowers-team
Sound / Music
- MusicGen Web - a Hugging Face Space by Xenova
- Riffusion • Spectrogram To Music - a Hugging Face Space by fffiloni
Images
- FLUX.1 [dev] - a Hugging Face Space by black-forest-labs
Agents
- Travel Planning Agent - a Hugging Face Space by acidtib (from an index at smolagents and tools gallery - a Hugging Face Space by davidberenstein1957 – see the “execution logs” in the chat results)

Embedding Models

Explore Sentence Embeddings (name: u08s1-sentence-embeddings.ipynb; show preview, open in Colab) (this one needs Colab; Kaggle doesn’t seem to support the tensorboard viewer that this uses.). Try computing the embeddings for some sentences that you create by hand, and then computing the cosine similarity between them. What do you find? Are similar sentences closer together in the embedding space?
Try the SigLIP demo that embeds images and text together. Try computing the dot products between a few texts that you write by hand. Does the dot product reflect the similarity of the texts? Repeat with images. What do you find? (This one definitely needs Colab.)

Use an AI to make an AI

Today we’re going to try out using an AI (a chatbot) to make an AI (an LLM-powered web-app).

Objectives:

I’ve used a chatbot to generate starter code.
I’ve called the OpenAI Chat Completions API to generate text.
I can explain what the inputs and outputs of the Chat Completions API are, especially its “messages” data structure.

Getting an API key

You’ll need an API key to access most LLM APIs. For this activity, we’ll use the OpenAI Chat Completions API - but we don’t necessarily need an OpenAI account! Several companies provide APIs that are compatible with OpenAI’s API format.

Think of it like phone chargers: even though we often call them “iPhone chargers” or “Android chargers”, any USB-C charger works with any USB-C device. Similarly, we can use any API that’s “OpenAI-compatible” with code that expects to talk to OpenAI’s API.

You have two options for getting an API key:

OpenAI: Create an account and add a few dollars of credit
Google AI Studio: Get a free API key that works with OpenAI-compatible code at Google Gemini API key page

Important: Save your API key in a safe place. You’ll need it to access the API, and the page only shows it once.

Setting up your API key

Before running any code, you need to store your API key in a secrets.toml file. This file should never be committed to version control. It lives in a special folder named .streamlit in your project directory.

Create a .streamlit/secrets.toml file with this content:

OPENAI_API_KEY = "your-key-here"

If you’re using the Google Gemini API key, use the same format; the code will still look for OPENAI_API_KEY since we’re using the OpenAI-compatible interface.

Using an LLM to generate starter code

First, choose what task you want to have the AI do. Here’s a few I’ve tried:

The user describes their reflections on something going on in their life, and the AI provides Scripture-grounded reactions.
The app uses the OpenAI API to play a game of tic-tac-toe with the user.
The user provides an image and the app makes a table of the items in the image with a detailed description of each one and its location in the image.
The user pastes an essay they’re working on and the AI coaches them on how to improve it.

Once you’ve chosen a task, use a chatbot to generate some starter code. For example, I prompted Claude with: “Can we build a Streamlit app that uses the OpenAI API to have the user play a game of tic-tac-toe?”.

To ensure that the generated code used the correct OpenAI API, I pasted an example of the setup code. Specifically, I went to the Gemini docs on OpenAI-compatible API and copied the example code snippet on that page, then at the end of the prompt I added: “Here’s a code example for the variant of the API we’re using.” and pasted the code snippet.

Note: you might need to change the GEMINI_API_KEY secret to OPENAI_API_KEY in the generated code.

Iterate with the chatbot

Don’t treat the chatbot as a one-and-done tool; have a conversation. For example, for the tic-tac-toe game, I followed up with “The AI opponent is pretty weak. Maybe it could explain a few moves that it’s thinking about before making its choice?” and then “I was getting an error because the AI response was ```json instead of the JSON object itself. That was hard to figure out because the st.rerun swallowed the error message.”

Running the Streamlit app

If you’re lucky, you just got some starter code for the app you want to build. Now to run it. Two options: local or Streamlit cloud.

Local setup

Install uv. This tool makes it really easy to manage your Python environment.
Create a new folder for this project.
Create the .streamlit/secrets.toml file as described above.
In that folder, run uv init and then uv add streamlit openai.
Paste the code that the chatbot generated into a file called streamlit_app.py.
Run uv run streamlit run streamlit_app.py. (Yes there’s two “run"s in that command.)

Streamlit cloud setup

Start with the Streamlit chatbot template. Click “Use Template” in the top-right corner.
Click Advanced and edit the Secrets box to include the content of your secrets.toml file. e.g.,:
```
OPENAI_API_KEY="your-key-here"
```
Choose whether to edit in Codespaces (in browser) or locally (clone the repo).
If using Codespaces, note that you may need to manually create the .streamlit/secrets.toml file with your API key (even though you set it in the template).

Reflecting on the OpenAI API

Find the part of the code that calls the OpenAI API.

What does the input to the API look like? Write a brief description of the data structure that is used to give the user’s input to the LLM.
What does the output of the API look like? You might add some print statements to the code to see the output.
Did the AI generate any error-handling code? What errors is it looking for, and how does it handle them?

Notice specifically that messages is a list of messages, both user and assistant.

Think about how you might use this API to do the code-generating task that we just did with the chatbot. Sketch out what the messages object might look like for a two-step conversation about generating and refining code.

Showcase

Once you have a working app, upload it to Moodle in the provided dropbox.

Note: you don’t have to include the answers to the reflection questions in your submission. They’re just to help you understand the code you’re working with. When using this activity to demonstrate meeting a course objective, expect to discuss these questions in a meeting with your instructor.

Key questions