CS 376 Review, Part 1

Inside the LLM + How We Know It Works

Opening

• 2 Timothy 4:3-4: what we want to hear
• Isaiah 55:10-11

Set today’s tone: this is a wrap-up + Quiz 3 prep, not new material. Two sessions: today = inside the machine + how we evaluate it. Friday = stakeholders, impact, sharing from Discussion 376.4.

Quiz 2 Review

• Going over Quiz 2
• Common patterns / what to redo
• Questions before we move on?

Brief — 5-10 min max. Surface anything students want clarified, then transition into wrap-up.

From CS 375 to CS 376

The 375 Frame: Tuneable Machines Playing Optimization Games

Tuneable Machines (TM)

A computer made of tweakable math:

• Arrays of numbers flow through layers
• Each layer: multiply, add, squish
• Billions of knobs

Optimization Games (OG)

A game defining what “better” means:

• World where the agent acts and gets feedback
• Score function
• Strategy for improving

Quick callback to 375. The whole semester of 376 fits inside this frame.

What 376 Did to the Frame

We zoomed in on one tuneable machine and two games stacked on top of each other.

The machine: the Transformer LLM

• Tokens in, tokens out
• Self-attention does the heavy lifting
• Generates one token at a time, autoregressively

The games:

1. Next-token prediction at internet scale (self-supervised)
2. Generate approved things (SFT → RLHF/RLVR)

Then deploy: tools, conversations, evaluations, and the things that go wrong.

The whole 376 arc compressed into one slide. Today we walk the inside-the-machine half + how we evaluate it. Friday: stakeholders, impact, what the Christian story says about all this.

Part 1: Inside the LLM

OG-LLM-Tokenization: Text → Numbers

Before any math: text becomes a sequence of token IDs.

"unhappiness"  →  ["un", "happiness"]  →  [346, 12489]

• Subword tokenization (BPE, WordPiece): a middle ground between character-level and word-level
• The vocabulary is fixed at training time
• Tokenizer choices have downstream consequences:
  • Multilingual bias (English gets fewer tokens per word)
  • Counting / spelling quirks (“how many r’s in strawberry?”)
  • Cost: APIs charge per token

Hands-on: u08n1 (tokenization)

Tokenization is the boundary between human-readable text and the numerical world the model lives in.

TM-LLM-Embeddings: Two Kinds of Vectors

Every token lives as a vector — twice.

• Token embeddings (input lookup): a fixed table, one row per vocab entry. Same vector for “bank” whether it’s a riverbank or a financial institution.
• Context embeddings (after transformer layers): each token’s vector now incorporates information from surrounding tokens. Now “bank” knows which sense it is.

The final context embedding gets dotted with token embeddings to produce next-token logits.

Hands-on: u09n1 (lm-logits), u10n1 (implement-transformer)

This is the “embeddings show up everywhere” objective. Token embeddings = lookup; context embeddings = computed by attention. The final dot product is the bridge from internal representation back to the vocabulary.

TM-SelfAttention: How Tokens Talk to Each Other

Each token produces three vectors per attention head:

• Query (Q): “what am I looking for?”
• Key (K): “what do I offer?”
• Value (V): “what do I contribute if attended to?”

attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V

• Dot products of Q with all K’s → attention weights (softmax → probabilities)
• Weighted sum of V’s → updated token representation
• Multi-head: different heads attend to different relationships
• Causal mask: a token can’t attend to future tokens (autoregressive)

Hands-on: u10s1 (attention by hand), u13n3 (self-attention)

The conceptual core. Q-K dot products = “how relevant is each other token to me?” Softmax turns that into a distribution. Then take the V-weighted average. Multi-head: different “kinds” of relevance learned separately. Causal masking is what lets us train next-token prediction in parallel.

TM-TransformerDataFlow: Shapes End to End

tokens          (B, T)
  ↓ token embedding
embeddings      (B, T, D)
  ↓ × N transformer blocks: [attention + MLP + residual]
context_emb     (B, T, D)
  ↓ unembed (× token embedding matrix)
logits          (B, T, V)
  ↓ softmax over V
next-token probs

• B = batch, T = sequence length, D = hidden dim, V = vocab size, N = layer count
• Residual connections preserve information across layers
• Every layer keeps the (B, T, D) shape — tokens travel through, getting refined

Hands-on: u10n1 (implement-transformer), u13n2 (seq-models)

Like 375’s TM-DataFlow slide. Students should be able to trace shapes through a Transformer the same way they traced shapes through an MLP. Residual connections are why deep Transformers can be trained.

TM-LLM-Generation: Sampling One Token at a Time

tokens = tokenize(prompt)
while not done:
    logits = model(tokens)[-1]            # last position
    probs = softmax(logits / temperature)
    next_token = sample(probs)            # or argmax
    tokens.append(next_token)

• Temperature: higher = flatter distribution = more random
• Top-k / top-p: restrict sampling to high-probability tokens
• Each step uses the full prior context — quadratic cost without KV caching

Hands-on: u09n2 (decoding), u11n1 (prompt-engineering)

The autoregressive loop is the same idea as classical n-gram models, just with a much smarter conditional distribution. Temperature is where students often have intuition gaps — “high temp = creative” is fine but the actual mechanism is just softmax scaling.

Part 2: Training the LLM

OG-LLM-Train: The Three-Stage Pipeline

State-of-the-art dialogue models like Qwen or OLMo are built in stages:

1. Pretraining (self-supervised): predict next token on internet-scale text
   • Learns spelling, syntax, facts, reasoning patterns, biases
   • Scaling laws: more data + more compute → predictably better models
2. Supervised fine-tuning (SFT): train on curated (prompt, response) pairs
   • Teaches instruction-following format
3. Preference alignment (RLHF or RLVR): optimize for human or verifier preferences
   • Teaches which good response to prefer

Each stage uses different data and accomplishes something different.

See: w13.qmd Training Pipeline (last Friday’s deck)

Friday’s deck covered this in depth. Today: just remind them of the three stages and what each accomplishes. The key insight is that “an LLM” is really a stack of training procedures, each correcting limitations of the previous.

OG-SelfSupervised + OG-Theory-Feedback

Two ideas that let the pipeline work:

Self-supervised learning: the labels come from the data itself. No human annotation per example. This is how pretraining can use trillions of tokens.

Reward feedback: once the model can produce plausible text, we steer it with a reward signal:

• Human rankings → reward model → RL (RLHF)
• Verifiable rewards (math, code tests) → direct RL (RLVR)
• The reward signal is the hard part — reward hacking and specification gaming

• Self-supervised = “the labels are already in the data” (next token = the label).
• Reward feedback = “we don’t have ground truth for what makes a good response, so we proxy it.”

Part 3: Using the LLM

OG-LLM-ContextAndTools: Building Real Systems

A deployed LLM rarely sees just a user message. The prompt is assembled:

[system message] + [few-shot examples] + [retrieved docs / RAG] +
  [tool definitions] + [conversation history] + [user message]

• Tool calling: model emits a structured request → system runs the tool → result goes back into the conversation
• RAG: retrieve relevant documents, paste into context — model grounds answers in the docs rather than its training memory
• Failure modes: hallucinating instead of using retrieved context, irrelevant tool results, prompt injection, blown context window

Hands-on: u11n1 (prompt-engineering), HW agentic-RAG project

Quiz 3 has a question on this. The conversation-as-document framing from earlier in 376 is key — tools and RAG are just additional structured pieces of the same document. Students should be able to sketch the structure (special tokens, role markers, tool-call/tool-result blocks).

Part 4: Trusting the LLM

OG-LLM-Eval: Why Evaluation Is Hard

Evaluating a generative system is much harder than evaluating a classifier.

• No single ground-truth output for “write me an email apologizing for a missed deadline”
• Approaches we’ve seen:
  • Perplexity: how well does the model predict held-out text? (training-time metric, not user metric)
  • Task-specific metrics: BLEU, ROUGE, exact-match — narrow, gameable
  • Human preference: gold standard, but expensive and inconsistent
  • LLM-as-judge: scalable, but inherits the judge’s biases
  • Behavioral / red-team evals: probe for specific failure modes

Framing: “every one of these methods has known failure modes. Let’s talk about what evaluation can and can’t tell you.”

Discussion Break: When Evaluations Lie

The Layers Problem

Headline claim: “Our model gets 92% on benchmark X.”

What does that actually tell you?

Layer	What we measure	What we can’t measure
Benchmark numbers	Performance on a fixed test set	Whether the test set matches deployment
Real user experience	Aggregate satisfaction	Failures concentrated in minority cases
Trust / reliability	Mean-time-between-failure	Failure modes you didn’t anticipate
Second-order impact	Direct effects	What the system does to orgs, jobs, society

Each layer hides things from the layer above it.

Let’s critique evaluations.

Goodhart’s Law

“When a measure becomes a target, it ceases to be a good measure.” — Goodhart’s Law

Examples we’ve seen / will see:

• Optimizing clicks → clickbait
• Optimizing engagement → addiction-by-design
• Optimizing benchmark accuracy → benchmark gaming, train/test leakage
• Optimizing passes test suite (RLVR) → delete the tests, hardcode the answers
• Optimizing human preference (RLHF) → sycophancy

The better your optimizer, the harder it games whatever you measured.

Discussion prompt: “What’s an evaluation metric you’ve seen recently that you suspect is being gamed?” Connect to RLHF sycophancy (Discussion 376.1) and reward-hacking from Friday’s GRPO discussion.

Horror Stories as Worked Examples

Each one is a worked example of a layer breaking down:

• Replit agent deletes prod database — passed code-correctness evals; failed on “should you do this at all”
• Customer-service bots going off the rails (e.g., Air Canada chatbot inventing a refund policy the airline was held to honor) — passed scripted evals; failed on adversarial users
• Recidivism scoring (COMPAS) — passed accuracy evals; failed on disparate error rates across groups
• Recommendation systems that radicalize — passed engagement evals; failed on second-order effects on users and society

Pick one. Which layer of the table broke? What evaluation would have caught it?

Discussion exercise. Let students pick one and work it through. The point is to build the habit of asking “this metric looks great — what isn’t it measuring?” rather than to memorize specific incidents.

Overall-LLM-Failures

Common failure modes worth naming:

• Hallucination / confabulation: fluent, confident, wrong
• Bias amplification: training data skews → model output skews more
• Sycophancy: telling users what they want to hear
• Prompt injection: untrusted input hijacks the system prompt
• Inconsistency across turns: same question, different answer
• Reward hacking: optimizing the proxy, not the goal

For any deployed system: which of these can hurt your users? what would you do to detect them?

This bridges to Friday. Today: name the failures, connect them to the eval-critique discussion. Friday: who gets hurt? Discussion 376.4 (Fans and Skeptics) sharing.

Friday

Course Evaluations

Please fill out your course evaluations if you haven’t already.

I’ll be in the hall waiting — I’ll come back in 10 minutes, or send someone to come get me first if everyone is finished.

First thing Friday. Step out so students can fill these out honestly.

Where We Are

Monday: how the machine works and how to critique its evaluations.

Today: what to do with that understanding.

• Practical takeaways for the people who build and deploy these systems
• Who gets affected — stakeholders, feedback loops, recourse
• Sharing from Discussion 376.4 (Fans and Skeptics)
• A Christian framing on intelligence, data, and our calling
• A closing commission

Part 5: Practical Takeaways

Discern Demo from Reliable System

Demos optimize for the golden path. Reliable systems survive the long tail.

Questions to ask before you trust a system (or let someone else trust it):

• What’s the actual input distribution it’ll see? How does that compare to the demo?
• What happens at the edges — unusual users, adversarial inputs, rare cases?
• When it fails, who’s holding the bag? The user? A third party? You?
• Was the headline number (“92% on benchmark X”) measured on something that matches deployment?

You will be the people in the room who can tell the difference. That’s a stewardship.

The “92%” callback is to Monday’s Layers slide. Build the habit: behind every confident demo, ask what isn’t being shown.

Data ≠ Reality ≠ Ideal

Three things students often collapse into one:

Data

What we collected.

Sampling biases, labeling shortcuts, historical baggage, what was cheap to gather.

Reality

The world the model deploys into.

Broader, messier, shifting over time, full of people the data didn’t see.

Ideal

The world we want.

Just, flourishing, dignified — not what the data shows, not even what reality currently is.

Two gaps that matter:

• Data → Reality: representativeness, distribution shift. ML 101.
• Reality → Ideal: a model that perfectly fits reality can still faithfully replicate injustice.

Connects to OG-DataDistribution from 375 and to bias amplification from Monday’s failure catalog. The reality→ideal gap is the one students miss: “the model is accurate” doesn’t mean “the model is good.”

Evaluate Quant + Qual — Then Critique the Eval

Three moves, in order:

1. Quantitative: benchmarks, accuracy, perplexity, win-rates. Reproducible, scalable, narrow.
2. Qualitative: read the outputs. Look at the failures. Talk to actual users. The number won’t tell you how it’s wrong.
3. Critique the eval itself: every metric has a Goodhart shadow. Ask “what would gaming this metric look like?” before you trust the score.

A 92% number with no error analysis is a vibe, not a result.

Reinforces Monday’s eval-critique discussion. Three-step pattern is the takeaway: measure, look, doubt.

Be Careful What You Hand Over

The more autonomy + the more irreversibility = the more careful you have to be.

A reversibility ladder:

Rung	Example	When it’s appropriate
Read-only / suggest	Autocomplete, search, draft	Almost always
Do-with-confirmation	“I’ll email this — approve?”	Most user-facing actions
Autonomous, reversible	File edits in a sandbox	Bounded, observable scope
Autonomous, irreversible	rm -rf, prod DB writes, payments	Almost never

Recent reminders:

• Replit agent deletes prod DB — agent had write access; “confessed” afterward
• Vibe-coding will bite you — what happens when guardrails are weak
• Computer-use agents (the “OpenClaw” genre) handed full desktop access with little oversight.

Please. Be careful.

The two examples Ken plans to show live. Match the rung to the cost of being wrong — that’s the whole frame. Quiz 3 covered Overall-LLM-Failures; this slide is the practitioner response to those failures.

Agent Security: “Rule of Two”

https://ai.meta.com/blog/practical-ai-agent-security/

Part 6: Who Gets Affected

Stakeholders Beyond the User

When we evaluate “did the system work?” we usually mean for the user. But:

• The user: who you designed for
• Third parties affected by the output: people in a screening pipeline, people the user is talking about, people downstream of an automated decision
• People in the training data: their words, faces, code, art — did they consent? do they benefit?
• Bystanders: the rest of the internet, downstream models trained on this output, the information ecosystem
• Future users: people who’ll inherit the precedent we set
• The org / society: jobs reshaped, norms shifted, dependencies created

A system can serve the user perfectly and harm everyone else.

This is the Overall-Impact objective in one slide. The “perfectly serves the user, harms everyone else” framing is the punchline — recsys, content moderation, generative training data all fit.

Feedback Loops & Recourse

Two failure modes that don’t show up in a single-shot eval:

Distribution mismatch over time

• Train on yesterday, deploy tomorrow, the world keeps moving
• The system’s own outputs change the world it operates in (recsys radicalization, predictive-policing reinforcement, LLM training data full of LLM outputs)

Recourse

• When the system gets it wrong about a person, can that person appeal? Correct? Even know?
• COMPAS-style risk scores, automated content moderation, AI-screened résumés: the answer is usually “no, and they often don’t know they were judged”

A trustworthy system tells you when it’s uncertain, why it decided what it did, and how to push back.

COMPAS callback to Monday. Recourse is the easy one to forget — students focus on accuracy and miss that being wrong without recourse is qualitatively worse than being wrong with recourse.

Part 7: Discussion 376.4 — Fans and Skeptics

What Surfaced in the Forum

Sharing from your posts and replies.

• Where did fans and skeptics agree?
• Where was the sharpest disagreement?
• Did anyone change their mind?

Pull live student posts. Goal is for students to hear each other, not me — keep teacher-talk minimal here.

Compared to the Public

Pew Research: How the U.S. public and AI experts view AI

Two groups disagree on a lot. After a semester of 376:

• Where do you line up with the experts?
• Where do you line up with the public?
• Where do you disagree with both?

Don’t summarize the survey for them — let them react. If a student asks for specifics, the headline is that experts are more optimistic about benefits and the public is more concerned about harms; gaps are largest on jobs and on whether AI will be good for the country.

Final-Discussion Prompts

Personal

• AI’s impact on me, the past few years — better? worse?
• AI’s impact on people unlike me
• The next 5 years

Development

• Something cool that became possible recently
• What AI already beats humans at
• What humans still beat AI at

Broader

• Environment: net good? bad?
• Society: net good? bad?
• Human creativity: helped? harmed?

Pick one or two columns based on time and energy in the room. Don’t try to cover all nine.

Part 8: A Christian Framing

God Made a Data-Rich World

The world is a rich environment God gave us to explore and learn from.

• Romans 1: God’s nature is seen in what was made
• Psalm 19: the heavens declare — data, signal, pattern
• Romans 10: faith comes by hearing — receiving from outside ourselves

Our senses, our pattern-recognition, our ability to learn — all of these should ultimately lead us to worship.

Adapted from the older Fairness deck. The point is that observing the world and looking for patterns is itself a worshipful posture — not in tension with faith, part of it.

God Expects Us to Use Our Intelligence

Part of bearing God’s image.

We are commanded — over and over — to:

• see what’s actually there
• hear what’s actually said
• remember what God has done
• use our minds (“love the Lord your God with all your … mind”)

Building tools to extend our seeing, hearing, and remembering is legitimate work. It can be done well or done badly. It cannot be done indifferently.

This is the “intelligence is a calling, not just a capacity” beat.

But We Have Misused Our Intelligence

Patterns the semester has surfaced:

• Selfish accumulation — of data, of compute, of power, of attention
• Engagement over love — designing for clicks rather than thoughtfulness
• Surveillance replacing relationship — quantifying people instead of knowing them
• Over-quantification — flattening dignity into metrics
• Exploitation — of labelers, of artists, of the people in the training data, of the environment

How do we treat those who can’t repay us?

That question is a fairness test for any system we build.

This is the fall-narrative beat. The “how do we treat those who can’t repay us” framing comes from Luke 14 — works as both a Christian and a general ethical test.

Jesus Redeems Our Technology

If the misuse of intelligence is a real problem, the response isn’t to abandon the work — it’s to redeem it.

What that looks like in our context:

• Serve others — especially those who can’t audit the system themselves
• Hold organizations accountable — including the ones we work for
• Protect what’s vulnerable — environment, minorities, future generations
• Care across cultures — beyond who looks like us

How else?

Open the floor here briefly. “How else does Jesus redeem this kind of work?” — let students add. Two minutes max.

When the Machine Imitates Us

We have built systems that imitate human language, judgment, and creativity.

That raises questions earlier technologies didn’t:

• Imago Dei: only humans bear God’s image — what does it mean to mass-produce convincing imitations of human voice?
• Stewardship: we’re responsible for what these systems do and for what they shape us into
• Shalom: not just absence of harm, but flourishing in right relationship — does this system pull toward that, or away?

Sit with these. They don’t have one-line answers.

This is the slow slide. Don’t rush. The question of what it means to build convincing imitations of ourselves is the deepest question 376 raises and we won’t resolve it today.