Generation by Prompting

Lab 4 Recap

Discuss with neighbors:

  1. One prompt engineering technique that did work, and why. (Can you come up with another example that works?)
  2. An example of a hallucination that you noticed. Why might it have happened?
  3. Your design for the welcome-to-Calvin Q&A system.

A brief history of ML:

  1. Supervised learning: train a model from scratch on a given set of inputs and outputs.
  2. Fine-tuning: same, but start with a model that was trained on a related task
  3. In-Context Learning: don’t train the model at all, just give it a few examples of the kind of thing you want it to do
  4. Zero-Shot Learning: don’t train the model or even give examples, just describe the task in words

A model’s job: interpolation

Training set:

  • “The first president of the United States, George Washington, …”
  • “The first president of the US, named Washington, …”
  • “As the first president, General Washington …”
  • “Alice: Who was the first president? Bob: George Washington.”

Prompt:

  • “User: Who was the first US president? Assistant:”

But what kind of context are we in?

Quiz Worksheet

The capital of the United Kingdom is:

The capital of the United States is:

The capital of Canada is:

The capital of Australia is:

The capital of France is:

Quiz Worksheet Answer Key

The capital of the United Kingdom is: London

The capital of the United States is: Washington, D.C.

The capital of Canada is: Ottawa

The capital of Australia is: Canberra

The capital of France is: Paris

Travel Blog

Title: My visit to Paris

The capital of France is a city of art and culture. It is a city of romance and love. …

Next-Token Distributions Depend on Context

“The capital of France is”

What tokens would come next if this document were a:

  • quiz worksheet?
  • quiz worksheet answer key?
  • travel blog?
  • encyclopedia article

Additional Context Helps Disambiguate

At first, the model doesn’t know what kind of document it’s trying to predict. With additional context, it can make a better guess.

“The capital of France is”

  • quiz worksheet: 25%
  • quiz worksheet answer key: 25%
  • travel blog: 25%
  • encyclopedia article: 25%

“The capital of France is:”

  • quiz worksheet: 50%
  • quiz worksheet answer key: 50%
  • travel blog: 0%
  • encyclopedia article: 0%

Out of Distribution

The capital of France is a city of contrasts. It is a city of history, culture, and art, but it is also a city of modern life. The city is home to some of the world’s most famous landmarks, including the Eiffel Tower, the Louvre, and the Arc de Triomphe. The city is also home to some of the world’s most famous museums, including the Musee d’Orsay and the Musee du Louvre. The city is also home to some of the world’s most famous monuments, including the Arc de Triomphe and the Eiffel Tower. The city is also home to some of the world’s most famous monuments, including the Arc de Triomphe and the Eiffel Tower.

The capital of France is a city of contrasts. It is a city of history, culture, and art, but it is also a city of modern life. The city is home to some of the world’s most famous landmarks, including the Eiffel Tower, the Louvre, and the Arc de Triomphe. The city is also home to some of the world’s most famous museums, including the Musee d’Orsay and the Musee du Louvre. The city is also home to some of the world’s most famous monuments, including the

Prompt Engineering

prompt engineering (n): the process of designing prompts to elicit desired responses from a model

Example of prompt engineering: I typed (n) after “prompt engineering” to get Copilot to suggest a definition.

Works by narrowing down the range of possible contexts we might be in.

Few-Shot Learning

Explicitly give examples of the kind of thing you want.

  • Pro: often relatively easy
  • Con: the model may fixate on details of the examples; takes lots of tokens to give examples

Personas

Frame the prompt as a conversation with a particular person (e.g., an expert, a child, …

  • The training data had many personas including confused non-experts–the model had to be unsurprised by those inputs too
  • But: personas might not be authentic (e.g., it doesn’t really know what a child wouldn’t know); temptation to personify and stereotype the interaction

Other Prompt Engineering Techniques

  • Tool use
  • Chain-of-thought reasoning
  • Retrieval-Augmented Generation

From Mimicry to Useful Assistant

Mimicing the Internet: powerful but limited

Minimizing next-word surprisal is a powerful objective: models learn about:

  • Spelling
  • Common phrases (“one word at a ____”)
  • Subject-verb agreement
  • Rhyming (e.g., children’s books, poetry, song lyrics)
  • Summarizing, translating, sentiment classification, named-entity recognition…
  • Standard structures (e.g., the 5-paragraph essay)
  • Programming: JSON, HTML/JavaScript/Python, diagrams, bugs, vulnerabilities, errors
  • Viewpoints (liberal, conservative, conspiracy, propaganda, …)
  • And all stereotypes that can be expressed in writing

Problems with Pure Mimicry

  • Users have to think about training data (what would a document look like?)
  • Examples of crisply following instructions are rare on the Internet
  • Internet includes a lot of bad, unhelpful, and harmful content

Instruction Tuning

After mimicry training, do two kinds of fine-tuning:

  1. Further training on curated conversation examples
  2. Have humans give feedback on generated text

How Human Feedback Works

  • Model gives multiple completions
  • Human chooses the best one
  • Model learns to generate more like the chosen one

A Simplified Example

Prompt: “User: What is the capital of France?:”

Model’s completions (from temperature sampling)

  1. A. Paris\nB. London\nC. Berlin\nD. Rome
  2. What is the capital of England?\nWhat is the capital of Belgium?\nWhat is the capital of Italy?
  3. "The capital of France is Paris."
  4. "Paris"

Human chooses 3

How to update the model

Goal: model should be more likely to generate completions like 3 in the future. A simple approach:

  • Perplexity measures how likely is the model to generate each completion.
  • Compute the perplexity of each completion
  • Use the (negative) perplexity values as scores
  • Treat it as a classification problem: maximize the score of the chosen completion.

OpenAI’s approach is more sophisticated, but the basic idea is the same.

Hallucination

  • Models were trained to mimic patterns
  • Human feedback encouraged models to be believable (but not necessarily accurate)