376 Preparation 3

Warning: This content has not yet been fully revised for this year.

• Watch 3blue1brown - Attention in transformers, visually explained
• Skim through this LLM visualization
• Read Evaluation Metrics for Language Modeling; stop at “Reasoning about entropy as a metric”

Supplemental Material

• A story-like derivation of Transformers: Transformers from Scratch
• Review the Thinking like Transformer blog post.
• Other study material:
  • Transformers Study Materials at a range of levels of detail.
  • Twitter threads more your thing? Part 1, Part 2
• Transformer architecture: Annotated Transformer

Supplemental: HuggingFace NLP Course

• Read chapter 2 of the Hugging Face NLP Course; do the end-of-chapter quiz.
• Read chapter 3 of the course. Do the end-of-chapter quiz. Additionally, be able to answer the following questions:
  • Section 2
    • In the first code chunk:
      • Was the model given the desired output for each sentence?
      • For how many iterations was the model trained?
      • Review: What does loss.backward() do?
      • Note: In case you haven’t seen an optimizer object before, go back to the Extension section of Fundamentals u05n3
    • What information is contained in each row of the MPRC dataset?
    • How does the tokenizer tell the model which part of the input is the first sentence vs second sentence?
    • Why do we need to pad the the inputs?
  • Section 3
    • What does a Trainer do?
    • What information do you need to pass when constructing a Trainer?
    • What information do you need to pass when computeing a metric? What information is given in the results?
      • note: f1 summarizes a model’s accuracy in a way that balances precision and recall. Technically, it is the harmonic mean of precision and recall. It’s not perfect, but it’s very commonly used.
  • Section 4
    • Note: look at the for - break. That’s a useful Python trick for debugging iterable things (like data loaders) in notebooks.
    • What does model(**batch) give us? (Note: the ** means to pass everything in batch as keyword arguments (“kwargs”) to the function. So gets parameters like input_ids=SOMETHING, attention_mask=SOMETHING, labels=SOMETHING.)
    • Be able to explain what each line of code in the code chunk right before “The evaluation loop” does.
    • You can skip the section on accelerate.