• Unit 1: Introduction
• Unit 2: Data
• Unit 3: Data and Ethics
• Unit 4: Models
• Unit 5: Learning and Classification
• Unit 6: Recap and Regularization
• Unit 7: Learned Representations (Embeddings)
• Unit 8: NLP Intro
• Unit 9: NLP Modeling
• Unit 10: Transformers
• Unit 11: Generation
• Unit 12: Review, Neural Architectures
• Unit 13: Generation and Miscellaneous Topics

Unit 6: Recap and Regularization

In this unit, after reviewing where we’ve been, we push towards state-of-the-art models (still focusing on computer vision). We’ll first show how our work last 2 weeks connects to the pre-trained models we used in the opening weeks. Then, we’ll introduce or revisit tools that allow our models to achieve high performance, such as data augmentation and regularization. Finally, we’ll get more practice with how neural networks work from the ground up as we implement our own simple neural net image classifier from scratch (definitely something to mention in an interview!). Students who complete this unit will demonstrate that they can:

• Explain how a pre-trained model can be repurposed for a new task by separating it into a general-purpose “body” (aka “encoder”) and a task-specific “head”.
• Identify some examples of data augmentation and regularization.
• Predict the effect of data augmentation and regularization on model training.
• Implement a multi-layer neural network using basic numerical computing primitives

Preparation

The fastai course videos are still a bit disorganized, sorry about that.

• Read Deep Learning for Coders chapter 6 (open in Colab). Pay attention to:
  • What has to change if classes are or aren’t mutually exclusive?
  • What has to change if you want a regression instead of classification?
• Skim Deep Learning for Coders chapter 7 (open in Colab).
• If you haven’t yet, read Deep Learning for Coders chapter 17 (open in Colab).
• Video
  • Some of this material is part of the video you already watched, FastAI Course Lecture 3. Then some of it gets delayed to FastAI Course Lecture 8, I know, it’s confusing, sorry.
  • Instead, if you want to watch something, I recommend getting ahead with next week’s videos, because they split some material between two different videos again. See Prep 7 for the details.

Supplemental Materials

A nice intuition about why layers matter: Why depth matters in a neural network (Deep Learning / AI) - YouTube

Strategies for getting state-of-the-art performance:

• Simple considerations for simple people building fancy neural networks
• How to be an effective Deep Learning Researcher/Engineer

We’ll be doing some automatic differentiation this week:

• Yet another backpropagation tutorial by Harvard CS Professor Boaz Barak Overview of PyTorch Autograd Engine | PyTorch
• Automatic differentiation in machine learning: a survey
• autograd-for-dummies: A minimal autograd engine and neural network library for machine learning students.

Finally, I sometimes remark that “machine learning is lazy” (in that it tends to focus on superficial easy features). Here’s a more precise statement of a related claim: What do deep networks learn and when do they learn it. A recent paper describes what to do about it: Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks | Abstract

Class Meetings

Contents

• Lab 6
• not-sure-what-to-call-this-assignment
• Week 6 Q&A