Warning: This content has not yet been fully revised for this year.
Readings in Perusall:
- Deep Learning with Python chapter 3
Additionally, work through the following interactive articles:
- Neural Networks
- Note: when they draw a box with an activation function in it, they actually mean a linear layer with that activation function on the output
Prep Notes
- For this week, focus on how things are used rather than the underlying math
- The book uses “rank” to refer to the number of axes of a tensor, but “rank” means something different in linear algebra. To avoid confusion, let’s call it “number of axes”, or perhaps “number of dimensions” (abbreviated “ndim” in PyTorch).
- For example, a length-5 column vector times a length-4 row vector would give a matrix (tensor) with two axes (2-dimensional), with shape (5, 4) and rank 1 in the linear algebra sense. See this notebook.
Optional Material
Watch Zero to Hero part 1: The spelled-out intro to neural networks and backpropagation: building micrograd
- First, watch the video and take conceptual notes.
- Then, do these exercises
- Then, re-watch and write the code along with him.
There are too many videos out there on deep learning to list here, but here’s a few very different styles:
- (MIT) Lecture 1 of MIT 6.S191 Deep Learning
- (elementary) CrashCourse AI Playlist
- (mathy) 3Blue1Brown Neural Networks (highly recommended channel for math, but these particular videos seem less intuitive than their typical videos)
I made a video walking through backpropagation on a simple example.
Elo Notes
We’re using Elo scores for intuition a few times this week, but we’re intentionally not diving deep on it. If you do want to dive deep: