In this unit we extend our modeling skills to encompass classification models, and start to build the tools that will let us represent complex functions by using hidden layers. Both of these objectives require us to learn about nonlinear operations. We’ll focus on the two most commonly used ones: the softmax operator (which converts scores to probabilities) and the rectifier (“ReLU”, which clips negative values).
Students who complete this unit will demonstrate that they can:
Describe the difference between a metric and a loss function.
Describe and compute cross-entropy loss
Explain the purpose and mathematical properties of the softmax operation.
Explain the role of nonlinearities in a neural network (e.g., why they are used between linear layers)
Implement a logistic regression model using basic numerical computing primitives optional for 22SP
Preparation
The fastai course videos are a bit disorganized here, sorry about that.
This should reinforce what we’ve been studying about how linear regression works and how Tensors work, and give you a preview of how we’ll extend it to a full neural net.
Supplemental Material
We’re using Elo scores for intuition a few times this week, but we’re intentionally not diving deep on it. If you do want to dive deep: