Plan:
- Exodus 21:33-36
- Discussion topic presentation: Autonomous Vehicles
- PyTorch can compute the gradient for us
- Intuition of backpropagation
- Not numerical gradient computation. It’s “ill-conditioned” (viz. delta of 1e-3 vs 1e-4)
“Using the limit definition of the derivative for finite difference approximation commits both cardinal sins of numerical analysis: “thou shalt not add small numbers to big numbers”, and “thou shalt not subtract numbers which are approximately equal”. Automatic differentiation in machine learning: a survey
- Not symbolic differentiation
- Not numerical gradient computation. It’s “ill-conditioned” (viz. delta of 1e-3 vs 1e-4)
- How: tell it we want the gradient; call
result.backward()
- Intuition of backpropagation
- Stochastic Gradient Descent: we only need a few points to find a descent direction
- Extend to a linear combination of basis functions
- PyTorch has a
Linearlayer. - Loss functions and their gradients: MSE vs MAE