Warning: This content has not yet been fully revised for this year.
Projects
Text and Image Generation – short and very clear YouTube videos
Reading Recess Reminders
Midterm
- Academic Integrity: Obviously I’m trusting you to do your own work on the midterm and not to discuss it with others. If you have questions about what is and isn’t allowed, please ask me.
- Work on paper. If you would like a printout, stop by my office and I’ll print one for you.
- Clarifications: A few have already been posted. Please don’t hesitate to ask if anything is unclear. You are welcome to send me an answer that you’re not sure about; if it shows that something is unclear in the question, I’ll clarify it for everyone.
- Study Guide: There were some things on the study guide that we didn’t go over in class. So feel free to ask me about those (by Teams, or Ed, or in person)
- Time: If you’re spending more than 2 hours on it, that probably means you’re stuck on something. Please reach out to me, and in the meantime, work on your project.
Project
- Here’s an example template for presentations, just so you don’t spend time worrying about how to structure or format it.
- Check-ins: if you haven’t yet had a check-in meeting with me, please reach out to me so we can schedule one. I’m available until 3:30 today, and then again on Monday and Tuesday.
- Going deeper: The first rubric item is “The project dug below the surface in some way, to connect to fundamental ML concepts.” If you’re not sure what that means, here are some ideas:
- How do your modeling choices relate to your task? e.g., you chose a specific pretrained model (over another one) or augmentation strategy because of some characteristic of your data. You chose a non-obvious loss function or evaluation metric because of the task you’re addressing. etc.
- What did the model learn? e.g., you make a hypothesis about what it learned, then test that hypothesis in some way (e.g., by predicting how it would work on some examples you put together by hand, by visualizing the weights in some way, by looking at the model’s internal representations for different dataset samples, etc.)
- What are the limitations of your model? e.g., you identify some ways in which your model could be improved, or some ways in which it might not generalize well to other data.
- What improvements might you make to your model? even if you don’t have time or skill to implement them, you can still describe what the improvements might be and why they might be helpful.
- etc.
Final Week
Congratulations on making it to the end of the semester!
Timeline of the rest of the semester
- Monday, Wednesday, Thursday in lab. Schedule: 15-20 minute review, 30 min project work time
- Thursday: Midterm 2 released, due no later than final project presentations
- Thursday: last day to submit assignments and revisions
- Wednesday 4/26 9am: Final project presentations (in lab)
Notes
- There’s one Final Homework and one final Discussion. Choose one or the other.
- I’ve prepared a Study Guide for Midterm 2.
- Each project team should meet with me at least once by next Friday (Reading Recess).
- Please complete your Course Evaluations using the link you received in your Calvin email. This course is being re-architected into 2 8-week courses next year; your feedback on what worked and what didn’t will help us make sure that we don’t inadvertently lose anything that was valuable to you.
- I keep forgetting to mention this in class: several common mistakes in Homework 3 were:
- Not writing it like a report (what are you doing, why, what did you find, explain choices)
- Leaving code that was just pedagogical from the labs, rather than writing the relevant code yourself
- Forgetting to use the validation set!
- Making some choices without explaining them, e.g., whether to have a bias term, or what changes to make in the tuned model
- I’ve written an example of a Homework 3 report with just the structure that I’d suggest for a report like this.
- Revisions are still welcome but must be in by Thursday end of day.
- As noted in the syllabus, I’d like to check in with everyone to discuss where your grade is now and what you need to do to get to the grade you want. I’ll be reaching out by Teams individually, but if you want to stop by my office (e.g., Monday afternoon before 4), that could also be great too.
Final Project Presentations
- Each team will have 10 minutes to present their project, plus 3 minutes for questions.
- Presentations will be in lab on Wednesday 4/26 at 9am.
- Presentations should (I tab-completed this list from Copilot):
- Describe the problem you’re trying to solve
- Describe the data and/or model you used
- Describe the results you got (including a demo, if applicable)
- Describe what you learned
- Describe what you would do next
See you on Monday in lab!
Week 11
Sorry I didn’t get a logistics email out last week. Between traveling, advising, and our midterm, I’ve been swamped. I recognize you all may have been swamped too, so if you need leniency on any due dates, please let me know.
This week:
- The take-home midterm should be returned to my office by Tuesday morning.
- No prep reading this week. If you have bandwidth, review previously posted material about the Transformer architecture.
- We’ll be in lab on Monday, classroom on Wednesday, and lab on Friday
I’m sorry for the continued delay in getting project proposal feedback back to you, and the gradebook is still not accurate. I hope to catch up this week!
Since we’re a bit lighter this week, take the opportunity to catch up on anything you’ve missed – or revise a prior assignment for a better grade. Let me know if you choose to do that.
See you in lab on Monday!
Week 9
This week:
- Prep 9. The assignment itself includes a “study guide” of some questions to help you focus your thinking. The Moodle check-in quiz is just one open-ended reflection, but please turn that in so I can give you credit for the reading. There are two options for the reading; the textbook is a new option and I’m curious what you think of it.
- Lab 8 is due on Friday as usual. If you didn’t get to do a peer feedback on Homework 3 during lab, you’ll need to find someone to do that with this week.
- Discussion for Week 8 – this was going to be homework, but I’ve changed it into a discussion post because it should be more interesting that way. Due at the normal homework time, though (Friday end of day).
Reminders:
- Mid-Semester Survey has gotten 10 responses so far. Everybody loves the labs so far, but other topics are more mixed, so it will be helpful to get your perspectives!
- Project Proposals were due yesterday, but many people haven’t gotten them in yet. Please do so. Since so many initial proposals aren’t in yet, I’m pushing back the deadline for revised proposals.
- Initial ChatGPT responses to exam questions were due yesterday also.
- The course outline has been updated with some material from this past week.
I’ll be out of town on Friday. I might be able to hold a remote session, though, so don’t cancel your plans quite yet.
Week 8
Welcome back from Spring Break! We’re starting the second half of the class(*), switching from the basics of deep learning to one of its most transformational applications: language models. Yup, this is where we learn about ChatGPT and its cousins. We’ll start with a high-level view of how language modeling works, then dig into the strange yet strangely simple Transformer family of neural architectures. We’ll also discuss image generation, human-AI interaction, transparency, and a few other topics in future weeks.
Logistics
- Make an OpenAI account here. For Monday’s class, it’ll be helpful if a good fraction of the class has an account. It requires a phone number verification and maybe a credit card, but they provide a generous trial balance.
- Please fill out the Mid-Semester Survey sometime this week.
- The next step in our alternative exam model is for you to quiz ChatGPT. I’ve put some preliminary instructions on the Exam Instructions page. To debug this, let’s have everyone try doing one question by the end of the week.
- Homework 3 was due last week, but turning in the Monday after Spring Break counts as only one day late.
- Many students are plotting the loss for both the training and validation, which is a really good idea.
- We will be doing peer review with the notebooks, so please try to finish soon!
- Project proposals are due this week. To keep it simple for everyone, especially since we haven’t gotten to see much of NLP, all I’m asking you to do is choose which initiative to join (or, alternatively, propose your own independent project).
- Same schedule as usual (in lab on Friday).
Notes
- I’ve found that some people aren’t aware that I’ve been writing an outline of the course on the website. It’s not complete; suggestions for additions welcome either by asking me or by just directly editing it on GitHub.
- I’ve posted the embedding worksheet we annotated in class (where we worked out the backpropagation logic) on Moodle.
- I’ve caught up some on the prep quizzes on Moodle.
- I grade Moodle quizzes by spot-check, so if the autograder made a mistake, please lmk.
- Another consequence of spot-check grading is that I don’t get to actually responding to the Q&As until it’s past that part of the class. And responses get lost in Moodle anyway. So if there’s something you actually want to discuss, please raise it in class or on Ed. There were lots of interesting issues raised in Chapter 3, for example–bring those up on Ed!
See you Monday!
(*) It’s actually more than halfway through the semester, thanks to days off for Advising and Easter.
Week 7 mid-week
- We had an excellent class meeting today.
- If you didn’t come, I highly recommend you work through the Embeddings Worksheet on your own: docx, PDF
- Many students were still computing gradients like diligent calculus students, not like neural net coders. We’ll review this on Friday.
- As a warm-up, try writing the code that would compute the gradient as a function of
m1, no PyTorch.
- Midterm collective exam questions
- Second question should be done Friday.
- There are lots of questions about loss function selection already. Review what’s already there before you suggest your own.
- Also first review. Either open a PR suggesting a specific fix, or open an Issue or Discussion with a link to what you’re commenting on.
- To link to a specific line in a file, go to that file, click the
<>in the bar above the file contents, click the line number, then select “Reference in new issue” from the...menu. - Use
@to reference the original author, so they can revise based on your feedback.
- To link to a specific line in a file, go to that file, click the
- Many of the learning objectives don’t actually match the questions. In some cases that’s because I didn’t accurately describe the relevant objective on the course website! Suggestions for improvement there count as a Review.
- Homework 3 due Friday also
- Assignment document suggests a strategy.
- Students today reported that starting with u05n3 was helpful.
- Lab 6 should be done by Friday also. Remember that the main point is the check-in quizzes on Moodle.
- Projects: share ideas and inspirations in the Discussion Forum. Goal: find partners.
See you Friday!
Week 7 old (embeddings)
In this unit we introduce one of the most powerful concepts in machine learning: the embedding. It’s the idea that instead of being explicitly given a representation for something, we learn it from data based on the properties it should have. We’ve actually already seen embeddings of various kinds (the inputs to the last layer of an image classifier for one), but we’ll look at two examples where the embedding aspect is even clearer: movies and words.
News:
- In lieu of a normal Discussion this week, we get to start thinking about project ideas! I’ve put up a Forum on Moodle for this week if you’d like to pitch an idea and are looking for partners. We’ll talk more about projects in class this week; here’s the overall description.
Logistics:
- Prep 7 is due on Monday as usual.
- Also due Monday is for you to contribute the first of your Exam Questions; see the Instructions.
- Homework 3 is due on Friday.
- No discussion this week either.
- Lab 6 is due on Friday as usual.
See you Monday in class!
Week 7: Vision and Perspectives
This will be the last week of CS 375! Many, but not all, of you will be continuing with us after Spring Break in CS 376. I hope the course has been valuable to you so far, and that you’re excited to continue learning about machine learning.
We’ll discuss two final topics briefly this week: (1) convolutional neural networks, and (2) some broader contexts of machine learning, historically and in today’s society.
Logistics:
- Preparation 7 is due Monday as usual. (I moved one reading from last week to this week, and just added a short video introducing an interactive demo that you should try out.)
- Homework 3 is finally posted; it took a lot of logistics to get it ready, but hopefully it works. Since I posted it late, I delayed the due date until Saturday.
- Quiz 3 is Friday in the lab. There’s no final exam or project for 375 (but there will be for 376), so this is the last graded assessment of the course.
- Lab 6 is due this evening as usual.
- Lab 7 will also be on Friday. We’ll probably explore something with convolutional networks, but I have a lot of potential material we could do or review, so requests welcome!
- Discussion 3 was due last week; if you haven’t posted and replied, get that in soon.
My wife has been sick with the flu, so I’ve been behind on things in general as I take care of her and our kids. I’m hopeful that I won’t get it myself, but if I do, you might have a guest lecture.
Hopefully I’ll see you Monday!
Week 6: Generalization and Tasks
Happy Saturday, Neural Network coders!
The last few labs were things that you can be proud of: in Lab 3 you trained a simple neural network entirely by hand; in Lab 4 you trained and evaluated a deep neural network in Keras and understood exactly why each part of the network was needed (linear layers, nonlinearities, softmax, cross-entropy loss, etc.), and in Lab 5 you upgraded the input to use images and understood exactly what it meant for a convolutional neural network to be a “feature extractor”.
This week we’ll discuss how to make sure our models can generalize to tasks that they weren’t explicitly trained on. Speaking of tasks, we’ll also spend some time thinking about how to set up neural networks to handle various kinds of tasks.
Logistics:
- Quiz 2 has been graded. Many people did well. I manually reviewed all of the grades and added comments accordingly; please review your feedback in Moodle. (Same for Quiz 1.) Quiz 3 will be the Friday after next, i.e., our last day of class.
- Please review your feedback on lab and homework. Note the revision policy on the Syllabus.
- Preparation 6 is due Monday as usual.
- Discussion 3 will be due on Wednesday. I’m still writing the prompt, though.
Want more resources to help understand what’s going on?
- I added an optional “assignment” to Perusall where I’ve collected a few resources.
- A group called Machine Learning Tokyo has a playlist walking through our textbook.
- If you want all the math details spelled out, the Deep Learning book by Goodfellow, Bengio, and Courville is a great resource. I’ve uploaded two chapters from that to the optional Perusall assignment.
- There’s some videos linked in the Unit 4 Prep that you might have missed.
Week 5: Learning
Last week we studied the very basics of learning by gradient descent. We implemented one of the simplest possible models–linear regression–and saw how to fit it using gradient descent. This week we’ll see how to fit more complex models: we’ll change the loss function so that it can perform classification instead of regression, and we’ll add a nonlinearity to the model so that it can fit more complex functions. We’re well on our way to understanding how to build a neural network!
Logistics:
Notes:
Useful resources for practice:
- The softmax/cross-entropy interactive
- ReLU Regression Interactive (name:
u04n00-relu.ipynb; show preview, open in Colab)
Week 4: Models
Lab 3 was a rite of passage for an ML practitioner: you’ve trained a model by gradient descent entirely by hand, with no libraries beyond NumPy. It’s ok if you don’t quite understand everything you did the first time, but study it again a few times to make sure you understand. If you haven’t yet tried the toy2 dataset (with an outlier) to see the difference that MAE vs MSE makes, I highly encourage you to do that and think about why the results (should be) different.
But that was just linear regression. What about deep neural networks? This week we bridge the gap: linear (“dense”) layers with mulitple inputs and outputs, nonlinear layers, classification, classification losses, … and how a library like Keras lets us stay sane (e.g., not have to do backprop by hand). Hang on for the ride, and keep your favorite thing-explainer (YouTube, ChatGPT, friends, etc.) close at hand.
Logistics:
- Lab 3 is due tonight (but as long as it’s in before we grade on Monday it’s fine). We’ll review the main concepts on Monday.
- Prep 4 is due Monday.
- Some people have posted no or very few comments. This is an important part of our community learning, and the comments that are posted are often interesting. If you’re unsure what to do, see the Perusall help, including some concrete suggestions for how to write comments
- I’ve added (very simple) reading quizzes to Prep 2 and 3, which will give you an opportunity to boost the Perusall grade for those if you’re not happy with it.
- Perusall is still configured to ignore due dates, but I plan to turn that setting back on at the end of this week.
- This is a Discussion week; Discussion 2 is posted and is due Wednesday evening as usual. There are several possible prompts all on the topic of AI fairness.
- I failed to get Quiz 1 ready for yesterday. We’ll do it this Friday in lab instead, then another quiz next Friday also (to get back to an every-other-week rhythm).
- I have several practice quizzes that I hope to unhide soon so you can know what to expect.
- I’m giving the math/stat department seminar on Thursday 3:05pm, on the math and computing ideas behind AI, especially generative AI. Attendance isn’t required, but I’ll take a broader view on the math and computing involved, which many of you might find helpful.
See you on Monday!
Week 2: Supervised Learning
Foremost: can anyone in the 9:15am section switch to the 11am section? The earlier section is overfull, the later one has lots of space.
This week we dive into supervised machine learning (ML): how it fits into the landscape of AI, how to set up an ML task, some simple (tree-based) methods you can use for it, and some common pitfalls. On the coding side, we’ll learn about the industry-standard scikit-learn API for machine learning and the NumPy API for computing with arrays.
In class last week we mentioned the Dan Tepfer talk in the January Series, “Algorithms as the Shapers of Music”, which is still available to watch but only through February 14.
To wrap up week 1, make sure you’ve done all of the following (all on Moodle)
- Preparation 1, including the welcome survey
- Discussion 1 (actually not due till this coming Wednesday)
- optionally, do the Python warm-up quiz
Even though you might discover that you can find course materials in a few different places, always start in Moodle.
Once we get into the rhythm of the semester, Preparation activities will be due on Mondays. So Preparation 2 would be due on Monday the 22nd. But due to delays in me getting things posted and some students not even officially being in the class yet, I won’t count anything as late until Monday the 29th.
- On Monday we’ll meet in the classroom to review Lab 1 and start to discuss supervised learning.
- On Wednesday we’ll be in the classroom again and dive deeper into trees and numerical computing.
- On Friday we’ll be in the lab again.
Locations will be the same as Week 1: Monday and Wednesday are in the classroom (SB 382), Friday in the Gold lab (SB 354).
I’m here for you. We’re all here for each other. May our Lord give us all strength to persevere, joy in the journey, and love for each other along the way. I’m excited to have you all in class, and I hope you’re having a restful weekend.