Project (CS 376)

This rubric is a rough guide; adjustments may be made based on the specifics of your project. The choosing-a-project page has some details about evaluation of different types of projects.

Since different projects will have different emphases, projects wil lbe graded holistically based on their contribution to your portfolio:

• A = I could write you a recommendation letter based on this work
• B = you should definitely include this in your portfolio
• C = you could include this in your portfolio, but not if you have better work
• D = you should probably not include this in your portfolio

Here are some ways that projects can demonstrate these levels of quality. The strongest projects will demonstrate several of these.

• Connecting to fundamental ML concepts. You can dig below the surface in some way. For example, your report might include a substantive discussion of:

  • why a model (or hyperparameter of some model) was chosen by discussing how the model aligns with the task
  • analyzing what kinds of errors the model makes and why that might or might not make sense in light of the architecture
  • how a choice of loss function, training data, evaluation approach, etc. impacts the outcome
  • how a model type that was not covered in detail in class actually works (ideally with some specific examples)

• Evaluation, both quantitative and qualitative. Don’t just say it works, measure it. For example, you might:

  • compare the performance of two or three different approaches to a task (e.g., simple baseline method you implemented, state-of-the-art method that someone else implemented, and simple tweak to the baseline method that improves it a little)
  • quantitative (numbers that measure performance) and qualitative (specific examples of what the model does well or poorly)
  • robustness analyses

• Real-World Connections:

  • Why is this project interesting? What is the real-world problem that it addresses?
  • What choices are you making because of the real-world problem?
  • Connect your results about system performance to the real-world problem.
  • Are there any decisions that you might make differently because of ethical considerations? Be specific!

• Communication: Decisions and Rationale

  Explains what decisions were made during the project why they were made, and possible alternatives.

  This is usually done throughout the report, i.e., “We chose to use model [name of model] because [characteristic of the task]; reasonable alternatives might have been [other model name] or, if we thought of the problem as a zero-shot classification task, we could even have used [very different kind of model]”

  The strongest reports will discuss the results in terms of how they relate to these choices, e.g., what choices probably mattered a lot to the results; what other choices might have worked out better (thus future work).

  You might have several sections of code or experimentation. A general structure for each one might be:

  • Say what you’re trying to do
  • do it (describing important decisions and milestones along the way),
  • then discuss what you did and what you observe from it.

  On a high level, you might structure your report like:

  • we want to do X
  • so we adjusted Y
  • and we observed difference Z
  • which tells us ______ about the relationship between X and Y
  • so, were we successful at X? (and why or why not?)

• Clear communication of results:

  • Good visualizations, tables, textual summaries, examples, etc.
  • Connect the results to the real-world problem.

• Clear discussion of limitations and future directions

  • Not just “I ran out of time to do X”, but “these results assume that Y is true, but in the real world, some cases where Y might not be true are [examples].”
  • What are the implications of this? What would you do next if you had more time?
  • Specific limitations of the chosen approach are discussed (e.g., “our dataset only had examples of X, so we couldn’t test how well the model generalizes to Y”)
  • Ethical considerations are specific, e.g., rather than generic concerns about bias, the report gives specific examples of biases that might be present and what the consequences might be.
  • Future directions are plausible and described in enough detail that someone else could pick up the project and run with it.

Minimum expectations:

• Report:
  • The report is well organized (has a clear logical flow, uses headings to indicate sections, etc.)
  • It has no major writing issues.
  • The report is concise (any unnecessary information or outputs are moved to an appendix or removed)
  • The report is understandable without reading the code.
  • All resources that were used (except for provided course materials) are clearly cited.

An excellent project could become a submission to an academic conference or a blog post. At the very least it should be a good portfolio piece.

• What problem are you trying to solve?
• How you approached that problem:
  • How did you frame the problem in a way that you could apply a model to answer?
  • what model did you use, what did you train/fine-tune on, etc.?
  • How did you turn model’s outputs into something useful?
• What results did you get?
  • Include both specific examples and summary numbers, if applicable.
  • This would be a good place to give a demo – but maybe record a video in case it doesn’t work.
• What did you learn? you could take this in various ways:
  • about your problem?
  • about the model or data you’re using?
  • about AI/ML more generally?
  • about your problem-solving process?
  • etc.
• What should others take away?
  • If someone in the audience gets asked “what was that presentation about?”, how would you want them to answer?
  • What are some limitations of your project?
  • What broader questions might your project raise? How might you contribute to the discussion?
  • What might you do next?

Here is an example presentation that I threw together. Self-critique:

• The Goal slide is ok, but it’s not clear what the example is showing.
• Approach slide diagram could be clearer, with more intentional use of color.
• Details slide may be too dense; the code screenshot might not be helpful.
• Evaluation is quick and dirty and not finished. (But it’s a good idea to show some results, even if they’re not great.)

Reports should be organized in a way that makes sense for your project. The following is a general outline of what a report might look like. You can use this as a starting point, but feel free to adjust it to fit your project.

• A succinct but descriptive title
• A real-world question or goal and why it’s interesting.
• A description of the dataset: what sort of data does it contain? Where did it come from? Why did you choose it? What are its strengths and limitations?
• A specific technical goal or question
• Your technical approach for achieving that goal or answering that question
• What you noticed from exploring the data (e.g., counts by category, distributions of continuous variables, things you notice from inspecting individual samples at random)
• Your modeling setup: what are your features? Targets? Metrics? Loss function?
• Your validation approach: train-val-test split? cross-validation?
• Your baseline results: applying the simplest model you can think of; how good were the results (quantitatively and perhaps qualitatively)?
• Your attempts at improved results: what did you adjust, and why? How did the results change?
• An analysis of errors (quantitatively and perhaps qualitatively)
• An analysis of the effects of alternative choices. You can consider differences in model architecture, specific task, hyperparameter choices, inclusion/exclusion criteria, etc. Remember to think about the choice of metrics and the uncertainty involved in any estimate of them.
• A summary of your findings. Did you achieve your goal or answer your question?
• Limitations and future directions

Artistic or exploratory projects may need other elements.

Checklist:

• Describes why you made various decisions
• Backs up claims with evidence (e.g., numbers, examples)
• Cites sources for any ideas that are not your own (and describes what you took from each source)