The goal of this project is to practice a complete predictive
analytics workflow, from honing a good question to communicating your
results.
Results will be presented as a report, supporting code, and a brief
presentation to the class. The report should be at the level of polish
and formality of a blog post (more than a class homework assignment,
less than an academic paper). The overview and primary visualizations
should be intelligible to a non-technical audience; the methods should
be described in precise technical language as appropriate.
An important component of this project is a critique of some
related prior work. That is, you are not simply demonstrating that you
can perform an analysis, but also that you can evaluate the strengths
and weaknesses of others’ analyses. This may be the most important
take-away from this class for your future career.
Logistics
Teams: Both individual and team projects are
permitted, with a mild encouragement towards teams.
Presentations will be done over VoiceThread.
Grading
The project will be graded as follows:
- Milestones (20 points)
- Exploratory proposal (2 points)
- Initial proposal (2 points)
- Revised proposal (3 points)
- Dataset and exploratory visualization (5 points)
- Initial modeling (5 points)
- Walkthrough meeting with professor (3 points)
- Presentation video or VoiceThread (20 points)
- The presentation is well organized around a central story or main
point. (5 points)
- Background is given about the objective, task, and dataset. (5
points)
- Claims are anchored in data, using both clear text and legible,
relevant plots and tables. (5 points)
- Thoughtful comments are left on at least two other presentations (5
points).
- Report (80 points)
- It clearly states the project goal and motivates it by a real-world
scenario (5 points).
- It describes the structure and content of the dataset and assess how
the data can address the goal (10 points).
- It critically evaluates some prior work, describing what inspiration
was drawn from both that work’s strengths and its weaknesses (see
Critique of Prior Work below) (15 points).
- It draws useful insights from exploratory data analysis, clearly
presented using a selected set of plots and tables (10 points).
- It clearly connects the setup of the modeling task (features,
targets, validation/splitting strategy) with the real-world scenario and
insights from data structure and EDA (10 points).
- It evaluates model(s) using an appropriate overall quantative
measure (accuracy, MAE, etc.) and at least one other characteristic or
approach (such as EDA on errors) (10 points).
- It summarizes results in terms of overall success at the goal,
recommendations for how to use (or not use) the model in a business
context, and summarizes limitations and caveats (10 points).
- The report is clear, concise, and well organized, and has no major
writing issues (5 points).
- All resources that were used (except for provided course materials)
are clearly cited (5 points).
Choosing your own project
Your project must include predictive analytics of the kind discussed
in this class, or other approaches by permission of the instructor.
Although students often think of inference questions (e.g.,
“what is the relationship between X and Y?”) for projects, I enourage
you to think of prediction questions (e.g., “can we predict Y
from X?”) instead, although either is acceptable.
A Kaggle competition is a good place to start looking for project
ideas and data. But:
- Kaggle competitions often don’t document the data well; you may need
to do some extra digging to understand what the data is and how it was
collected.
- Kaggle competitions helpfully show you what other people have done,
but many of those analyses are flashy and overcomplicated. Read through
others’ work for inspiration and ideas, but in your work, keep it simple
and focus on interpreting the results and connecting them to the real
world.
Your data should be rich enough to support the kind of modeling you
want to do, but not too complex to work with. For example, large
datasets (i.e., bigger than about 500 MB) may require careful handling
to avoid exhausting your computing resources.
Deliverables
Report
Overall:
- Be mindful of decisions. Report:
- What decisions you made
- Why you made them
- What might have been alternative choices.
- Tell the “rational
reconstruction” of the story of how the analysis was done.
- Don’t give the play-by-play of everything you
tried, every idea you had, etc., but…
- Do include things you tried that led to an
important observation later on.
- If you use any code from the Internet, you should acknowledge its
source and provide a link.
- You should submit all of the code needed to replicate your results,
but your report should be understandable without looking at the
code.
Elements of a report will vary depending on the specific project, but
should generally include:
- A succinct but descriptive title
- A one-paragraph executive summary of the main
conclusions of your analysis, aimed at a decision-maker. If possible,
make a visual to supplement your text.
- Overview of the vision of the project, the data you
use, the kind of analysis you will perform, and the main results you
obtained.
- Include any background information needed to
understand the situation.
- Critique of Prior Work - see details below.
- Approach you took, including:
- What data you’re using. Give enough detail for the
reader to evaluate how appropriate the data is for your task. (What’s
its origin? size? structure? how it relates with your vision, etc.) The
reader should be able to obtain the data themselves.
- Include some Exploratory Data Analysis (EDA)
here–plots, summary tables, etc.
- Summarize what you observed in your EDA and how that informs your
modeling.
- What analytics question you’re asking (in terms of
the specific characteristics of the data you have), and how that relates
to your overall vision.
- Describe the setup (e.g., what target variable
you’re trying to predict, how you’ll measure accuracy, how you’ll
validate, etc.)
- Alternative: instead of a supervised prediction
task, you can define an unsupervised learning task and use clustering.
In this case, clearly state what you want to understand through the
clustering, and report your observations.
- Describe your analytics process
- What sort of techniques and hyperparameters did you choose?
Why?
- What steps did you perform to get your result?
- What results did you get? Summarize them quantitatively and
qualitatively.
- How did you refine the process?
- Describe your findings:
- The strongest reports will include insightful visualizations of the
model, its predictions, and/or its mistakes, and a discussion of what
those plots tell us.
- Report on the final accuracy of your best model the test set, if
applicable.
- Summarize the analyses you performed and what the results told you.
What do your findings say about the real-world and prediction (or
clustering) questions you posed?
- Limitations: What are some limitations of your
analyses? Did you notice any potential biases in the data you used or
analysis you did? Any other ethical questions raised during this
project?
- Future Directions: What new questions came up
following your exploration of this data? Identify at least one question
that would require new data or a new analysis approach, and specify what
steps would be required.
(Project descriptions originally thanks to Ofra Amir)
Critique of Prior Work
This section of the report should include:
- A clear citation to the article you’re critiquing, including a link
if applicable.
- You can find examples of predictive analytics work on sites like
Kaggle, TowardsDataScience, Reddit, YouTube, and GitHub. Also check out
the TidyTuesday project and r-bloggers.
- An overview of their work (see Overview below)
- Technical Choices:
- What was a technical choice that the author(s) made? (note: people
are not always aware of the choices that they make)
- What would have been some alternative options they could have
chosen?
- What are some pros and cons of the option they chose?
- Trustworthiness
- What did the author do to get you to trust their results more?
(attention to detail? careful validation? clear explanation? etc.)
- What aspect of the work do you trust the least? Why?
- In what ways do you intend for this project to extend or enhance
that prior work? (Save the details of how for the Approach
section of your report.)
You should pick one specific prior work and analyze it in depth. If
you find several articles, you may include links and brief discussions
of them as well.
Presentation
Prepare a short (around 5 minutes or less) of your work. See the Unit
16 slides for some guidelines.
I suggest using presentation software rather than tools like Quarto
slides or Xaringan, but you’re very welcome to try them out if you
like.