{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "382e0018", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "# Initialize Otter\n", "import otter\n", "grader = otter.Notebook(\"practice05.ipynb\")" ] }, { "cell_type": "markdown", "id": "p05-001", "metadata": {}, "source": [ "**Student names and e-mails:**\n", "\n", "_YOUR NAME — your@calvin.edu_\n", "\n", "_YOUR NAME — your@calvin.edu_" ] }, { "cell_type": "markdown", "id": "p05-002", "metadata": {}, "source": [ "# Practice 05 — Catching Spam with kNN\n", "\n", "5,160 real text messages, each labeled **spam** (junk) or **ham** (a normal, wanted message — anything that is *not* spam). Can k-nearest neighbors learn to tell them apart — and when it's wrong, *which* way is it wrong?\n", "\n", "| SLO | Description |\n", "|-----|-------------|\n", "| **06A** | Train and interpret k-NN classification models and explain how the choice of k affects overfitting and underfitting |\n", "| **06B** | Compute and interpret classification metrics — accuracy, precision, recall, and the confusion matrix |\n", "| **06C** | Split data into training and test sets to evaluate how well a model generalizes to unseen data |" ] }, { "cell_type": "markdown", "id": "p05-003", "metadata": {}, "source": [ "---\n", "## The Dataset: SMS Spam\n", "\n", "\"A\n", "\n", "**Why \"spam\"?** In a 1970 *Monty Python* sketch, a café serves Spam (the canned meat) with everything, and a table of Vikings keeps singing \"Spam, Spam, Spam…\" until nobody can hear anything else. Early internet users borrowed the word for junk messages that drown out the ones you want. ([More here](https://en.wikipedia.org/wiki/Spam_(Monty_Python)) — culture matters.)\n", "\n", "The messages come from the [SMS Spam Collection](https://archive.ics.uci.edu/dataset/228/sms+spam+collection) (Almeida & Gómez Hidalgo, 2011; CC BY 4.0), gathered in the UK and Singapore around 2010 — expect lots of \"txt\", \"u\" and \"£\". Repeated messages were removed, so each message appears once.\n", "\n", "| Column | What it holds |\n", "|---|---|\n", "| `label` | **spam** (junk) or **ham** (not spam: a normal message from a friend, family, work…) — the answer the model learns from |\n", "| `text` | the message itself |" ] }, { "cell_type": "markdown", "id": "p05-004", "metadata": {}, "source": [ "### Setup (given — just run it)" ] }, { "cell_type": "code", "execution_count": null, "id": "p05-005", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import plotly.express as px\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix\n", "\n", "pd.set_option(\"display.max_colwidth\", 200) # show whole messages\n", "\n", "sms = pd.read_csv(\"https://cs.calvin.edu/courses/data/202/26fa/datasets/spam.csv\")\n", "y = sms[\"label\"] == \"spam\" # the label: True = spam, False = ham\n", "print(len(sms), \"messages |\", y.sum(), \"spam\")\n", "sms.head()" ] }, { "cell_type": "markdown", "id": "p05-006", "metadata": {}, "source": [ "---\n", "## From Messages to Numbers\n", "\n", "kNN measures distances between rows of numbers. A text message is not a row of numbers — yet. One way to make it one: give every word its own column and count how often it appears. This is called a **bag of words** (word order is thrown away):\n", "\n", "| message | free | win | meeting | lunch | you |\n", "|---|---|---|---|---|---|\n", "| \"Win a free trip!\" | 1 | 1 | 0 | 0 | 0 |\n", "| \"Free lunch meeting tomorrow\" | 1 | 0 | 1 | 1 | 0 |\n", "| \"Are you coming to the meeting?\" | 0 | 0 | 1 | 0 | 1 |\n", "\n", "The problem: a word like \"you\" shows up in almost every message, so it says little about *this* one. **TF-IDF** reweights the counts:\n", "\n", "* **TF** (term frequency) — how often the word appears in *this* message\n", "* **IDF** (inverse document frequency) — how *rare* the word is across all the messages\n", "\n", "A word that is frequent here but rare elsewhere (\"win\", \"prize\") gets a high weight; a word that is everywhere (\"you\") gets a low one:\n", "\n", "| message | free | win | meeting | lunch | you |\n", "|---|---|---|---|---|---|\n", "| \"Win a free trip!\" | 0.6 | 0.7 | 0.0 | 0.0 | 0.0 |\n", "| \"Free lunch meeting tomorrow\" | 0.4 | 0.0 | 0.3 | 0.3 | 0.0 |\n", "| \"Are you coming to the meeting?\" | 0.0 | 0.0 | 0.2 | 0.0 | 0.4 |\n", "\n", "scikit-learn's `TfidfVectorizer` does this for us. But first — the split." ] }, { "cell_type": "markdown", "id": "p05-007", "metadata": {}, "source": [ "---\n", "## Part 1 — Hold Some Messages Out (SLO 06C)" ] }, { "cell_type": "markdown", "id": "p05-009", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 06C.1 — Train/Test Split *(2 pts)*\n", "\n", "Split the **messages** `sms[\"text\"]` and the labels `y` into a training set and a test set, with `train_test_split`: **20%** for testing and `random_state=42`. Store the four pieces in `text_train, text_test, y_train, y_test` (same order as in class).\n", "\n", "Also pass **`stratify=y`**. Only about 1 message in 8 is spam; `stratify` makes sure the training and test sets both keep that same share, instead of leaving it to luck." ] }, { "cell_type": "code", "execution_count": null, "id": "p05-011", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "text_train, text_test, y_train, y_test = ...\n", "print(len(text_train), \"training messages |\", len(text_test), \"test messages\")\n", "print(\"share of spam:\", round(y_train.mean(), 3), \"(training) |\", round(y_test.mean(), 3), \"(test)\")" ] }, { "cell_type": "code", "execution_count": null, "id": "89cb5da4", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"06C.1\")" ] }, { "cell_type": "markdown", "id": "p05-018", "metadata": {}, "source": [ "### Turning Text into TF-IDF (given — just run it)\n", "\n", "The vectorizer builds its word list — and learns how rare each word is — with `fit`. Wednesday's rule applies here too: **test messages never go into `fit`**. So:\n", "\n", "* `fit_transform` on the **training** messages: learn the words, then turn those messages into numbers\n", "* `transform` only on the **test** messages: use the words learned from training — the way a real filter meets tomorrow's messages" ] }, { "cell_type": "code", "execution_count": null, "id": "p05-019", "metadata": {}, "outputs": [], "source": [ "vectorizer = TfidfVectorizer(\n", " stop_words=\"english\", # drop very common English words (\"the\", \"and\", ...)\n", " min_df=2, # keep only words that appear in at least 2 messages\n", " token_pattern=r\"(?u)\\b[a-zA-Z]{2,}\\b\", # a \"word\" = 2 or more letters, so no numbers\n", ")\n", "X_train = vectorizer.fit_transform(text_train) # fit: learns the words from the TRAINING messages only\n", "X_test = vectorizer.transform(text_test) # the same columns, for the test messages\n", "\n", "words = vectorizer.get_feature_names_out()\n", "print(X_train.shape, X_test.shape, \"->\", len(words), \"word columns\")" ] }, { "cell_type": "markdown", "id": "p05-020", "metadata": {}, "source": [ "Each message is now a row with one number per word — almost all of them 0. Here is one training message, with the words it contains and their weights:" ] }, { "cell_type": "code", "execution_count": null, "id": "p05-021", "metadata": {}, "outputs": [], "source": [ "row = 2\n", "print(text_train.iloc[row], \"\\n\")\n", "weights = pd.Series(X_train[row].toarray()[0], index=words)\n", "weights[weights > 0].sort_values(ascending=False).round(2)" ] }, { "cell_type": "markdown", "id": "p05-022", "metadata": {}, "source": [ "*Could we do better?* A **lemmatizer** would reduce each word to its base form (\"offers\" → \"offer\", \"winning\" → \"win\"), so the variants share one column. We'll stop here — look it up if you're curious." ] }, { "cell_type": "markdown", "id": "p05-024", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 06C.2 *(1 pt)*.** The vectorizer's word list came from the training messages only. What happens to a word that appears in a test message but never in the training messages? Why is that the honest way to test a spam filter?" ] }, { "cell_type": "markdown", "id": "7289cec7", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p05-029", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## Part 2 — A kNN Spam Filter (SLO 06A)" ] }, { "cell_type": "markdown", "id": "p05-031", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 06A.1 — Fit and Predict *(2 pts)*\n", "\n", "Train a `KNeighborsClassifier` with **k = 5** on the training set, store it in `knn`, and store its predictions for the **test** messages in `y_pred`.\n", "\n", "Two extra settings, for text: `metric=\"cosine\"` and `algorithm=\"brute\"`. Cosine distance compares *which words* two messages use, not how long they are — and it is fast on rows that are mostly zeros." ] }, { "cell_type": "code", "execution_count": null, "id": "p05-033", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "knn = ...\n", "y_pred = ...\n", "print(\"messages called spam:\", y_pred.sum(), \"of\", len(y_pred))" ] }, { "cell_type": "code", "execution_count": null, "id": "43e12343", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"06A.1\")" ] }, { "cell_type": "markdown", "id": "p05-040", "metadata": {}, "source": [ "---\n", "## Part 3 — How Good Is the Filter? (SLO 06B)\n", "\n", "> 🎯 **Predict First:** A lazy \"filter\" that lets every message through — it never says spam. What is its accuracy on the test set?" ] }, { "cell_type": "code", "execution_count": null, "id": "p05-041", "metadata": {}, "outputs": [], "source": [ "lazy = np.zeros(len(y_test), dtype=bool) # never says spam\n", "print(\"lazy filter accuracy:\", round(accuracy_score(y_test, lazy), 3))" ] }, { "cell_type": "markdown", "id": "p05-042", "metadata": {}, "source": [ "Almost 88% — and it catches **no** spam at all. Same as Wednesday's Chinstraps: when a group is rare, accuracy alone hides the model's mistakes. Here, **spam** is the rare group we want to find (the \"positive\" class)." ] }, { "cell_type": "markdown", "id": "p05-044", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 06B.1 — Confusion Matrix and Metrics *(3 pts)*\n", "\n", "Reminder: **ham** = a normal message, i.e. *not* spam. In `y`, spam is `True` (the positive class we want to catch) and ham is `False`.\n", "\n", "For your k = 5 filter on the test set:\n", "\n", "1. **The confusion matrix is given** — the first lines of the cell build it for you. `confusion_matrix(y_test, y_pred, labels=[False, True])` counts the test messages in a 2×2 array: **rows = true label, columns = predicted label**, both in the order `[False, True]`, i.e. ham first, then spam. Wrapping that array in a `pd.DataFrame` with `index` and `columns` set to `[\"ham\", \"spam\"]` just puts readable names on the rows and columns, so you can look up a cell by name — e.g. `cm.loc[\"ham\", \"spam\"]` = messages that were really ham but predicted spam:\n", "\n", " | | predicted **ham** | predicted **spam** |\n", " |---|---|---|\n", " | **true ham** | true negative — normal message kept | **false positive** — normal message sent to the spam folder |\n", " | **true spam** | **false negative** — spam let through | true positive — spam caught |\n", "\n", "2. Store the test `accuracy`, `precision` and `recall` (for spam) — each a number between 0 and 1.\n", "\n", "The plot at the end is given." ] }, { "cell_type": "code", "execution_count": null, "id": "p05-046", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "# Given: rows = true label, columns = predicted label; order [False, True] = [ham, spam]\n", "names = [\"ham\", \"spam\"]\n", "cm = pd.DataFrame(confusion_matrix(y_test, y_pred, labels=[False, True]), index=names, columns=names)\n", "display(cm)\n", "\n", "accuracy = ...\n", "precision = ...\n", "recall = ...\n", "print(f\"accuracy: {accuracy:.3f} | precision: {precision:.3f} | recall: {recall:.3f}\")\n", "\n", "px.imshow(cm, text_auto=True, color_continuous_scale=\"Blues\",\n", " labels=dict(x=\"predicted\", y=\"true\", color=\"messages\"),\n", " title=\"Confusion Matrix: 1,032 Test Messages, k = 5\")" ] }, { "cell_type": "code", "execution_count": null, "id": "049e70b6", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"06B.1\")" ] }, { "cell_type": "markdown", "id": "p05-053", "metadata": {}, "source": [ "### Look at the Mistakes (given — just run it)\n", "\n", "The **false positives** (ham the filter called spam) and a few **false negatives** (spam it let through):" ] }, { "cell_type": "code", "execution_count": null, "id": "p05-054", "metadata": {}, "outputs": [], "source": [ "results = pd.DataFrame({\"text\": text_test, \"true\": y_test.map({True: \"spam\", False: \"ham\"}),\n", " \"predicted\": np.where(y_pred, \"spam\", \"ham\")})\n", "print(\"FALSE POSITIVES — ham called spam:\")\n", "display(results[(results[\"true\"] == \"ham\") & (results[\"predicted\"] == \"spam\")])\n", "print(\"FALSE NEGATIVES — spam let through (first 8):\")\n", "display(results[(results[\"true\"] == \"spam\") & (results[\"predicted\"] == \"ham\")].head(8))" ] }, { "cell_type": "markdown", "id": "p05-056", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 06B.2 *(1 pt)*.** For a spam filter, which mistake costs more: a false positive or a false negative? So which metric would you watch most closely — precision or recall? How does your k = 5 filter do on it?\n", "\n", "*(Remember: false positives lower **precision**; false negatives lower **recall**.)*" ] }, { "cell_type": "markdown", "id": "61fe0660", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p05-061", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## Part 4 — Choosing k: Overfitting and Underfitting (SLO 06A)\n", "\n", "k = 5 was a guess. Monday's penguin plot compared accuracy on the data the model learned from with accuracy on new data. Let's make the same plot for the spam filter — with recall and precision, since accuracy hides the rare spam." ] }, { "cell_type": "markdown", "id": "p05-063", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 06A.2 — Training vs. Test, for Many k *(3 pts)*\n", "\n", "For every k in `ks`, fit a kNN filter (same `metric=\"cosine\", algorithm=\"brute\"`) on the training set. Then compute its **recall** and **precision** on the **training** set and on the **test** set. The loop collects them into `scores`; the plots are given." ] }, { "cell_type": "code", "execution_count": null, "id": "p05-065", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "ks = list(range(1, 102, 4)) # k = 1, 5, 9, ..., 101\n", "rows = []\n", "for k in ks:\n", " # fit a kNN filter with k neighbors on the training set\n", " model = ...\n", " # predict the training messages and the test messages\n", " pred_train = ...\n", " pred_test = ...\n", " # recall and precision, on each set\n", " recall_train = ...\n", " recall_test = ...\n", " precision_train = ...\n", " precision_test = ...\n", " rows.append({\"k\": k, \"recall_train\": recall_train, \"recall_test\": recall_test,\n", " \"precision_train\": precision_train, \"precision_test\": precision_test})\n", "scores = pd.DataFrame(rows)\n", "\n", "px.line(scores, x=\"k\", y=[\"recall_train\", \"recall_test\"], markers=True,\n", " labels={\"value\": \"recall\", \"variable\": \"scored on\"}, title=\"Recall vs. k\").show()\n", "px.line(scores, x=\"k\", y=[\"precision_train\", \"precision_test\"], markers=True,\n", " labels={\"value\": \"precision\", \"variable\": \"scored on\"}, title=\"Precision vs. k\").show()\n", "scores.round(3).head()" ] }, { "cell_type": "code", "execution_count": null, "id": "413336da", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"06A.2\")" ] }, { "cell_type": "markdown", "id": "p05-073", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 06A.3 *(1 pt)*.** Where do you see **overfitting** in the plots, and where **underfitting**? Why does recall drop as k grows? (Think about how the vote goes when only 1 in 8 messages is spam.) Which k would you pick for your filter — and how much would you trust that choice?" ] }, { "cell_type": "markdown", "id": "861c442c", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p05-078", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## Bonus: Which Words Scream \"Spam\"? (given — just run it)\n", "\n", "The average TF-IDF weight of each word, in spam and in ham training messages. These are the words kNN's distances lean on — but always together with the rest of the message (\"Are you **free** Saturday?\" is not spam)." ] }, { "cell_type": "code", "execution_count": null, "id": "p05-079", "metadata": {}, "outputs": [], "source": [ "spam_words = pd.Series(np.asarray(X_train[y_train.values].mean(axis=0))[0], index=words)\n", "ham_words = pd.Series(np.asarray(X_train[~y_train.values].mean(axis=0))[0], index=words)\n", "pd.DataFrame({\"top spam words\": spam_words.sort_values(ascending=False).head(15).index,\n", " \"top ham words\": ham_words.sort_values(ascending=False).head(15).index})" ] }, { "cell_type": "markdown", "id": "p05-080", "metadata": {}, "source": [ "---\n", "## Submission\n", "\n", "Save your notebook, then upload the **`.ipynb` file** directly to our Moodle assignment page. Don't export, zip, or submit anything else — just the notebook." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" }, "otter": { "OK_FORMAT": true, "assignment_name": "practice05", "tests": { "06A.1": { "name": "06A.1", "points": 2, "suites": [ { "cases": [ { "code": ">>> assert knn.n_neighbors == 5, 'use k = 5'\n>>> assert len(y_pred) == 1032, 'predict the test messages'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "06A.2": { "name": "06A.2", "points": 3, "suites": [ { "cases": [ { "code": ">>> assert len(scores) == 26, 'one row per k in ks'\n>>> assert scores.drop(columns='k').apply(lambda c: c.between(0, 1).all()).all(), 'scores lie between 0 and 1'\n>>> assert scores.loc[0, 'recall_train'] > 0.99, \"k = 1 on the training set: each message's nearest neighbor is itself\"\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "06B.1": { "name": "06B.1", "points": 3, "suites": [ { "cases": [ { "code": ">>> assert list(cm.index) == ['ham', 'spam'] and list(cm.columns) == ['ham', 'spam']\n>>> assert cm.values.sum() == 1032, 'every test message should be counted once'\n>>> assert all((0 <= m <= 1 for m in [accuracy, precision, recall])), 'metrics lie between 0 and 1'\n>>> assert abs(accuracy - np.trace(cm.values) / 1032) < 1e-09, 'accuracy = the diagonal / all messages'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "06C.1": { "name": "06C.1", "points": 2, "suites": [ { "cases": [ { "code": ">>> assert len(text_train) == 4128 and len(text_test) == 1032, 'use test_size=0.2'\n>>> assert len(y_train) == len(text_train) and len(y_test) == len(text_test), 'split the messages and y together'\n>>> assert abs(y_train.mean() - y_test.mean()) < 0.002, 'use stratify=y so both sets keep the same share of spam'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] } } } }, "nbformat": 4, "nbformat_minor": 5 }