{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "1d8560c7", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "# Initialize Otter\n", "import otter\n", "grader = otter.Notebook(\"practice06.ipynb\")" ] }, { "cell_type": "markdown", "id": "p6-001", "metadata": {}, "source": [ "**Student names and e-mails:**\n", "\n", "_YOUR NAME — your@calvin.edu_\n", "\n", "_YOUR NAME — your@calvin.edu_" ] }, { "cell_type": "markdown", "id": "p6-002", "metadata": {}, "source": [ "# Practice 06 — Who Lives Here? Decision Trees, Cross-Validation and Grid Search\n", "\n", "Field researchers record what kind of organism they find at a site, along with the site's elevation, temperature and rainfall. Can a few yes/no questions about a site tell us what lives there? How many questions is too many? And how do we choose without peeking at the test set?\n", "\n", "| SLO | Description |\n", "|-----|-------------|\n", "| **07A** | Train and interpret decision tree models and explain how tree depth affects overfitting and underfitting |\n", "| **07B** | Use cross-validation to evaluate model performance across multiple data splits |\n", "| **07C** | Tune model hyperparameters using grid search |" ] }, { "cell_type": "markdown", "id": "p6-003", "metadata": {}, "source": [ "---\n", "## The Dataset: Organisms and Sites\n", "\n", "\"A\n", "\n", "The data is **simulated**: 1,000 observations, each one organism found at one site. Being simulated, its patterns were put there on purpose, and at the end of the practice you'll see exactly how. That makes it a good place to practice, but nothing here is a claim about real ecology.\n", "\n", "Like real field data, it is messy: some sightings are out of place (a bird that wandered far from home, a misidentified insect), so no model can get every site right.\n", "\n", "| Column | What it holds |\n", "|---|---|\n", "| `elevation_m` | the site's elevation, in meters |\n", "| `temperature_c` | the site's average temperature, in °C |\n", "| `precipitation_mm` | the site's yearly rainfall, in mm |\n", "| `soil` | soil type: Sandy, Clay, Loamy, Peaty or Chalky |\n", "| `habitat` | habitat type: Forest, Grassland, Wetland or Desert |\n", "| `organism` | **the label**: Plant, Insect, Bird, Mammal or Reptile |" ] }, { "cell_type": "markdown", "id": "p6-004", "metadata": {}, "source": [ "### Setup (given — just run it)" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-005", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import plotly.express as px\n", "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n", "from sklearn.tree import DecisionTreeClassifier, plot_tree\n", "from sklearn.metrics import accuracy_score\n", "\n", "sites = pd.read_csv(\"https://cs.calvin.edu/courses/data/202/26fa/datasets/species.csv\")\n", "print(len(sites), \"observations\")\n", "print(sites[\"organism\"].value_counts().to_string())\n", "sites.head()" ] }, { "cell_type": "markdown", "id": "p6-006", "metadata": {}, "source": [ "**Two columns we leave out.** `soil` and `habitat` are words, not numbers, and scikit-learn's trees only accept numbers. Turning categories into numbers is a topic for later in the course. Here we lose little by dropping them. `habitat` is just rainfall in four bands, which `precipitation_mm` already tells us in more detail:" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-007", "metadata": {}, "outputs": [], "source": [ "sites.groupby(\"habitat\")[\"precipitation_mm\"].agg([\"min\", \"max\"])" ] }, { "cell_type": "markdown", "id": "p6-008", "metadata": {}, "source": [ "And in this data, `soil` has nothing to do with which organism shows up. So the features are the three numeric columns, and the label is `organism`. The split is given: 20% for testing, with `stratify=y` so both sets keep the same share of each organism. **The test set stays locked** until a step says to use it." ] }, { "cell_type": "code", "execution_count": null, "id": "p6-009", "metadata": {}, "outputs": [], "source": [ "features = [\"elevation_m\", \"temperature_c\", \"precipitation_mm\"]\n", "X = sites[features]\n", "y = sites[\"organism\"]\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)\n", "print(len(X_train), \"training sites |\", len(X_test), \"test sites\")" ] }, { "cell_type": "markdown", "id": "p6-010", "metadata": {}, "source": [ "---\n", "## Part 1 — A Tree With Two Questions (SLO 07A)\n", "\n", "> 🎯 **Predict First:** A tree with `max_depth=2` asks at most 2 questions per site. How many final boxes (leaves) can it have? There are 5 organisms. Can it predict all of them?" ] }, { "cell_type": "markdown", "id": "p6-012", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 07A.1 — Fit a Depth-2 Tree *(2 pts)*\n", "\n", "Fit a `DecisionTreeClassifier` with `max_depth=2` and `random_state=0` on the **training** set, and store it in `tree2`. Then store its accuracy on the training set in `train_acc` and on the test set in `test_acc`.\n", "\n", "The plot of the tree is given." ] }, { "cell_type": "code", "execution_count": null, "id": "p6-014", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "tree2 = ...\n", "train_acc = ...\n", "test_acc = ...\n", "print(f\"training accuracy: {train_acc:.3f} | test accuracy: {test_acc:.3f}\")\n", "\n", "plt.figure(figsize=(14, 6))\n", "plot_tree(tree2, feature_names=features, class_names=tree2.classes_, filled=True, fontsize=9)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "eede0ae0", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"07A.1\")" ] }, { "cell_type": "markdown", "id": "p6-021", "metadata": {}, "source": [ "### Does the Tree Make Sense? (given — just run it)\n", "\n", "The tree first asks about `precipitation_mm`, then about `temperature_c` on the drier side. Here are the training sites on those two features, with those two cuts drawn in:" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-022", "metadata": {}, "outputs": [], "source": [ "fig = px.scatter(X_train.assign(organism=y_train), x=\"precipitation_mm\", y=\"temperature_c\", color=\"organism\",\n", " opacity=0.7, title=\"800 Training Sites: Rainfall vs. Temperature\")\n", "fig.add_vline(x=1774.65, line_dash=\"dash\", line_color=\"black\") # the root question\n", "fig.add_shape(type=\"line\", x0=0, x1=1774.65, y0=19.65, y1=19.65, # the second question, drier side only\n", " line=dict(dash=\"dot\", color=\"black\"))\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "p6-024", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 07A.2 *(1 pt)*.** Read the tree and the plot.\n", "\n", "1. In words, what does the tree predict for a **drier** site (precipitation ≤ 1774.65 mm) that is **warm** (temperature > 19.65 °C)? How sure is it — look at `value` in that leaf?\n", "2. Which organism does the tree **never** predict? Look at where those organisms sit in the `value` lists of the four leaves, and where they sit in the plot. Why can't a depth-2 tree predict it?" ] }, { "cell_type": "markdown", "id": "bcbcac58", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p6-029", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## Part 2 — How Many Questions? Depth and Overfitting (SLO 07A)\n", "\n", "Depth 2 gets 63% of the test sites right. That's far better than guessing one of 5 (about 20%), but it can't even predict reptiles. More questions should help. But how many?" ] }, { "cell_type": "markdown", "id": "p6-031", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 07A.3 — Training vs. Test, for Every Depth *(3 pts)*\n", "\n", "For every depth in `depths` (1 to 22; at depth 22 every leaf is pure, so the tree can't grow further), fit a tree with that `max_depth` and `random_state=0` on the training set. Store its accuracy on the training set and on the test set, and also the number of leaves (`model.get_n_leaves()`). The loop collects them into `scores`; the plot is given.\n", "\n", "*(We look at the test set here only to see the overfitting shape, like the decision-trees class before Fall Break. In Part 3 we'll choose the depth the fair way.)*" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-033", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "depths = list(range(1, 23))\n", "rows = []\n", "for depth in depths:\n", " # fit a tree with this max_depth on the training set\n", " model = ...\n", " # accuracy on the training set and on the test set\n", " acc_train = ...\n", " acc_test = ...\n", " rows.append({\"max_depth\": depth, \"leaves\": model.get_n_leaves(), \"train\": acc_train, \"test\": acc_test})\n", "scores = pd.DataFrame(rows)\n", "\n", "px.line(scores, x=\"max_depth\", y=[\"train\", \"test\"], markers=True,\n", " labels={\"value\": \"accuracy\", \"variable\": \"scored on\"},\n", " title=\"Decision Tree Accuracy vs. Depth\").show()\n", "scores.round(3)" ] }, { "cell_type": "code", "execution_count": null, "id": "52e12c1a", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"07A.3\")" ] }, { "cell_type": "markdown", "id": "p6-041", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 07A.4 *(1 pt)*.** Use the plot and the table.\n", "\n", "1. Where is the tree **underfitting**? Where is it **overfitting**? What in the plot tells you?\n", "2. The deepest tree has 228 leaves for 800 training sites and scores 100% on them. What are all those leaves learning, and what happens to them on the test sites? *(Remember: some sightings are out of place.)*" ] }, { "cell_type": "markdown", "id": "440f0c83", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p6-046", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## Part 3 — Choosing Fairly: Cross-Validation (SLO 07B)\n", "\n", "In Part 2 the test set did the judging. If we pick the depth that way, the test set is no longer \"new data\". The fix from the last class before Fall Break: keep the test set locked, and judge inside the **training** set with **5-fold cross-validation**:\n", "\n", "1. Cut the 800 training sites into 5 folds of 160\n", "2. Fit on 4 folds, score on the 5th. Repeat 5 times so every fold takes a turn\n", "3. The mean of the 5 scores is the cross-validation score" ] }, { "cell_type": "markdown", "id": "p6-048", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 07B.1 — Cross-Validating the Full Tree *(2 pts)*\n", "\n", "The tree with **no depth limit** scored 100% on the training set. Use `cross_val_score` with `cv=5` to score a `DecisionTreeClassifier(random_state=0)` (no `max_depth`) on the **training** set. Store the 5 scores in `fold_scores` and their mean in `cv_mean`." ] }, { "cell_type": "code", "execution_count": null, "id": "p6-050", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "fold_scores = ...\n", "cv_mean = ...\n", "print(\"5 rounds:\", fold_scores.round(3))\n", "print(\"mean: \", round(cv_mean, 3))" ] }, { "cell_type": "code", "execution_count": null, "id": "37a02671", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"07B.1\")" ] }, { "cell_type": "markdown", "id": "p6-057", "metadata": {}, "source": [ "100% on the data it learned from; about 63% on the folds it didn't see. Cross-validation tells the same story as the test set — without touching it." ] }, { "cell_type": "markdown", "id": "p6-059", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 07B.2 — A Cross-Validation Score for Every Depth *(3 pts)*\n", "\n", "For every depth in `depths`, compute the 5-fold cross-validation scores of a tree with that `max_depth` (and `random_state=0`) on the **training** set. Store the **mean** and the **standard deviation** (`.std()`, how much the 5 rounds disagree) of the 5 scores. The plot is given.\n", "\n", "Then store in `best_depth` the depth with the highest mean cross-validation score. *(Hint: `cv_scores.loc[cv_scores[\"mean\"].idxmax(), \"max_depth\"]`.)*" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-061", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "rows = []\n", "for depth in depths:\n", " # 5-fold cross-validation scores for a tree with this max_depth, on the training set\n", " s = ...\n", " rows.append({\"max_depth\": depth, \"mean\": s.mean(), \"spread\": s.std()})\n", "cv_scores = pd.DataFrame(rows)\n", "\n", "best_depth = ...\n", "print(\"best depth by cross-validation:\", best_depth)\n", "\n", "px.line(cv_scores, x=\"max_depth\", y=\"mean\", error_y=\"spread\", markers=True,\n", " labels={\"mean\": \"mean CV accuracy\"},\n", " title=\"5-Fold Cross-Validation Accuracy vs. Depth (800 training sites)\").show()" ] }, { "cell_type": "code", "execution_count": null, "id": "7285d0f7", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"07B.2\")" ] }, { "cell_type": "markdown", "id": "p6-068", "metadata": {}, "source": [ "Now, **once**, open the locked test set for the depth that cross-validation chose:" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-069", "metadata": {}, "outputs": [], "source": [ "final_tree = DecisionTreeClassifier(max_depth=best_depth, random_state=0).fit(X_train, y_train) # all 800 training sites\n", "print(f\"depth {best_depth} | CV: {cv_scores['mean'].max():.3f} | test: {accuracy_score(y_test, final_tree.predict(X_test)):.3f}\")" ] }, { "cell_type": "markdown", "id": "p6-070", "metadata": {}, "source": [ "> 📌 **One split:** that test score is on *these* 200 test sites." ] }, { "cell_type": "markdown", "id": "p6-071", "metadata": {}, "source": [ "---\n", "## Part 4 — Tuning Several Knobs: Grid Search (SLO 07C)\n", "\n", "`max_depth` isn't the only knob. The pruning knobs from Week 7 also include `min_samples_leaf` (every leaf needs at least that many sites) and `min_samples_split` (a box needs at least that many sites before it may ask another question). **Grid search** cross-validates every combination and keeps the best one." ] }, { "cell_type": "markdown", "id": "p6-073", "metadata": { "deletable": false, "editable": false }, "source": [ "### Task 07C.1 — Grid Search *(3 pts)*\n", "\n", "1. Build a `GridSearchCV` with a `DecisionTreeClassifier(random_state=0)`, the grid `grid` below, and `cv=5`. Store it in `search` and fit it on the **training** set.\n", "2. Store how many trees the search fit during cross-validation in `n_fits` — combinations × folds, as a number. *(Count the combinations from `grid`; don't count the final refit.)*\n", "3. Store the best cross-validation accuracy in `best_cv`, and the accuracy of the winning tree (`search.best_estimator_`) on the **test** set in `winner_test`.\n", "\n", "The table of the top 10 combinations is given." ] }, { "cell_type": "code", "execution_count": null, "id": "p6-075", "metadata": { "tags": [ "otter_answer_cell" ] }, "outputs": [], "source": [ "grid = {\"max_depth\": [2, 4, 6, 8, 10, None], # None = no limit\n", " \"min_samples_leaf\": [1, 5, 10, 20],\n", " \"min_samples_split\": [2, 10, 20]}\n", "\n", "# build the grid search, then fit it on the training set\n", "search = ...\n", "...\n", "# trees fit during cross-validation: combinations x folds\n", "n_fits = ...\n", "\n", "# best CV accuracy, and the winner's accuracy on the test set\n", "best_cv = ...\n", "winner_test = ...\n", "print(\"best combination:\", search.best_params_)\n", "print(f\"best CV accuracy: {best_cv:.3f} | test accuracy: {winner_test:.3f} | trees fit: {n_fits}\")\n", "\n", "results = pd.DataFrame(search.cv_results_)\n", "results[[\"param_max_depth\", \"param_min_samples_leaf\", \"param_min_samples_split\", \"mean_test_score\", \"std_test_score\"]] \\\n", " .sort_values(\"mean_test_score\", ascending=False).head(10).round(3)" ] }, { "cell_type": "code", "execution_count": null, "id": "8cbcc890", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "grader.check(\"07C.1\")" ] }, { "cell_type": "markdown", "id": "p6-082", "metadata": {}, "source": [ "### Three Numbers for Two Trees (given — just run it)" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-083", "metadata": {}, "outputs": [], "source": [ "full = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)\n", "winner = search.best_estimator_\n", "pd.DataFrame({\n", " \"leaves\": [full.get_n_leaves(), winner.get_n_leaves()],\n", " \"training accuracy\": [accuracy_score(y_train, full.predict(X_train)), accuracy_score(y_train, winner.predict(X_train))],\n", " \"CV accuracy\": [cv_mean, best_cv],\n", " \"test accuracy\": [accuracy_score(y_test, full.predict(X_test)), winner_test],\n", "}, index=[\"no limit\", \"grid winner\"]).round(3)" ] }, { "cell_type": "markdown", "id": "p6-085", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "**Question 07C.2 *(1 pt)*.** Look at the table above and the top-10 table.\n", "\n", "1. Did the grid winner overfit? Compare its training, CV and test accuracy, and compare it with the tree with no limit.\n", "2. Look at the top-10 table. Several combinations tie for first place, and the plain depth-4 tree (`min_samples_leaf=1`, `min_samples_split=2`) is close behind. How many of the 800 training sites does that difference amount to? Which tree would you choose, and why?" ] }, { "cell_type": "markdown", "id": "3efe81b3", "metadata": { "tags": [ "otter_answer_cell" ] }, "source": [ "_Type your answer here, replacing this text._" ] }, { "cell_type": "markdown", "id": "p6-090", "metadata": { "deletable": false, "editable": false }, "source": [ "\n", "\n", "---\n", "## The Answer Key: How the Data Was Made (given — just run it)\n", "\n", "With real data, nobody hands you the true rules. Here we can peek, because we simulated the data. Each site's organism came from these rules:\n", "\n", "| if… | organism |\n", "|---|---|\n", "| rainfall < 600 mm and temperature > 18 °C | Reptile |\n", "| rainfall < 600 mm and temperature ≤ 18 °C | Mammal |\n", "| otherwise, elevation > 1500 m | Bird |\n", "| otherwise, rainfall > 1800 mm | Plant |\n", "| otherwise, temperature > 20 °C | Insect |\n", "| otherwise | Mammal |\n", "\n", "Then **1 sighting in 4** was replaced by a random organism: the out-of-place sightings. No model can predict those, so even the true rules get only about **80%** right:" ] }, { "cell_type": "code", "execution_count": null, "id": "p6-091", "metadata": {}, "outputs": [], "source": [ "E, T, P = X_test[\"elevation_m\"], X_test[\"temperature_c\"], X_test[\"precipitation_mm\"]\n", "true_rules = np.where(P < 600, np.where(T > 18, \"Reptile\", \"Mammal\"),\n", " np.where(E > 1500, \"Bird\",\n", " np.where(P > 1800, \"Plant\",\n", " np.where(T > 20, \"Insect\", \"Mammal\"))))\n", "print(\"the true rules, on the test sites:\", accuracy_score(y_test, true_rules))" ] }, { "cell_type": "markdown", "id": "p6-092", "metadata": {}, "source": [ "* Compare the rules with your depth-4 tree's questions (`plot_tree(search.best_estimator_, ...)` if you're curious): rainfall ≈ 1775 and ≈ 600, temperature ≈ 19.7, elevation ≈ 1485. The tree found almost the same cuts from 800 noisy sites\n", "* Depth 4 scores about as well as the **true rules**. Every question past that point can only learn the out-of-place sightings, which is exactly what overfitting means\n", "* Depths 1–2 can't even express the rules: they have too few leaves. That's underfitting\n", "* In real data, there is no answer key, so we can't know where the 80% ceiling is. That's why we let cross-validation choose" ] }, { "cell_type": "markdown", "id": "p6-093", "metadata": {}, "source": [ "---\n", "## Submission\n", "\n", "Save your notebook, then upload the **`.ipynb` file** directly to our Moodle assignment page. Don't export, zip, or submit anything else — just the notebook." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" }, "otter": { "OK_FORMAT": true, "assignment_name": "practice06", "tests": { "07A.1": { "name": "07A.1", "points": 2, "suites": [ { "cases": [ { "code": ">>> assert tree2.get_depth() == 2, 'use max_depth=2'\n>>> assert 0 <= train_acc <= 1 and 0 <= test_acc <= 1, 'accuracies lie between 0 and 1'\n>>> assert tree2.n_features_in_ == 3, 'fit on X_train (the three numeric features)'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "07A.3": { "name": "07A.3", "points": 3, "suites": [ { "cases": [ { "code": ">>> assert len(scores) == 22, 'one row per depth in depths'\n>>> assert scores[['train', 'test']].apply(lambda c: c.between(0, 1).all()).all(), 'accuracies lie between 0 and 1'\n>>> assert scores['train'].is_monotonic_increasing, 'training accuracy should never drop as the tree gets deeper'\n>>> assert scores.loc[21, 'train'] == 1.0, 'at depth 22 every leaf is pure'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "07B.1": { "name": "07B.1", "points": 2, "suites": [ { "cases": [ { "code": ">>> assert len(fold_scores) == 5, 'use cv=5'\n>>> assert abs(cv_mean - np.mean(fold_scores)) < 1e-09, 'cv_mean is the mean of the 5 scores'\n>>> assert cv_mean < 0.95, 'score the tree on folds it did not see — not on the data it learned from'\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "07B.2": { "name": "07B.2", "points": 3, "suites": [ { "cases": [ { "code": ">>> assert len(cv_scores) == 22, 'one row per depth in depths'\n>>> assert cv_scores['mean'].between(0, 1).all(), 'mean scores lie between 0 and 1'\n>>> assert (cv_scores['spread'] > 0).all(), 'store the standard deviation of the 5 scores'\n>>> assert best_depth == cv_scores.loc[cv_scores['mean'].idxmax(), 'max_depth']\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] }, "07C.1": { "name": "07C.1", "points": 3, "suites": [ { "cases": [ { "code": ">>> assert isinstance(search, GridSearchCV) and search.cv == 5, 'use GridSearchCV with cv=5'\n>>> assert hasattr(search, 'best_estimator_'), 'fit the search on the training set'\n>>> assert n_fits == 360, 'count the combinations in grid, then multiply by the number of folds'\n>>> assert abs(best_cv - search.best_score_) < 1e-09\n", "hidden": false, "locked": false } ], "scored": true, "setup": "", "teardown": "", "type": "doctest" } ] } } } }, "nbformat": 4, "nbformat_minor": 5 }