{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2cc9a68",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"practice07.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-001",
   "metadata": {},
   "source": [
    "**Student names and e-mails:**\n",
    "\n",
    "_YOUR NAME — your@calvin.edu_\n",
    "\n",
    "_YOUR NAME — your@calvin.edu_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-002",
   "metadata": {},
   "source": [
    "# Practice 07 — Who Gets Adopted? Scaling, Encoding, Pipelines and Missing Data\n",
    "\n",
    "An animal shelter records every animal that comes in, and whether it was adopted within 30 days. The shelter would like to know early which animals are likely to wait, so volunteers can give them extra attention. Can kNN learn that from the records?\n",
    "\n",
    "Not as they are. The records mix dollars with years, words with numbers, and some boxes were left blank. This week's work is getting the data ready.\n",
    "\n",
    "| SLO | Description |\n",
    "|-----|-------------|\n",
    "| **09A** | Apply preprocessing steps (scaling and one-hot encoding) and explain why each is needed before modeling |\n",
    "| **09B** | Build a scikit-learn pipeline that chains preprocessing and modeling steps into a single reproducible workflow |\n",
    "| **09C** | Identify how missing data arise (MCAR, MAR, MNAR) and choose appropriate imputation strategies for each case |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-003",
   "metadata": {},
   "source": [
    "---\n",
    "## The Dataset: 2,000 Shelter Animals\n",
    "\n",
    "The data is **simulated**. Its patterns were put there on purpose, and at the end you'll see exactly how, including the true values hidden behind every blank. With real data, nobody can show you those. That makes it a good place to practice, but nothing here is a claim about real shelters.\n",
    "\n",
    "| Column | What it holds |\n",
    "|---|---|\n",
    "| `animal_id` | the shelter's ID for the animal |\n",
    "| `species` | Dog, Cat or Rabbit |\n",
    "| `size` | Small, Medium, Large or Extra large |\n",
    "| `intake_type` | how the animal arrived: **Stray** (found), **Owner surrender** (given up by its owner) or **Transfer** (sent by another shelter) |\n",
    "| `age_years` | age, in years |\n",
    "| `weight_kg` | weight at intake, in kg |\n",
    "| `behavior_score` | temperament assessment, from 1 (very stressed or reactive) to 10 (calm and friendly) |\n",
    "| `adoption_fee` | in dollars |\n",
    "| `adopted` | **the label**: 1 = adopted within 30 days, 0 = not |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-004",
   "metadata": {},
   "source": [
    "### Setup (given — just run it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import plotly.express as px\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.impute import SimpleImputer, KNNImputer\n",
    "\n",
    "shelter = pd.read_csv(\"https://cs.calvin.edu/courses/data/202/26fa/datasets/shelter.csv\")\n",
    "print(len(shelter), \"animals | adopted within 30 days:\", round(shelter[\"adopted\"].mean(), 3))\n",
    "shelter.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-006",
   "metadata": {},
   "source": [
    "Three columns have blanks. As in Monday's class, Parts 1–3 use only the animals with **nothing blank**, and Part 4 brings back the rest.\n",
    "\n",
    "The features are the four numeric columns, `size`, and the two word columns. The split is given: 25% for testing, with `stratify=y`. **The test set stays locked** until a step says to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric = [\"age_years\", \"weight_kg\", \"behavior_score\", \"adoption_fee\"]\n",
    "words = [\"species\", \"intake_type\"]\n",
    "sizes = [\"Small\", \"Medium\", \"Large\", \"Extra large\"]          # smallest to largest\n",
    "\n",
    "complete = shelter.dropna()                                  # Parts 1-3: the animals with no blanks\n",
    "X = complete[numeric + [\"size\"] + words]\n",
    "y = complete[\"adopted\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=1)\n",
    "print(len(complete), \"animals |\", len(X_train), \"training |\", len(X_test), \"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-008",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 1 — Dollars vs. Points: Scaling (SLO 09A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.box(X_train[numeric], title=\"Four Numeric Columns, Four Different Rulers (957 training animals)\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-010",
   "metadata": {},
   "source": [
    "> 🎯 **Predict First:** kNN adds up the differences in all four columns to measure how far apart two animals are. Looking at the box plot, which column will decide who counts as a neighbor?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-012",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09A.1 — Raw vs. Scaled *(2 pts)*\n",
    "\n",
    "Score kNN with `n_neighbors=15` on the four **numeric** columns of the training set, with 5-fold `cross_val_score`:\n",
    "\n",
    "1. On the raw columns. Store the mean of the 5 scores in `cv_raw`.\n",
    "2. With a `StandardScaler` in front. Use `make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15))`: a two-step pipe, so in every round the scaler learns its means and standard deviations from the 4 training folds only. Store the mean score in `cv_scaled`.\n",
    "\n",
    "The third score is given: kNN on `behavior_score` **alone**, unscaled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-014",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=15)\n",
    "\n",
    "# 1. raw columns\n",
    "cv_raw = ...\n",
    "\n",
    "# 2. scaler + kNN, as one pipe\n",
    "scaled_knn = ...\n",
    "cv_scaled = ...\n",
    "\n",
    "cv_behavior = cross_val_score(knn, X_train[[\"behavior_score\"]], y_train, cv=5).mean()\n",
    "print(\"four columns, raw:     \", round(cv_raw, 3))\n",
    "print(\"four columns, scaled:  \", round(cv_scaled, 3))\n",
    "print(\"behavior_score alone:  \", round(cv_behavior, 3))\n",
    "print(\"lazy model, 'every animal is adopted':\", round(y_train.mean(), 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31562a5e",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09A.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-021",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 2 — Words to Numbers: One-Hot and Ordinal Encoding (SLO 09A)\n",
    "\n",
    "kNN needs numbers. Three columns hold words, and they are not all the same kind of word:\n",
    "\n",
    "* `species` and `intake_type` name **kinds**: a cat is not \"between\" a dog and a rabbit. They get **one-hot** encoding: one 0/1 column per category\n",
    "* `size` has a real **order**: Small < Medium < Large < Extra large. **`OrdinalEncoder`** turns ranked words into one column of 0, 1, 2, 3, in the order you give it with `categories=`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-023",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09A.2 — Two Encoders *(1 pt)*\n",
    "\n",
    "Fit both encoders on the **training** set:\n",
    "\n",
    "1. `onehot`: a `OneHotEncoder()` fit on `X_train[words]`\n",
    "2. `ordinal`: an `OrdinalEncoder(categories=[sizes])` fit on `X_train[[\"size\"]]`, so the codes follow the sizes from smallest to largest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-025",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "onehot = ...\n",
    "ordinal = ...\n",
    "\n",
    "print(\"one-hot columns:\", onehot.get_feature_names_out())\n",
    "print(\"size codes 0, 1, 2, 3:\", ordinal.categories_[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06310205",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09A.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-031",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 3 — One Pipe for Every Step: Pipelines and Grid Search (SLO 09B)\n",
    "\n",
    "A `ColumnTransformer` sends different columns down different branches, and a `Pipeline` joins it to the model. Everything is fit on training rows only."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-033",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09B.1 — Build the Pipeline *(3 pts)*\n",
    "\n",
    "1. Build a `ColumnTransformer` called `preprocess` with three branches:\n",
    "   * `(\"num\", StandardScaler(), numeric)`\n",
    "   * `(\"size\", OrdinalEncoder(categories=[sizes]), [\"size\"])`. The codes 0–3 already have a standard deviation close to 1, about the same ruler as the scaled columns, so this branch skips the scaler\n",
    "   * `(\"words\", OneHotEncoder(), words)`\n",
    "2. Build a `Pipeline` called `model` with two steps: `(\"prep\", preprocess)` and `(\"knn\", KNeighborsClassifier(n_neighbors=15))`.\n",
    "3. Store the mean 5-fold cross-validation accuracy of `model` on the training set (all of `X_train`) in `cv_pipe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-035",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "# 1. one branch per kind of column\n",
    "preprocess = ...\n",
    "\n",
    "# 2. preprocessing + kNN\n",
    "model = ...\n",
    "\n",
    "# 3. cross-validate the whole pipe, on the training set\n",
    "cv_pipe = ...\n",
    "\n",
    "print(\"cross-validated accuracy:\", round(cv_pipe, 3))\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a61ba2fc",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09B.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-042",
   "metadata": {},
   "source": [
    "73.4% with the four numbers only → about 78% with the words added: 4 scaled numbers + 1 size column + 3 species + 3 intake types = 11 columns into kNN. One object holds every step, so in every round the cross-validation refit the scaler and both encoders on 4 folds of the training animals."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-044",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09B.2 — Grid Search Over the Whole Pipe *(2 pts)*\n",
    "\n",
    "Choose k fairly, as at the end of Monday's class: build a `GridSearchCV` on `model` with the grid `{\"knn__n_neighbors\": ks}`, `cv=5` and `return_train_score=True`. Store it in `grid` and fit it on the **training** set. (The knob's name is the step's name, two underscores, then the knob.)\n",
    "\n",
    "The rest is given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-046",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "ks = [1, 3, 5, 9, 15, 25, 51, 101, 201]\n",
    "grid = ...\n",
    "...\n",
    "\n",
    "best_k = grid.best_params_[\"knn__n_neighbors\"]\n",
    "best_cv = grid.best_score_\n",
    "print(\"best k:\", best_k, \"| cross-validated accuracy:\", round(best_cv, 3))\n",
    "\n",
    "results = pd.DataFrame(grid.cv_results_)\n",
    "px.line(results, x=\"param_knn__n_neighbors\", y=[\"mean_train_score\", \"mean_test_score\"], markers=True, log_x=True,\n",
    "        title=\"Choosing k: Training Folds (train) vs. Cross-Validated (test)\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a97a547",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09B.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-053",
   "metadata": {},
   "source": [
    "Now, **once**, open the locked test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-054",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"best k:\", best_k, \"| cross-validated:\", round(best_cv, 3))\n",
    "print(\"test accuracy:\", round(grid.score(X_test, y_test), 3))        # refit on all 957 training animals, scored once\n",
    "print(\"lazy model, 'every animal is adopted':\", round(y_test.mean(), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-055",
   "metadata": {},
   "source": [
    "> 📌 **One split:** that test score is on *these* 319 test animals, and only the animals with nothing blank."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-056",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 4 — The Blanks: Missing Data (SLO 09C)\n",
    "\n",
    "Parts 1–3 used 1,276 animals. The model never saw the other **724**. Now all 2,000 come back, with the same split recipe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-057",
   "metadata": {},
   "outputs": [],
   "source": [
    "blank_cols = [\"age_years\", \"weight_kg\", \"behavior_score\"]\n",
    "print(shelter[blank_cols].isna().sum().to_string())\n",
    "\n",
    "X = shelter[numeric + [\"size\"] + words]\n",
    "y = shelter[\"adopted\"]\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=1)\n",
    "print(len(X_train), \"training |\", len(X_test), \"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-058",
   "metadata": {},
   "source": [
    "**What the shelter staff say about the blanks:**\n",
    "\n",
    "> *Age:* \"Ages come from the records that arrive with the animal: from its owner, or from the shelter that sent it. Strays usually come with no records.\"\n",
    ">\n",
    "> *Weight:* \"Weights are written on the paper intake form. Some forms were scanned with the bottom edge cut off, and the weight box is at the bottom.\"\n",
    ">\n",
    "> *Behavior:* \"We do the temperament assessment during the first week. Staff skip it when an animal is too stressed or reactive to assess safely.\"\n",
    "\n",
    "The staff notes are one kind of evidence. The data is another."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-060",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09C.1 — Who Has the Blanks? *(2 pts)*\n",
    "\n",
    "We look for patterns in the **training** animals only: the test animals stay locked, even while we choose a fix.\n",
    "\n",
    "`blanks` is a True/False table (True = blank). The share of blanks by species is given as an example. Store in `blank_by_intake` the share of blanks by **intake type**. The adoption rates are given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-062",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "blanks = X_train[blank_cols].isna()\n",
    "blank_by_species = blanks.groupby(X_train[\"species\"]).mean()\n",
    "\n",
    "blank_by_intake = ...\n",
    "\n",
    "adoption_if_blank = pd.DataFrame({col: y_train.groupby(blanks[col]).mean() for col in blank_cols})\n",
    "adoption_if_blank.index = [\"adopted, if recorded\", \"adopted, if blank\"]\n",
    "\n",
    "display(blank_by_species.round(3))\n",
    "display(blank_by_intake.round(3))\n",
    "display(adoption_if_blank.round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdcd0345",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09C.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-070",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 09C.2 — A Fix for Each Mechanism *(2 pts)*\n",
    "\n",
    "Write the two fixes; the pipeline around them is given. Each fix is a small pipe of its own (*hint:* `make_pipeline(first_step, second_step)`):\n",
    "\n",
    "1. `fill_body`: `StandardScaler()`, then `KNNImputer(n_neighbors=5)`. This is fix C, for age (MAR) and weight (MCAR). Scale first, so the imputer's distances aren't all dollars. A `KNNImputer` only sees the columns in its own branch, so this branch also holds age's clues: weight and fee.\n",
    "2. `flag_behavior`: `SimpleImputer(strategy=\"median\", add_indicator=True)`, then `StandardScaler()`. This is fix D, for behavior (MNAR).\n",
    "\n",
    "The cross-validation is given. Like every score before it, it uses the training animals only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-072",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "fill_body = ...\n",
    "flag_behavior = ...\n",
    "\n",
    "by_mechanism = Pipeline([\n",
    "    (\"prep\", ColumnTransformer([\n",
    "        (\"body\", fill_body, [\"age_years\", \"weight_kg\", \"adoption_fee\"]),\n",
    "        (\"behavior\", flag_behavior, [\"behavior_score\"]),\n",
    "        (\"size\", OrdinalEncoder(categories=[sizes]), [\"size\"]),\n",
    "        (\"words\", OneHotEncoder(), words),\n",
    "    ])),\n",
    "    (\"knn\", KNeighborsClassifier(n_neighbors=best_k)),\n",
    "])\n",
    "cv_mechanism = cross_val_score(by_mechanism, X_train, y_train, cv=5).mean()\n",
    "print(\"by mechanism, cross-validated:\", round(cv_mechanism, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "282d59cb",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"09C.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-079",
   "metadata": {},
   "source": [
    "### Three Fixes Compared (given — just run it)\n",
    "\n",
    "**A:** drop every training animal with a blank, then cross-validate Part 3's pipeline on the rest. **B:** one median for every blank, no indicator. **By mechanism:** your pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-080",
   "metadata": {},
   "outputs": [],
   "source": [
    "median_everywhere = Pipeline([\n",
    "    (\"prep\", ColumnTransformer([\n",
    "        (\"num\", make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler()), numeric),\n",
    "        (\"size\", OrdinalEncoder(categories=[sizes]), [\"size\"]),\n",
    "        (\"words\", OneHotEncoder(), words),\n",
    "    ])),\n",
    "    (\"knn\", KNeighborsClassifier(n_neighbors=best_k)),\n",
    "])\n",
    "keep = X_train.notna().all(axis=1)\n",
    "cv_drop = cross_val_score(grid.best_estimator_, X_train[keep], y_train[keep], cv=5).mean()\n",
    "cv_median = cross_val_score(median_everywhere, X_train, y_train, cv=5).mean()\n",
    "\n",
    "display(pd.DataFrame({\"training animals scored\": [keep.sum(), len(X_train), len(X_train)],\n",
    "                      \"CV accuracy\": [cv_drop, cv_median, cv_mechanism]},\n",
    "                     index=[\"A: drop rows with blanks\", \"B: one median for every blank\",\n",
    "                            \"by mechanism: C for age and weight, D for behavior\"]).round(3))\n",
    "print(\"adoption rate, animals kept by A:   \", round(y_train[keep].mean(), 3))\n",
    "print(\"adoption rate, animals dropped by A:\", round(y_train[~keep].mean(), 3), f\"({(~keep).sum()} animals)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-081",
   "metadata": {},
   "source": [
    "Now, **once**, open the locked test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-082",
   "metadata": {},
   "outputs": [],
   "source": [
    "by_mechanism.fit(X_train, y_train)\n",
    "not_assessed = X_test[\"behavior_score\"].isna()\n",
    "print(\"test accuracy, all\", len(X_test), \"test animals:\", round(by_mechanism.score(X_test, y_test), 3))\n",
    "print(not_assessed.sum(), \"test animals were not assessed:\")\n",
    "print(\"   predicted adopted:\", round(by_mechanism.predict(X_test[not_assessed]).mean(), 3))\n",
    "print(\"   actually adopted: \", round(y_test[not_assessed].mean(), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-083",
   "metadata": {},
   "source": [
    "> 📌 **One split:** these scores are on *these* 500 test animals, from a shelter we simulated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-084",
   "metadata": {},
   "source": [
    "---\n",
    "## The Answer Key: How the Data Was Made (given — just run it)\n",
    "\n",
    "With real data, nobody hands you the missing values. Here we can peek, because we simulated them.\n",
    "\n",
    "**Adoption.** The chance of adoption rises with `behavior_score` and falls with age. Transfers were adopted more, strays less; rabbits, and large and extra-large dogs, less. The fee follows species and age (puppies and kittens cost more), plus noise, so it carries little information of its own.\n",
    "\n",
    "**The blanks:**\n",
    "\n",
    "| column | mechanism | how the blanks were made |\n",
    "|---|---|---|\n",
    "| `weight_kg` | MCAR | 7% of animals, at random |\n",
    "| `age_years` | MAR | 40% of strays, 3% of owner surrenders, 10% of transfers (and strays are younger) |\n",
    "| `behavior_score` | MNAR | 85% of scores ≤ 3, 35% of 4s, 8% of 5s, 2% of the rest |\n",
    "\n",
    "The file below holds the true value of every blank:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p07-085",
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden = pd.read_csv(\"https://cs.calvin.edu/courses/data/202/26fa/datasets/shelter_hidden.csv\")      # same animals, same order, no blanks\n",
    "\n",
    "rows = []\n",
    "for col in blank_cols:\n",
    "    blank = shelter[col].isna()\n",
    "    rows.append({\"column\": col, \"true mean, behind the blanks\": hidden.loc[blank, col].mean(),\n",
    "                 \"mean, recorded\": hidden.loc[~blank, col].mean()})\n",
    "display(pd.DataFrame(rows).set_index(\"column\").round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-086",
   "metadata": {},
   "source": [
    "* **Behavior (MNAR):** behind the blanks, the true scores average about 3.5, against 6.4 for the assessed animals. The median fill (6) put them among the calm animals. The indicator is what let the model find them again. Without the hidden file, only the staff note could have told us this\n",
    "* **Age (MAR):** the blank ages belong mostly to strays, who are younger. One median gives each of them 3.2 years. Filling from similar animals (by fee and weight) misses the true age by 1.5 years on average, against 1.9 for the median. The accuracy barely changed, but the filled data is closer to the truth\n",
    "* **Weight (MCAR):** the blank weights look just like the recorded ones. Any reasonable fix works\n",
    "* In real data there is no hidden file. That's why the mechanism has to come from **how the data was collected**, not from the scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "p07-087",
   "metadata": {},
   "source": [
    "---\n",
    "## Submission\n",
    "\n",
    "Save your notebook, then upload the **`.ipynb` file** directly to our Moodle assignment page. Don't export, zip, or submit anything else — just the notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "otter": {
   "OK_FORMAT": true,
   "assignment_name": "practice07",
   "tests": {
    "09A.1": {
     "name": "09A.1",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert 0 < cv_raw < 1 and 0 < cv_scaled < 1, 'store the MEAN of the 5 scores'\n>>> assert cv_scaled > cv_raw, 'put a StandardScaler in front of kNN for cv_scaled'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "09A.2": {
     "name": "09A.2",
     "points": 1,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert len(onehot.get_feature_names_out()) == 6, 'fit on X_train[words]: 3 species + 3 intake types'\n>>> assert list(ordinal.categories_[0]) == sizes, 'pass categories=[sizes] so the codes go from smallest to largest'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "09B.1": {
     "name": "09B.1",
     "points": 3,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert isinstance(model, Pipeline) and list(model.named_steps) == ['prep', 'knn'], 'two steps: prep and knn'\n>>> assert isinstance(preprocess, ColumnTransformer) and len(preprocess.transformers) == 3, 'three branches: num, size, words'\n>>> assert 0 < cv_pipe < 1, 'store the MEAN of the 5 scores'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "09B.2": {
     "name": "09B.2",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert isinstance(grid, GridSearchCV) and grid.cv == 5, 'use GridSearchCV with cv=5'\n>>> assert hasattr(grid, 'best_estimator_'), 'fit the grid search on the training set'\n>>> assert 'mean_train_score' in grid.cv_results_, 'use return_train_score=True'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "09C.1": {
     "name": "09C.1",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert blank_by_intake.shape == (3, 3), 'one row per intake type, one column per blank column'\n>>> assert blank_by_intake.max().max() <= 1, 'shares of blanks lie between 0 and 1'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "09C.2": {
     "name": "09C.2",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert isinstance(fill_body[0], StandardScaler) and isinstance(fill_body[-1], KNNImputer), 'StandardScaler, then KNNImputer'\n>>> assert isinstance(flag_behavior[0], SimpleImputer) and flag_behavior[0].add_indicator, 'SimpleImputer with add_indicator=True, then StandardScaler'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}