{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5696ef54",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"practice04.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d740a0",
   "metadata": {},
   "source": [
    "**Student names and e-mails:**\n",
    "\n",
    "_YOUR NAME — your@calvin.edu_\n",
    "\n",
    "_YOUR NAME — your@calvin.edu_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "783d6c68",
   "metadata": {},
   "source": [
    "# Practice 04 — Clustering and PCA on Birdsong\n",
    "\n",
    "Five kinds of North American birds, 4,268 short clips of their songs. Can an algorithm that never hears the species names find the species — and if it can't, what *does* it find?\n",
    "\n",
    "| SLO | Description |\n",
    "|-----|-------------|\n",
    "| **05A** | Apply k-means clustering to group data and interpret the resulting cluster assignments |\n",
    "| **05B** | Evaluate clustering quality using the elbow method and silhouette score to choose an appropriate k |\n",
    "| **05C** | Apply PCA to reduce dimensionality and interpret how much variance each component explains |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c901b7e6",
   "metadata": {},
   "source": [
    "---\n",
    "## The Dataset: Birdsong\n",
    "\n",
    "![Top row: one clip's spectrogram per species, pitch going up and time going across. Bottom row: the same clip averaged over time into 64 numbers.](https://cs.calvin.edu/courses/data/202/26fa/weeks/05/images/birdsong_spectrograms.png)\n",
    "\n",
    "The recordings come from [Xeno-canto](https://xeno-canto.org/), a community archive of bird sounds, by way of the [BirdCLEF 2021](https://www.kaggle.com/competitions/birdclef-2021) machine-learning competition. Five species: American Robin, Bewick's Wren, Northern Cardinal, Northern Mockingbird and Song Sparrow.\n",
    "\n",
    "Each of 377 recordings was cut into short **clips** — about 11 per recording, 4,268 in all. Each clip was turned into a **spectrogram**: a picture of the sound, with pitch going up and time going across (top row). Then each spectrogram was averaged over time, so every clip becomes **64 numbers** — how loud each of 64 pitch bands is, on average (bottom row). That's the same size as a handwritten digit from class: 64 numbers per row.\n",
    "\n",
    "| Column | What it holds |\n",
    "|---|---|\n",
    "| `recording_id` | the recording the clip was cut from |\n",
    "| `band_00` … `band_63` | average loudness in each pitch band, lowest pitch first |\n",
    "| `loudest_band` | the band (0–63) with the highest average loudness |\n",
    "| `common_name`, `scientific_name` | the species — our \"answer key,\" which the algorithms never see |\n",
    "| `country`, `recordist` | where the recording was made, and by whom |\n",
    "| `source_url` | **listen to the whole recording** on Xeno-canto |\n",
    "| `license` | the recording's Creative Commons license |\n",
    "\n",
    "*Each recording is shared by its recordist under the Creative Commons license in its row; this derived dataset is shared under CC BY-NC-SA 4.0.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fa4d464",
   "metadata": {},
   "source": [
    "### Setup (given — just run it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d640fbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import plotly.express as px\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.metrics import silhouette_score\n",
    "\n",
    "birds = pd.read_csv(\"https://cs.calvin.edu/courses/data/202/26fa/datasets/birdsong.csv\")\n",
    "bands = [f\"band_{i:02d}\" for i in range(64)]   # the 64 pitch-band columns\n",
    "X = birds[bands]                                 # what the algorithms see: 64 numbers per clip\n",
    "birds.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a1635e",
   "metadata": {},
   "source": [
    "### A First Look (given — just run it)\n",
    "\n",
    "What does an *average* clip of each species look like? This averages the 64 bands within each species, reshapes the result into long format (Week 4's `melt`), and draws one line per species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15cc3300",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile = birds.groupby(\"common_name\")[bands].mean().reset_index()\n",
    "profile_long = profile.melt(id_vars=\"common_name\", var_name=\"band\", value_name=\"loudness\")\n",
    "profile_long[\"band\"] = profile_long[\"band\"].str.replace(\"band_\", \"\").astype(int)\n",
    "\n",
    "px.line(profile_long, x=\"band\", y=\"loudness\", color=\"common_name\",\n",
    "        labels={\"band\": \"pitch band (low → high)\", \"loudness\": \"average loudness\", \"common_name\": \"species\"},\n",
    "        title=\"Average pitch profile of each species\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0677bc",
   "metadata": {},
   "source": [
    "The species' profiles differ — but each line is an *average* of hundreds of clips. Individual clips vary a lot more. Keep that in mind as you cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20763413",
   "metadata": {},
   "source": [
    "---\n",
    "## Part 1 — Clustering (SLO 05A)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2ac2926",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05A.1 — k-means *(2 pts)*\n",
    "\n",
    "Cluster the clips into **5** groups — one per species, if the algorithm could find them — using only the 64 band columns in `X`.\n",
    "\n",
    "Use `KMeans(n_clusters=5, n_init=10, random_state=42)`, store the fitted model in `kmeans`, and add each clip's cluster number to `birds` as a new column `\"cluster\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9443e086",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "kmeans = ...\n",
    "birds[\"cluster\"] = ...\n",
    "birds[\"cluster\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86ec409d",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05A.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b23cdc7",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05A.2 — Do the Clusters Match the Species? *(2 pts)*\n",
    "\n",
    "Count how many clips of each species (`common_name`) landed in each cluster, with `pd.crosstab(rows, columns)` — rows = clusters, columns = species. Store the table in `species_by_cluster`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a6d195d",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "species_by_cluster = ...\n",
    "species_by_cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63e3b96b",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05A.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5573356",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05A.3 — What Sets the Clusters Apart? *(2 pts)*\n",
    "\n",
    "Two steps:\n",
    "\n",
    "1. Add a column `mean_loudness` to `birds`: each clip's average over its 64 band columns. That's a mean **across each row**, not down each column — which `axis` is that?\n",
    "2. Build `cluster_profile`: one row per cluster, with named aggregation (as in Week 3) — `n_clips` (the `\"count\"` of `recording_id`: one per clip), `loudest_band` (mean of `loudest_band`) and `mean_loudness` (mean of `mean_loudness`). Sort it by `loudest_band`, lowest first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff7dd57",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "birds[\"mean_loudness\"] = ...\n",
    "cluster_profile = ...\n",
    "cluster_profile.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0e688d",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05A.3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5934b0fd",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05A.4 — Clips of the Same Recording *(2 pts)*\n",
    "\n",
    "Every recording was cut into about 11 clips. Do clips of the same recording land in the same cluster?\n",
    "\n",
    "1. Build `rec_by_cluster` with `pd.crosstab` as in Task 05A.2, but with `recording_id` as the rows, and add `normalize=\"index\"`: each row's counts become **shares** that add up to 1.\n",
    "2. Store each recording's **largest** share (the max across each row) in `top_share`, and the average of `top_share` in `avg_top_share`.\n",
    "\n",
    "If all of a recording's clips fall in one cluster, its top share is 1. The last line (given) computes the same average for the five *species*, for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2524092d",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "rec_by_cluster = ...\n",
    "top_share = ...\n",
    "avg_top_share = ...\n",
    "print(\"recordings:\", round(avg_top_share, 2))\n",
    "print(\"species:   \", round(pd.crosstab(birds[\"common_name\"], birds[\"cluster\"], normalize=\"index\").max(axis=1).mean(), 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d4d1c85",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05A.4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09fb26b5",
   "metadata": {},
   "source": [
    "### Listen! (given — just run it)\n",
    "\n",
    "The two recordings with the most clips in the cluster with the **lowest** average `loudest_band`, and the two with the most clips in the **highest**. Click each link and listen for a minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5de38e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "low, high = cluster_profile.index[0], cluster_profile.index[-1]\n",
    "print(\"lowest-pitch cluster:\", low, \"   highest-pitch cluster:\", high)\n",
    "\n",
    "examples = (birds[birds[\"cluster\"].isin([low, high])]\n",
    "            .groupby([\"cluster\", \"common_name\", \"source_url\"], as_index=False)\n",
    "            .size()                                   # how many clips each recording has in that cluster\n",
    "            .sort_values(\"size\", ascending=False)\n",
    "            .groupby(\"cluster\").head(2))              # the top two recordings of each cluster\n",
    "HTML(examples.to_html(index=False, render_links=True))   # render_links makes the URLs clickable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71709f17",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 05A.5 *(2 pts)*.** Do the clusters line up with the five species? If not, what is k-means grouping the clips by? Support your answer with numbers from `species_by_cluster`, `cluster_profile` and `avg_top_share`, and with what you heard."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d0a8f0",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7eb480b",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "---\n",
    "## Part 2 — How Many Clusters? (SLO 05B)\n",
    "\n",
    "We picked k = 5 because we know there are five species. With no labels, would the data itself suggest 5?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd9e6ef7",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05B.1 — Elbow and Silhouette *(3 pts)*\n",
    "\n",
    "For **k = 2 to 10**, fit k-means (same `n_init=10, random_state=42`) and store, in order of k, each model's `inertia_` in the list `inertias` and its `silhouette_score(X, km.labels_)` in the list `sil_scores`. Then store the k with the **highest** silhouette score in `best_k`. The plots at the end are given.\n",
    "\n",
    "*Hint:* `np.argmax(sil_scores)` gives the *position* of the highest score in the list — and position 0 holds k = 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "566ac74a",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "ks = list(range(2, 11))\n",
    "inertias, sil_scores = [], []\n",
    "for k in ks:\n",
    "    # fit k-means with k clusters (n_init=10, random_state=42)\n",
    "    km = ...\n",
    "    # add its inertia_ to inertias, and its silhouette score to sil_scores\n",
    "    ...\n",
    "    ...\n",
    "\n",
    "best_k = ...\n",
    "print(\"best k by silhouette:\", best_k)\n",
    "\n",
    "px.line(x=ks, y=inertias, markers=True,\n",
    "        labels={\"x\": \"number of clusters k\", \"y\": \"inertia\"}, title=\"Elbow method\").show()\n",
    "px.line(x=ks, y=sil_scores, markers=True,\n",
    "        labels={\"x\": \"number of clusters k\", \"y\": \"silhouette score\"}, title=\"Silhouette score\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c026242",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05B.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec44004e",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 05B.2 *(1 pt)*.** Does the elbow bend sharply at any k? Which k has the best silhouette, and how high is that score? What do the two plots together say about how clearly the clips form groups — and about k = 5?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea3708ed",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0d3caf8",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "---\n",
    "## Part 3 — PCA: 64 Numbers → 2 (SLO 05C)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "834041a1",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05C.1 — Two Components *(2 pts)*\n",
    "\n",
    "Reduce `X` to 2 components with `PCA(n_components=2, random_state=42)`. Store the 2-D result in `X_2d` and the share of the variance the two components keep (a number between 0 and 1) in `share_2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "744731b1",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "pca = ...\n",
    "X_2d = ...\n",
    "share_2 = ...\n",
    "print(X_2d.shape, round(share_2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2590861c",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05C.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938b933d",
   "metadata": {},
   "source": [
    "### Two Views of the Same Shadow (given — just run it)\n",
    "\n",
    "The same 2-D picture, colored two ways: by the true species, and by your k-means clusters. Compare them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab1f0d56",
   "metadata": {},
   "outputs": [],
   "source": [
    "view = pd.DataFrame({\"PC1\": X_2d[:, 0], \"PC2\": X_2d[:, 1],\n",
    "                     \"species\": birds[\"common_name\"], \"cluster\": birds[\"cluster\"].astype(str)})\n",
    "px.scatter(view, x=\"PC1\", y=\"PC2\", color=\"species\", opacity=0.5,\n",
    "           title=\"PCA view, colored by species\").show()\n",
    "px.scatter(view, x=\"PC1\", y=\"PC2\", color=\"cluster\", opacity=0.5,\n",
    "           title=\"PCA view, colored by k-means cluster\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a6220e7",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Task 05C.2 — How Many Components? *(2 pts)*\n",
    "\n",
    "Fit PCA with **all** components and find the smallest number of components that keeps at least **90%** of the variance. Store it in `n_90`.\n",
    "\n",
    "*Hint:* `np.cumsum()` of `explained_variance_ratio_` gives the share kept by the first 1, 2, 3, … components. Print it and count — or let `np.argmax(cumulative >= 0.9)` find the *position* of the first share that reaches 0.9 (position 0 = 1 component)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71d5a150",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "outputs": [],
   "source": [
    "pca_all = ...\n",
    "cumulative = ...\n",
    "print(cumulative.round(3))\n",
    "n_90 = ...\n",
    "n_90"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca5d0f86",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"05C.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3126ce36",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 05C.3 *(1 pt)*.** All 64 bands measure loudness on the same scale, but some vary much more than others (standard deviations from about 0.04 to 0.28). Would you use `StandardScaler` before PCA here? What would scaling change?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a67438",
   "metadata": {
    "tags": [
     "otter_answer_cell"
    ]
   },
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa87e5b",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "---\n",
    "## Submission\n",
    "\n",
    "Save your notebook, then upload the **`.ipynb` file** directly to our Moodle assignment page. Don't export, zip, or submit anything else — just the notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  },
  "otter": {
   "OK_FORMAT": true,
   "assignment_name": "practice04",
   "tests": {
    "05A.1": {
     "name": "05A.1",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert 'cluster' in birds.columns, 'add a \"cluster\" column to birds'\n>>> assert birds['cluster'].nunique() == 5, 'there should be exactly 5 clusters'\n>>> assert len(birds) == 4268, 'birds should still have one row per clip'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05A.2": {
     "name": "05A.2",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert species_by_cluster.shape == (5, 5), '5 clusters (rows) x 5 species (columns)'\n>>> assert species_by_cluster.values.sum() == 4268, 'every clip should be counted once'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05A.3": {
     "name": "05A.3",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert abs(birds['mean_loudness'].mean() - X.values.mean()) < 1e-06, 'mean_loudness should be a mean across each row (axis=1)'\n>>> assert list(cluster_profile.columns) == ['n_clips', 'loudest_band', 'mean_loudness']\n>>> assert len(cluster_profile) == 5\n>>> assert cluster_profile['loudest_band'].is_monotonic_increasing, 'sort by loudest_band, lowest first'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05A.4": {
     "name": "05A.4",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert rec_by_cluster.shape == (377, 5), 'one row per recording (377), one column per cluster (5)'\n>>> assert len(top_share) == 377, 'one top share per recording'\n>>> assert 0 < avg_top_share <= 1, 'avg_top_share is an average of shares, between 0 and 1'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05B.1": {
     "name": "05B.1",
     "points": 3,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert len(inertias) == 9 and len(sil_scores) == 9, 'one value per k, for k = 2 to 10'\n>>> assert np.all(np.diff(inertias) < 0), 'inertia should go down as k grows'\n>>> assert all((-1 <= s <= 1 for s in sil_scores)), 'silhouette scores lie between -1 and 1'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05C.1": {
     "name": "05C.1",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert X_2d.shape == (4268, 2), 'X_2d should have one row per clip and 2 columns'\n>>> assert 0 < share_2 < 1, 'share_2 is a share of the variance, between 0 and 1'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "05C.2": {
     "name": "05C.2",
     "points": 2,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> assert 1 <= n_90 <= 64, 'n_90 is a number of components, between 1 and 64'\n",
         "hidden": false,
         "locked": false
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}