class: left, top, title-slide .title[ # Predictive Analytics Unit 3: Intro to Prediction ] .author[ ### Ken Arnold
Calvin University ] --- .pull-left[ <img src="img/edmunds-estimate.png" width="60%" style="display: block; margin: auto;" /> ] .pull-right[ What data frame does Edmunds have? (What are the columns? What does each row represent?) ] .floating-source[Source: Edmunds.com] --- ## Predictive Modeling: A new use for modeling .pull-left[ Oversimplified summary: - Statistical inference: interested in *features* - Example: "Which *market segment* is the strongest driver of sales?" - Prediction: interested in values of *targets* - Example: "How many customers will click through our campaign?" ] .pull-right[ <img src="img/stat-prediction.png" width="75%" style="display: block; margin: auto;" /> ] --- ## What's your goal? * **Predict** unseen labels * How much will this house sell for? * Does this child have autism? * Is this a positive or negative movie review? * **Infer** relationships between features and labels * How much does home size affect price? * Is DNA methylation a marker of autism? * Does "sick" indicate a positive or negative review? * Understand the **causal** effect of interventions * How much will building an addition increase the price of my home? * Will antioxidants prevent autism? * Will cutting this scene make my movie get better reviews? --- ## Some terminology - **Supervised learning**: for each item independently, fill in an unobserved variable - **Regression**: fill in a *number* - **Classification**: fill in a *choice* - **Unsupervised learning**: identify *groups* of items - **Forecasting**: predict how a sequence will continue (future observations) - **Statistical inference**: fill in unknown model coefficients (we wanted a population but only got a sample) - **Causal inference**: fill in counterfactuals (what if?) --- ## Predictive Analytics Mostly *supervised learning* (*regression* and *classification*), some *forecasting*. * A powerful tool to turn data into action. * It works because God made the universe predictable (and successful prediction rewarding) * **Need for wisdom**: It can be used for great good and great harm --- ## Power of Predictive Modeling * **Medicine**: wearable monitor for seizures or falls, detect malaria from blood smears, find effective drug regimens from medical records * **Drug Discovery**: predict the efficacy of a synthesis plan for a drug * **Precision Agriculture**: predict effect of micro-climate on plant growth * **Urban Planning**: forecast resource needs, extreme weather risks, ... * **Government**: classify feedback from constituents * **Retail**: predict items in a grocery order * **Recommendation systems**: Amazon, Netflix, YouTube, ... * **User interfaces**: gesture typing, autocomplete / autocorrect and so much more... --- ## The universe is surprisingly predictable * God created the world with actionable structure * We gradually learn how to perceive that structure and act within it. * The better our perceptions align with how the universe is structured, the better our actions * We can discover that structure by learning to be less surprised by what we see ( = predicting our perceptions) * Perceptions are thus both accurate and fallable. --- ## Predictive modeling technology: Need for wisdom * Potential for great good * But also great harm: * Lack of **fairness** in facial recognition, sentencing, lending, job applicant scoring, ... * Lack of **transparency** in how "Big Data" systems make conclusions * Lack of **privacy** as data is increasingly collected and aggregated * Amplification of extreme positions in social media, YouTube, etc. * Oversimplification of human experience * Hidden human labor * Illusion of objectivity * ...! --- class: center, middle # Examples of Supervised Learning --- ## Regression example: Predict market price for a home From Ames, Iowa home sales, 2006-2010. (De Cock, 2011) .small[ <table> <thead> <tr> <th style="text-align:right;"> Lot_Area </th> <th style="text-align:left;"> Bldg_Type </th> <th style="text-align:right;"> Gr_Liv_Area </th> <th style="text-align:right;"> Garage_Cars </th> <th style="text-align:right;"> Sale_Price </th> </tr> </thead> <tbody> <tr> <td style="text-align:right;"> 12546 </td> <td style="text-align:left;"> OneFam </td> <td style="text-align:right;"> 1440 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;color: white !important;background-color: #8C2131 !important;"> 182.9 </td> </tr> <tr> <td style="text-align:right;"> 2645 </td> <td style="text-align:left;"> Twnhs </td> <td style="text-align:right;"> 1586 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;color: white !important;background-color: #8C2131 !important;"> 170.0 </td> </tr> <tr> <td style="text-align:right;"> 15312 </td> <td style="text-align:left;"> OneFam </td> <td style="text-align:right;"> 1138 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;color: white !important;background-color: #8C2131 !important;"> 148.0 </td> </tr> <tr> <td style="text-align:right;"> 8544 </td> <td style="text-align:left;"> Duplex </td> <td style="text-align:right;"> 1040 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;color: white !important;background-color: #8C2131 !important;"> 81.4 </td> </tr> <tr> <td style="text-align:right;"> 12677 </td> <td style="text-align:left;"> TwnhsE </td> <td style="text-align:right;"> 1518 </td> <td style="text-align:right;"> 2 </td> <td style="text-align:right;color: white !important;background-color: #8C2131 !important;"> 274.0 </td> </tr> </tbody> </table> ] (2412 total rows) * *y*: *response* variable (aka *outcome*, *dependent variable*): `Sale_Price` * *X*: *features* (aka *predictors*, *covariates*, etc.): everything else Note: `X` is much easier to measure than `y` --- ### Model <img src="slides03prediction_files/figure-html/gr_liv_stub-1.png" width="90%" style="display: block; margin: auto;" /> --- ### Predictions <img src="slides03prediction_files/figure-html/decision-tree-example-fit-1.png" width="100%" style="display: block; margin: auto;" /> --- ### Model Code ```r library(tidymodels) set.seed(10) # Make consistent train-test splits. ames_split <- initial_split(ames_home_sales, prop = 2/3) # Split our data randomly ames_train <- training(ames_split) ames_test <- testing(ames_split) ``` ```r *decision_tree_fit <- decision_tree(mode = "regression", tree_depth = 3) %>% set_engine("rpart") %>% fit( Sale_Price ~ Gr_Liv_Area + Bldg_Type, data = ames_train) ``` Plotting the tree: ```r decision_tree_fit %>% extract_fit_engine() %>% rpart.plot::rpart.plot(roundint = FALSE) ``` --- ## Were those predictions good? <table> <thead> <tr> <th style="text-align:left;"> metric </th> <th style="text-align:right;"> estimate </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> rsq </td> <td style="text-align:right;"> 0.5260505 </td> </tr> <tr> <td style="text-align:left;"> mae </td> <td style="text-align:right;"> 35.3756288 </td> </tr> <tr> <td style="text-align:left;"> mape </td> <td style="text-align:right;"> 21.1170598 </td> </tr> <tr> <td style="text-align:left;"> rmse </td> <td style="text-align:right;"> 50.5840333 </td> </tr> </tbody> </table> - Traditional R^2 (fraction of variance explained) - MAE: Mean Absolute Error ("predictions are usually off by $xxx") - MAPE: Mean Absolute Percent Error ("predictions are usually off by yy%") - RMSE: Root Mean Squared Error: mathy, but sorta like the standard deviation --- ## Modeling Workflow --- <img src="img/modeling-process.png" width="100%" style="display: block; margin: auto;" /> .small[Source: [Tidy Modeling with R ch1](https://www.tmwr.org/software-modeling.html)] --- ## Predictive Modeling Workflow Preliminaries: 1. **Define the problem**: predict *what*, based on *what*? What *metrics* will indicate success? (Measure success in multiple ways!) 2. **Explore your data** (EDA): understand its structure, make lots of plots -- Modeling: 1. **Pick a model**: Which type(s) of models are appropriate for task and data? 2. **Transform the data** as needed by the model ("feature engineering", preprocessing", "recipe") 3. **Split the data** to allow for validation. 4. **Fit and evaluate the model** 5. **Tune**: adjust model hyperparameters 6. **Analyze model errors** and refine all earlier steps --- ```r library(tidymodels) ``` Packages: * `parsnip`: **Specify** and **train** the model you want * `recipes`: **Prepare** the data * `rsample`: **Split** data into training and validation * `yardstick`: Compute **metrics** for performance * `tune`: Helps you set the dials. Many alternative packages available also. Famous example: Scikit-Learn (sklearn) in Python. --- ## Example data: Ames home sales ```r ames_home_sales <- read_builtin("ames", package = "modeldata") %>% mutate(Sale_Price = Sale_Price / 1000) %>% filter(Gr_Liv_Area < 4000, Sale_Condition == "Normal") ``` <!-- Like before, but we subset the data as [De Cock](http://jse.amstat.org/v19n3/decock.pdf) suggests. --> <!-- Again, see [Data dictionary](http://jse.amstat.org/v19n3/decock/DataDocumentation.txt) --> ```r nrow(ames_home_sales) ``` ``` [1] 2412 ``` ```r ames_home_sales %>% head(5) ``` ``` # A tibble: 5 × 74 MS_SubCl…¹ MS_Zo…² Lot_F…³ Lot_A…⁴ Street Alley Lot_S…⁵ Land_…⁶ <fct> <fct> <dbl> <int> <fct> <fct> <fct> <fct> 1 One_Story… Reside… 141 31770 Pave No_A… Slight… Lvl 2 One_Story… Reside… 80 11622 Pave No_A… Regular Lvl 3 One_Story… Reside… 81 14267 Pave No_A… Slight… Lvl 4 One_Story… Reside… 93 11160 Pave No_A… Regular Lvl 5 Two_Story… Reside… 74 13830 Pave No_A… Slight… Lvl # … with 66 more variables: Utilities <fct>, Lot_Config <fct>, # Land_Slope <fct>, Neighborhood <fct>, Condition_1 <fct>, # Condition_2 <fct>, Bldg_Type <fct>, House_Style <fct>, # Overall_Cond <fct>, Year_Built <int>, Year_Remod_Add <int>, # Roof_Style <fct>, Roof_Matl <fct>, Exterior_1st <fct>, # Exterior_2nd <fct>, Mas_Vnr_Type <fct>, Mas_Vnr_Area <dbl>, # Exter_Cond <fct>, Foundation <fct>, Bsmt_Cond <fct>, … ``` --- ## Defining the problem - Predict *what*? `Sale_Price` - How to measure success? ```r metrics <- yardstick::metric_set(mae, mape, rsq_trad) ``` --- ## Exploratory Analysis (EDA) .tiny-table.col2[ ```r skimr::skim(ames_home_sales) ``` <table style='width: auto;' class='table table-condensed'> <caption>Data summary</caption> <tbody> <tr> <td style="text-align:left;"> Name </td> <td style="text-align:left;"> ames_home_sales </td> </tr> <tr> <td style="text-align:left;"> Number of rows </td> <td style="text-align:left;"> 2412 </td> </tr> <tr> <td style="text-align:left;"> Number of columns </td> <td style="text-align:left;"> 74 </td> </tr> <tr> <td style="text-align:left;"> _______________________ </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Column type frequency: </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> factor </td> <td style="text-align:left;"> 40 </td> </tr> <tr> <td style="text-align:left;"> numeric </td> <td style="text-align:left;"> 34 </td> </tr> <tr> <td style="text-align:left;"> ________________________ </td> <td style="text-align:left;"> </td> </tr> <tr> <td style="text-align:left;"> Group variables </td> <td style="text-align:left;"> None </td> </tr> </tbody> </table> **Variable type: factor** <table> <thead> <tr> <th style="text-align:left;"> skim_variable </th> <th style="text-align:right;"> n_missing </th> <th style="text-align:right;"> complete_rate </th> <th style="text-align:left;"> ordered </th> <th style="text-align:right;"> n_unique </th> <th style="text-align:left;"> top_counts </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> MS_SubClass </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 16 </td> <td style="text-align:left;"> One: 866, Two: 464, One: 247, One: 155 </td> </tr> <tr> <td style="text-align:left;"> MS_Zoning </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 7 </td> <td style="text-align:left;"> Res: 1890, Res: 394, Flo: 92, Res: 20 </td> </tr> <tr> <td style="text-align:left;"> Street </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 2 </td> <td style="text-align:left;"> Pav: 2403, Grv: 9 </td> </tr> <tr> <td style="text-align:left;"> Alley </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 3 </td> <td style="text-align:left;"> No_: 2258, Gra: 100, Pav: 54 </td> </tr> <tr> <td style="text-align:left;"> Lot_Shape </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 4 </td> <td style="text-align:left;"> Reg: 1533, Sli: 802, Mod: 65, Irr: 12 </td> </tr> <tr> <td style="text-align:left;"> Land_Contour </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 4 </td> <td style="text-align:left;"> Lvl: 2184, Bnk: 91, HLS: 86, Low: 51 </td> </tr> <tr> <td style="text-align:left;"> Utilities </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 2 </td> <td style="text-align:left;"> All: 2410, NoS: 2, NoS: 0 </td> </tr> <tr> <td style="text-align:left;"> Lot_Config </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> Ins: 1765, Cor: 410, Cul: 151, FR2: 74 </td> </tr> <tr> <td style="text-align:left;"> Land_Slope </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 3 </td> <td style="text-align:left;"> Gtl: 2295, Mod: 106, Sev: 11 </td> </tr> <tr> <td style="text-align:left;"> Neighborhood </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 28 </td> <td style="text-align:left;"> Nor: 395, Col: 226, Old: 204, Edw: 155 </td> </tr> <tr> <td style="text-align:left;"> Condition_1 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 9 </td> <td style="text-align:left;"> Nor: 2083, Fee: 130, Art: 77, Pos: 35 </td> </tr> <tr> <td style="text-align:left;"> Condition_2 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 8 </td> <td style="text-align:left;"> Nor: 2390, Fee: 12, Art: 2, Pos: 2 </td> </tr> <tr> <td style="text-align:left;"> Bldg_Type </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> One: 2001, Twn: 188, Twn: 93, Dup: 78 </td> </tr> <tr> <td style="text-align:left;"> House_Style </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 8 </td> <td style="text-align:left;"> One: 1189, Two: 725, One: 270, SLv: 115 </td> </tr> <tr> <td style="text-align:left;"> Overall_Cond </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 9 </td> <td style="text-align:left;"> Ave: 1282, Abo: 473, Goo: 352, Ver: 139 </td> </tr> <tr> <td style="text-align:left;"> Roof_Style </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> Gab: 1936, Hip: 432, Gam: 18, Fla: 13 </td> </tr> <tr> <td style="text-align:left;"> Roof_Matl </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 7 </td> <td style="text-align:left;"> Com: 2381, Tar: 15, WdS: 7, WdS: 6 </td> </tr> <tr> <td style="text-align:left;"> Exterior_1st </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 15 </td> <td style="text-align:left;"> Vin: 781, HdB: 396, Met: 390, Wd : 371 </td> </tr> <tr> <td style="text-align:left;"> Exterior_2nd </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 16 </td> <td style="text-align:left;"> Vin: 770, Met: 388, HdB: 363, Wd : 357 </td> </tr> <tr> <td style="text-align:left;"> Mas_Vnr_Type </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 4 </td> <td style="text-align:left;"> Non: 1492, Brk: 749, Sto: 153, Brk: 18 </td> </tr> <tr> <td style="text-align:left;"> Exter_Cond </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> Typ: 2080, Goo: 266, Fai: 53, Exc: 11 </td> </tr> <tr> <td style="text-align:left;"> Foundation </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> CBl: 1090, PCo: 1002, Brk: 264, Sla: 41 </td> </tr> <tr> <td style="text-align:left;"> Bsmt_Cond </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> Typ: 2169, Goo: 86, Fai: 84, No_: 67 </td> </tr> <tr> <td style="text-align:left;"> Bsmt_Exposure </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> No: 1611, Av: 315, Gd: 218, Mn: 199 </td> </tr> <tr> <td style="text-align:left;"> BsmtFin_Type_1 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 7 </td> <td style="text-align:left;"> GLQ: 694, Unf: 644, ALQ: 378, Rec: 250 </td> </tr> <tr> <td style="text-align:left;"> BsmtFin_Type_2 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 7 </td> <td style="text-align:left;"> Unf: 2026, Rec: 96, LwQ: 78, No_: 68 </td> </tr> <tr> <td style="text-align:left;"> Heating </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> Gas: 2374, Gas: 24, Gra: 6, Wal: 5 </td> </tr> <tr> <td style="text-align:left;"> Heating_QC </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> Exc: 1183, Typ: 726, Goo: 422, Fai: 80 </td> </tr> <tr> <td style="text-align:left;"> Central_Air </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 2 </td> <td style="text-align:left;"> Y: 2259, N: 153 </td> </tr> <tr> <td style="text-align:left;"> Electrical </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> SBr: 2207, Fus: 158, Fus: 39, Fus: 7 </td> </tr> <tr> <td style="text-align:left;"> Functional </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> Typ: 2239, Min: 63, Min: 55, Mod: 30 </td> </tr> <tr> <td style="text-align:left;"> Garage_Type </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 7 </td> <td style="text-align:left;"> Att: 1420, Det: 684, Bui: 140, No_: 116 </td> </tr> <tr> <td style="text-align:left;"> Garage_Finish </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 4 </td> <td style="text-align:left;"> Unf: 1066, RFn: 671, Fin: 558, No_: 117 </td> </tr> <tr> <td style="text-align:left;"> Garage_Cond </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 6 </td> <td style="text-align:left;"> Typ: 2204, No_: 117, Fai: 65, Goo: 13 </td> </tr> <tr> <td style="text-align:left;"> Paved_Drive </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 3 </td> <td style="text-align:left;"> Pav: 2175, Dir: 178, Par: 59 </td> </tr> <tr> <td style="text-align:left;"> Pool_QC </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> No_: 2403, Typ: 3, Exc: 2, Fai: 2 </td> </tr> <tr> <td style="text-align:left;"> Fence </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> No_: 1910, Min: 286, Goo: 107, Goo: 99 </td> </tr> <tr> <td style="text-align:left;"> Misc_Feature </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 5 </td> <td style="text-align:left;"> Non: 2316, She: 87, Gar: 5, Oth: 3 </td> </tr> <tr> <td style="text-align:left;"> Sale_Type </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 9 </td> <td style="text-align:left;"> WD : 2324, COD: 39, Con: 21, CWD: 10 </td> </tr> <tr> <td style="text-align:left;"> Sale_Condition </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> FALSE </td> <td style="text-align:right;"> 1 </td> <td style="text-align:left;"> Nor: 2412, Abn: 0, Adj: 0, All: 0 </td> </tr> </tbody> </table> **Variable type: numeric** <table> <thead> <tr> <th style="text-align:left;"> skim_variable </th> <th style="text-align:right;"> n_missing </th> <th style="text-align:right;"> complete_rate </th> <th style="text-align:right;"> mean </th> <th style="text-align:right;"> sd </th> <th style="text-align:right;"> p0 </th> <th style="text-align:right;"> p25 </th> <th style="text-align:right;"> p50 </th> <th style="text-align:right;"> p75 </th> <th style="text-align:right;"> p100 </th> <th style="text-align:left;"> hist </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Lot_Frontage </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 55.44 </td> <td style="text-align:right;"> 33.53 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 37.00 </td> <td style="text-align:right;"> 60.00 </td> <td style="text-align:right;"> 76.25 </td> <td style="text-align:right;"> 313.00 </td> <td style="text-align:left;"> ▇▇▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Lot_Area </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 10055.45 </td> <td style="text-align:right;"> 8221.14 </td> <td style="text-align:right;"> 1300.00 </td> <td style="text-align:right;"> 7389.50 </td> <td style="text-align:right;"> 9360.00 </td> <td style="text-align:right;"> 11401.00 </td> <td style="text-align:right;"> 215245.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Year_Built </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1969.43 </td> <td style="text-align:right;"> 29.49 </td> <td style="text-align:right;"> 1872.00 </td> <td style="text-align:right;"> 1953.00 </td> <td style="text-align:right;"> 1971.00 </td> <td style="text-align:right;"> 1998.00 </td> <td style="text-align:right;"> 2010.00 </td> <td style="text-align:left;"> ▁▂▃▇▇ </td> </tr> <tr> <td style="text-align:left;"> Year_Remod_Add </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1983.27 </td> <td style="text-align:right;"> 20.33 </td> <td style="text-align:right;"> 1950.00 </td> <td style="text-align:right;"> 1965.00 </td> <td style="text-align:right;"> 1991.00 </td> <td style="text-align:right;"> 2002.00 </td> <td style="text-align:right;"> 2010.00 </td> <td style="text-align:left;"> ▅▃▂▅▇ </td> </tr> <tr> <td style="text-align:left;"> Mas_Vnr_Area </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 95.20 </td> <td style="text-align:right;"> 170.88 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 148.00 </td> <td style="text-align:right;"> 1600.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> BsmtFin_SF_1 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 4.08 </td> <td style="text-align:right;"> 2.22 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 3.00 </td> <td style="text-align:right;"> 7.00 </td> <td style="text-align:right;"> 7.00 </td> <td style="text-align:left;"> ▆▆▁▁▇ </td> </tr> <tr> <td style="text-align:left;"> BsmtFin_SF_2 </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 55.15 </td> <td style="text-align:right;"> 177.32 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1526.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Bsmt_Unf_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 528.35 </td> <td style="text-align:right;"> 417.42 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 209.50 </td> <td style="text-align:right;"> 438.50 </td> <td style="text-align:right;"> 776.00 </td> <td style="text-align:right;"> 2336.00 </td> <td style="text-align:left;"> ▇▅▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> Total_Bsmt_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1022.24 </td> <td style="text-align:right;"> 408.04 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 784.00 </td> <td style="text-align:right;"> 970.00 </td> <td style="text-align:right;"> 1246.00 </td> <td style="text-align:right;"> 3206.00 </td> <td style="text-align:left;"> ▂▇▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> First_Flr_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1133.31 </td> <td style="text-align:right;"> 365.55 </td> <td style="text-align:right;"> 334.00 </td> <td style="text-align:right;"> 866.00 </td> <td style="text-align:right;"> 1060.00 </td> <td style="text-align:right;"> 1349.25 </td> <td style="text-align:right;"> 3820.00 </td> <td style="text-align:left;"> ▇▇▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Second_Flr_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 338.61 </td> <td style="text-align:right;"> 422.13 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 704.00 </td> <td style="text-align:right;"> 1836.00 </td> <td style="text-align:left;"> ▇▂▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> Gr_Liv_Area </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1476.23 </td> <td style="text-align:right;"> 479.69 </td> <td style="text-align:right;"> 334.00 </td> <td style="text-align:right;"> 1100.00 </td> <td style="text-align:right;"> 1430.50 </td> <td style="text-align:right;"> 1724.00 </td> <td style="text-align:right;"> 3820.00 </td> <td style="text-align:left;"> ▂▇▃▁▁ </td> </tr> <tr> <td style="text-align:left;"> Bsmt_Full_Bath </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 0.43 </td> <td style="text-align:right;"> 0.51 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:left;"> ▇▁▆▁▁ </td> </tr> <tr> <td style="text-align:left;"> Bsmt_Half_Bath </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 0.06 </td> <td style="text-align:right;"> 0.24 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Full_Bath </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1.54 </td> <td style="text-align:right;"> 0.54 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 4.00 </td> <td style="text-align:left;"> ▁▇▇▁▁ </td> </tr> <tr> <td style="text-align:left;"> Half_Bath </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 0.38 </td> <td style="text-align:right;"> 0.50 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:left;"> ▇▁▅▁▁ </td> </tr> <tr> <td style="text-align:left;"> Bedroom_AbvGr </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 2.85 </td> <td style="text-align:right;"> 0.81 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 3.00 </td> <td style="text-align:right;"> 3.00 </td> <td style="text-align:right;"> 6.00 </td> <td style="text-align:left;"> ▁▃▇▂▁ </td> </tr> <tr> <td style="text-align:left;"> Kitchen_AbvGr </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1.04 </td> <td style="text-align:right;"> 0.20 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 3.00 </td> <td style="text-align:left;"> ▁▇▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> TotRms_AbvGrd </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 6.37 </td> <td style="text-align:right;"> 1.52 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 5.00 </td> <td style="text-align:right;"> 6.00 </td> <td style="text-align:right;"> 7.00 </td> <td style="text-align:right;"> 13.00 </td> <td style="text-align:left;"> ▂▇▆▁▁ </td> </tr> <tr> <td style="text-align:left;"> Fireplaces </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 0.60 </td> <td style="text-align:right;"> 0.65 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 4.00 </td> <td style="text-align:left;"> ▇▇▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Garage_Cars </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1.73 </td> <td style="text-align:right;"> 0.73 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 2.00 </td> <td style="text-align:right;"> 5.00 </td> <td style="text-align:left;"> ▅▇▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> Garage_Area </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 461.10 </td> <td style="text-align:right;"> 201.58 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 315.00 </td> <td style="text-align:right;"> 471.50 </td> <td style="text-align:right;"> 576.00 </td> <td style="text-align:right;"> 1488.00 </td> <td style="text-align:left;"> ▃▇▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> Wood_Deck_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 95.76 </td> <td style="text-align:right;"> 130.14 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 168.00 </td> <td style="text-align:right;"> 1424.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Open_Porch_SF </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 44.66 </td> <td style="text-align:right;"> 62.95 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 24.00 </td> <td style="text-align:right;"> 68.00 </td> <td style="text-align:right;"> 570.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Enclosed_Porch </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 23.40 </td> <td style="text-align:right;"> 64.47 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 1012.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Three_season_porch </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 2.50 </td> <td style="text-align:right;"> 25.24 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 508.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Screen_Porch </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 16.30 </td> <td style="text-align:right;"> 57.10 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 576.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Pool_Area </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 1.78 </td> <td style="text-align:right;"> 31.38 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 800.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Misc_Val </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 51.50 </td> <td style="text-align:right;"> 514.78 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 0.00 </td> <td style="text-align:right;"> 15500.00 </td> <td style="text-align:left;"> ▇▁▁▁▁ </td> </tr> <tr> <td style="text-align:left;"> Mo_Sold </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 6.12 </td> <td style="text-align:right;"> 2.63 </td> <td style="text-align:right;"> 1.00 </td> <td style="text-align:right;"> 4.00 </td> <td style="text-align:right;"> 6.00 </td> <td style="text-align:right;"> 7.25 </td> <td style="text-align:right;"> 12.00 </td> <td style="text-align:left;"> ▃▆▇▃▃ </td> </tr> <tr> <td style="text-align:left;"> Year_Sold </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 2007.87 </td> <td style="text-align:right;"> 1.31 </td> <td style="text-align:right;"> 2006.00 </td> <td style="text-align:right;"> 2007.00 </td> <td style="text-align:right;"> 2008.00 </td> <td style="text-align:right;"> 2009.00 </td> <td style="text-align:right;"> 2010.00 </td> <td style="text-align:left;"> ▆▇▇▇▅ </td> </tr> <tr> <td style="text-align:left;"> Sale_Price </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 175.33 </td> <td style="text-align:right;"> 70.01 </td> <td style="text-align:right;"> 35.00 </td> <td style="text-align:right;"> 129.50 </td> <td style="text-align:right;"> 158.75 </td> <td style="text-align:right;"> 206.22 </td> <td style="text-align:right;"> 625.00 </td> <td style="text-align:left;"> ▇▇▂▁▁ </td> </tr> <tr> <td style="text-align:left;"> Longitude </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> -93.64 </td> <td style="text-align:right;"> 0.03 </td> <td style="text-align:right;"> -93.69 </td> <td style="text-align:right;"> -93.66 </td> <td style="text-align:right;"> -93.64 </td> <td style="text-align:right;"> -93.62 </td> <td style="text-align:right;"> -93.58 </td> <td style="text-align:left;"> ▅▅▇▇▁ </td> </tr> <tr> <td style="text-align:left;"> Latitude </td> <td style="text-align:right;"> 0 </td> <td style="text-align:right;"> 1 </td> <td style="text-align:right;"> 42.03 </td> <td style="text-align:right;"> 0.02 </td> <td style="text-align:right;"> 41.99 </td> <td style="text-align:right;"> 42.02 </td> <td style="text-align:right;"> 42.03 </td> <td style="text-align:right;"> 42.05 </td> <td style="text-align:right;"> 42.06 </td> <td style="text-align:left;"> ▂▂▇▇▆ </td> </tr> </tbody> </table> ] --- ## Exploratory Analysis (EDA): Make lots of plots. ```r ames_home_sales %>% ggplot(aes(x = Gr_Liv_Area, y = Sale_Price, color = Fireplaces > 0)) + geom_point(alpha = .2) + facet_wrap(vars(Bldg_Type)) ``` <img src="slides03prediction_files/figure-html/eda-living-area-1.png" width="70%" style="display: block; margin: auto;" /> --- ```r ames_home_sales %>% select(Sale_Price, Gr_Liv_Area, Lot_Area, Full_Bath, Half_Bath, Fireplaces) %>% pivot_longer(-c(Sale_Price, Fireplaces)) %>% ggplot(aes(x = value, y = Sale_Price, color = Fireplaces > 0)) + geom_point(size = .5, alpha = .25) + facet_wrap(vars(name), scales = "free") + theme_bw() ``` <img src="slides03prediction_files/figure-html/eda-several-predictors-1.png" width="70%" style="display: block; margin: auto;" /> --- # Specifying a Model --- ## Example (without validation) Specify the model: ```r *my_model_spec <- parsnip::decision_tree(mode = "regression") ``` Train it ("fit") on data: ```r my_trained_model <- my_model_spec %>% fit(Sale_Price ~ Gr_Liv_Area, data = ames_home_sales) ``` --- ## Example (without validation) Predict on new data: ```r made_up_data <- tibble(Gr_Liv_Area = c(1000, 2000), Sale_Price = c(239, 185)) my_trained_model %>% predict(made_up_data) %>% bind_cols(made_up_data) # Put back the original data ``` ``` # A tibble: 2 × 3 .pred Gr_Liv_Area Sale_Price <dbl> <dbl> <dbl> 1 119. 1000 239 2 228. 2000 185 ``` Evaluate on all data: ```r predictions <- my_trained_model %>% predict(ames_home_sales) %>% bind_cols(ames_home_sales) predictions %>% metrics(truth = Sale_Price, estimate = .pred) ``` ``` # A tibble: 3 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 mae standard 34.5 2 mape standard 21.4 3 rsq_trad standard 0.528 ``` --- <img src="https://imgs.xkcd.com/comics/electoral_precedent_2x.png" width="100%" style="display: block; margin: auto; max-height: inherit;"> .floating-source[ <https://xkcd.com/1122/> ] --- ## Example, with validation --- 1. Hold out some data to use for validation: ```r set.seed(10) ames_split <- initial_split(ames_home_sales, prop = 3/4) ames_train <- training(ames_split) ames_test <- testing(ames_split) glue("Using {nrow(ames_train)} sales to train, {nrow(ames_test)} to test") ``` ``` Using 1809 sales to train, 603 to test ``` --- 1. Hold out some data to use for validation. 2. Specify the model to use. 3. Train the model **on the training set**: ```r my_trained_model <- my_model_spec %>% * fit(Sale_Price ~ Gr_Liv_Area, * data = ames_train) ``` --- 1. Hold out some data to use for validation. 2. Specify the model to use. 3. Train the model **on the training set** 4. Evaluate on training set (optional): .pull-left[ ```r train_predictions <- my_trained_model %>% predict(ames_train) %>% bind_cols(ames_train) # Put back the original data train_predictions ``` ``` # A tibble: 1,809 × 75 .pred MS_SubClass MS_Zo…¹ Lot_F…² Lot_A…³ Street Alley Lot_S…⁴ <dbl> <fct> <fct> <dbl> <int> <fct> <fct> <fct> 1 118. One_Story_1… Reside… 65 7150 Pave No_A… Regular 2 155. One_Story_1… Reside… 0 10710 Pave No_A… Regular 3 194. One_Story_P… Reside… 0 3621 Pave No_A… Regular 4 118. One_Story_1… Reside… 50 5000 Pave No_A… Regular 5 297. Two_Story_1… Reside… 105 13693 Pave No_A… Regular 6 118. One_Story_1… Reside… 109 8724 Pave No_A… Regular # … with 1,803 more rows, 67 more variables: Land_Contour <fct>, # Utilities <fct>, Lot_Config <fct>, Land_Slope <fct>, # Neighborhood <fct>, Condition_1 <fct>, Condition_2 <fct>, # Bldg_Type <fct>, House_Style <fct>, Overall_Cond <fct>, # Year_Built <int>, Year_Remod_Add <int>, Roof_Style <fct>, # Roof_Matl <fct>, Exterior_1st <fct>, Exterior_2nd <fct>, # Mas_Vnr_Type <fct>, Mas_Vnr_Area <dbl>, Exter_Cond <fct>, … ``` ] .pull-right[ ```r train_predictions %>% metrics(truth = Sale_Price, estimate = .pred) ``` ``` # A tibble: 3 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 mae standard 35.1 2 mape standard 22.0 3 rsq_trad standard 0.513 ``` ] --- 1. Hold out some data to use for validation 2. Specify the model to use. 3. Train the model **on the training set** 4. Evaluate on training set (optional) 5. Evaluate on **test set**: ```r my_trained_model %>% predict(ames_test) %>% bind_cols(ames_test) %>% metrics(truth = Sale_Price, estimate = .pred) ``` ``` # A tibble: 3 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 mae standard 33.2 2 mape standard 19.6 3 rsq_trad standard 0.563 ``` --- > What's the optimal ratio of train to test? .question[ What's the trade-off? What happens if train is too small? If test is too small? ] --- ## Many models, same interface ```r trained_linear_model <- * parsnip::linear_reg() %>% fit(Sale_Price ~ Gr_Liv_Area, data = ames_train) trained_linear_model %>% predict(ames_test) %>% bind_cols(ames_test) %>% metrics(truth = Sale_Price, estimate = .pred) ``` ``` # A tibble: 3 × 3 .metric .estimator .estimate <chr> <chr> <dbl> 1 mae standard 31.0 2 mape standard 18.0 3 rsq_trad standard 0.586 ``` --- ## Types of models * Linear models * ordinary least-squares (OLS) * Lasso, Ridge, etc.: penalize large coefficients * Generalized Linear Models: outputs get transformed * Logistic Regression (also Support Vector Machine): transform output to *score* for each class * Decision Lists and Trees * extension: Random Forests * Neural Networks: layered combinations of the above * many, many more --- ## Which variables mean what? The *formula interface*: * `y ~ x` * predict *y* using *x*. `Sale_Price ~ Gr_Liv_Area` * `y ~ x1 + x2 + x3` * predict *y* using *x1* and *x2* and *x3* * `Sale_Price ~ Gr_Liv_Area + Lot_Area + Full_Bath` Don't get confused: they "forgot" the coefficients! A fitted linear model will actually look like: <br> <br> Sale_Price = .red[c1] \* Gr_Liv_Area + .red[c2] \* Lot_Area + .red[c3] \* Full_Bath + intercept <br> But this works for specifying even models that aren't linear.