---
title: "Data Cleansing Demo"
author: "Keith VanderLinden"
date: "Spring, 2022"
output: html_document
---

```{r include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

This document demonstrates data cleansing in the context of a pet-ownership dataset.

```{r warning = FALSE, message = FALSE}
library(tidyverse)
```

## Dataset: Dog Owners

We&rsquo;ll work with the following (contrived) dataset of dog ownership information. This dataset is inspired by an example from [DSBox](https://datasciencebox.org).

```{r}
dogs <- read_csv("data/dog_lovers.csv")
dogs
```

## Computing the Average Number of Dogs

We&rsquo;d like to compute the average number of dogs for the owners listed in the dataset.

```{r}
dogs %>%
  summarize(mean_dog_count = mean(number_of_dogs)) 
```

Use the error or warning message to debug the problem. Somehow, the `number_of_dogs` field is non-numeric. Looking through the data, we see that there are some `NA` values, presumably for people who didn&rsquo;t enter a value for this field. We can fix those by ignoring `NA` values when computing the average. 


```{r}
dogs %>%
  summarize(mean_dog_count = mean(number_of_dogs, na.rm = TRUE)) 
```

We still get an `NA` result, and on going back to the original data and see all sorts of odd values for the `number_of_dogs` field. It appears that we&rsquo;ll need to clean up the values for Tony and Ginger as well.

```{r}
dogs_cleansed <- dogs %>%
  mutate(
    number_of_dogs = case_when(                                
      name == "Tony Garcia"         ~ 2,
      name == "Ginger Clark"        ~ 2,
      TRUE                          ~ as.numeric(number_of_dogs)
    )
  )

dogs_cleansed %>%
  summarize(mean_dog_count = mean(number_of_dogs, na.rm = TRUE)) 
```  

Finally, we get a number, but it&rsquo;s rather high. On yet further investigation, we se that the entries for Terrance and Travis are probably out-of-range. We can fix that using a function that sets outliers to `NA` (a version of `NA` designed for double values).

```{r}
remove_outliers <- function(x, 
                            min_limit=0, 
                            max_limit=10) {
  case_when(
    x < min_limit ~ NA_real_,
    x > max_limit ~ NA_real_,
    TRUE          ~ x
  )
}
```

This function is a (self-documenting) abstraction that we can use to cleanse outliers.

```{r}
dogs_cleansed <- dogs_cleansed %>%
  mutate(
    count = remove_outliers(
      as.numeric(number_of_dogs)
      )
    )

dogs_cleansed %>%
  select(name, count, number_of_dogs) %>%
  arrange(desc(is.na(count)))
```

Finally, we can get a reasonable average value.

```{r}
dogs_cleansed %>%
  summarize(mean_dog_count = mean(count, na.rm = TRUE)) 
``` 

The moral of this story is that you can&rsquo;t assume that data entered freely by users is *clean*.
