---
title: "Textual Data<br>Web Scraping Demo"
author: "Keith VanderLinden<br>Calvin University"
output:
  xaringan::moon_reader:
    nature:
      titleSlideClass: ["left", "top"]
      highlightLines: true
      ratio: "16:9"     
---

```{r child = "../../../scripts/slide-setup.Rmd"}
```

```{r echo=FALSE, warning=FALSE, message=FALSE}
options(dplyr.print_max = 12)
library(tidyverse)
```

# Checking Access Restrictions

It&rsquo;s proper to check that scraping data is permitted.

```{r}
library(robotstxt)
paths_allowed("http://www.imdb.com")
paths_allowed("http://www.facebook.com")
paths_allowed("https://www.gutenberg.org")
paths_allowed("https://www.gutenberg.org/ebooks/search")
```

???
- Don't assume that all Web text is free and open. 
- Because computers can attend to and remember everything, it's not clear that we can simply assume that all public web text is truly open to our analysis.

---
# Scraping the HTML Data

We can then scrape the HTML from the [IMDB Top Chart](https://www.imdb.com/chart/top/).

```{r}
library(rvest)

page <- read_html("https://www.imdb.com/chart/top/")
page
```

???
- This will be an ETL example. Here's the E(xtract).
- Note how ugly the raw HTML is in this case.

---
# Accessing the Relevant HTML Elements

.pull-left[
```{r imdb-1, results="hide", message = FALSE, warning = FALSE}
ratings <- page %>%
  html_nodes("strong") %>%
  html_text() %>%
  as.numeric()
ratings[1:5]
```

```{r imdb-2, results="hide", message = FALSE, warning = FALSE}
years <- page %>%
  html_nodes(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") %>%
  str_remove("\\)") %>%
  as.numeric()
years[1:5]
```

```{r imdb-3, results="hide", message = FALSE, warning = FALSE}
titles <- page %>%
  html_nodes(".titleColumn a") %>%
  html_text()
titles[1:5]
```
]
.pull-right[
```{r ref.label="imdb-1", echo=FALSE, message=FALSE, warning=FALSE, output.lines=10}
```
<br><br><br><br>
```{r ref.label="imdb-2", echo=FALSE, message=FALSE, warning=FALSE, output.lines=10}
```
<br><br><br><br><br><br>
```{r ref.label="imdb-3", echo=FALSE, message=FALSE, warning=FALSE, output.lines=10}
```
]

???
- This will be an ETL example. Here's the T(ransform).
- Show how to find all these HTML elements using the Firefox DOM inspector.

---
# Creating a Tidy Dataframe

.pull-left[
```{r}
imdb_top_250 <- tibble(
  title = titles, 
  year = years, 
  rating = ratings
  )

imdb_top_250 <- imdb_top_250 %>%
  mutate(rank = 1:nrow(imdb_top_250))
```
]
.pull-right[
```{r}
imdb_top_250
```
]

???
- This will be an ETL example. Here's the L(oad).

---
# Visualizing the Movie Ratings Over Time

.pull-left[
```{r imdb-5, fig.show="hide", message=FALSE, warning=FALSE}
imdb_top_250 %>% 
  group_by(year) %>%
  summarise(avg_score = mean(rating)) %>%
  ggplot() +
  aes(x = year,
      y = avg_score, 
  ) +
  geom_point() +
  geom_smooth() +
  labs(title = "Average Movie Ratings",
       x = "Movie Year", 
       y = "Average Movie Rating",
  )
```
]
.pull-right[
```{r ref.label="imdb-5", echo=FALSE, message=FALSE, warning=FALSE, output.lines=10}
```
]

???

