---
title: "Lab 12.1 - Top IMDB TV Shows"
author: "Author Goes Here"
date: "Semester Goes Here"
output:
  pdf_document:
    latex_engine: xelatex
---

```{r warning=FALSE, message=FALSE}
library(tidyverse)
library(rvest)
library(robotstxt)
```

The goal of this exercise is to practice scraping text data from a website. It will do an analysis similar to the analysis of IMDB movies demoed in the class session, but this time on IMDB television shows.

## Scraping the Webpage

🚧
Before scraping the IMDB website, check to make sure that scraping is permitted.

🚧
Retrieve the IMDB TV shows page using rvest and save it as `page`. Here's the code for the movies; change it to use this URL for TV shows: `https://www.imdb.com/chart/toptv`

```{r warning=FALSE, message=FALSE}
page <- read_html("https://www.imdb.com/chart/top/")
page
```

## Parsing Useful Information

We now wade through the hierarchical HTML structure to find the values of interest to us, namely the year, rating, and title for each of the 100 top TV shows. We'll do these each, in sequence, finding and cleaning up the values as needed.

🚧
Get the ratings first, they're probably the easiest. If we inspect the rating value on the webpage, we see that it is marked using the `ratingGroup--imdb-rating`` CSS class. We use this to find it in the HTML structure. You can see that the text is messed up, so use string commands to fix it. And remember to convert it to a numeric value.

```{r warning=FALSE, message=FALSE}
ratings <- page %>%
  html_nodes(".ratingGroup--imdb-rating") %>%
  html_text() %>%
  as.numeric()
ratings[1:10]
```

🚧
Get the years next. The year for TV shows is is stored using the class `cli-title-metadata` in the first four characters. You can now clean up these year strings and convert them to numbers. 

```{r warning=FALSE, message=FALSE}
years <- page %>%
  html_nodes(".cli-title-metadata") %>%
  html_text() %>%
  substr(0, 4) %>%
  as.numeric()
years[1:10]
```

🚧
Finally, get the titles. Use the HTML structure inspector to find the class name for the title values, and then clean up the value. Here, the titles are buried a bit under `cli-children` and then `ipc-title__text`, and they have the rank number in the title, which should be removed.

```{r warning=FALSE, message=FALSE}
titles <- page %>%
  html_nodes(".cli-children .ipc-title__text") %>%
  html_text() %>%
  str_remove("^\\d*. ")
titles[1:10]
```

🚧
Finally, create a dataframe that contains the data that we've collected and add a `rank` value. You'll need to sort these values first, they're not sorted as the movies were.

You can focus on the top 100 shows; as of Fall 2023, there&rsquo;s a show title at position 116 with non-standard character codes out there. 

```{r warning=FALSE, message=FALSE}
tvshows <- tibble(
  # Put variables here.
) 
# Wrangle the dataset here.
tvshows
```

## Analyzing the Data

🚧
Find out which, if any, of the top TV shows are from your birth year.

🚧
Visualize the average yearly rating as we did in class for the movies.

🚧
Count the TV shows in the top 100 per decade. You can visualize this using either a table or a plot.
