---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## Data Engineering

Keith VanderLinden<br>
Calvin University

::: {.notes}
In this module, we begin a discussion of data engineering.
:::

## Big Data

The Vs of *big data*:

- Volume
- Variety
- Velocity
- Veracity
- Value

*Data engineering* is the process of designing, building, and managing the infrastructure for big data.

::: {.notes}
- Big data is a term that refers to data that is so large, fast, or complex that it's difficult or impossible to process using traditional methods.
    - This is an older term that's more commonly addressed by so-called *data engineering*.
    - This course focuses on using more modern methods to deal with the V's. E.g., we're dealing with volume and, to some degree, variety, this week.

References:

- <https://en.wikipedia.org/wiki/Big_data>
- <https://madewithml.com/courses/mlops/data-engineering/>
:::

## Data Formats

Data are stored in a variety of formats.

- CSV &mdash; Comma Separated Values
- TSV &mdash; Tab Separated Values
- JSON &mdash; JavaScript Object Notation
- ZIP &mdash; Compressed Archive
- Pickle &mdash; Python-specific binary format

::: {.notes}
- We'll see these formats in this course.
    - *CSV* is a 2-dimensional, textual format that matches closely with spreadsheets and is human-readable. It's very common but problematic (see Huyen's discussion, p. 57).
    - *TSV* is like CSV, but uses tabs instead of commas. With respect to CSV, TSV is:
        - More robust because TABs rarely appear in textual data, whereas commas often do.
        - More efficient because TSV parsers don't have to escape commas.
        - Less common (than CSV), and thus not the default read format for many tools.
    - *JSON* is a hierarchical, textual format that's more flexible than CSV/TSV.
        - See the example in the text or in SLO (in the demo).
    - *ZIP*/*Gzip* is a compressed format that's more space efficient, but isn't human-readable.
    - *Pickle* is a binary format that's more space efficient, but isn't human-readable.
- The SLO dataset started in JSON, was filtered down to CSV/TSV.

References

- <https://stackoverflow.com/questions/11130120/choosing-between-tsv-and-csv>
:::

## Row vs Column Storage

The distinction between row-major vs column-major format can make a difference in performance.

::: {.notes}
- CSV/TSV are row-major formats; *Parquet* is a column-major format.
- Attend to this distinction, it can matter. Huyen's summary:
    - Row-major is better write-heavy applications (e.g., real-time data collection).
    - Column-major is better for column-based reads (e.g., building a model based on a sub-set of the features).
:::

## Versioning

Supporting reproducibility requires that we maintain version histories for everything.

- Code
- Configuration
- Data
- Models

::: {.notes}
- Mohandas points out that we should version all four of these types of artifacts.
    - We've already done code and config.
    - We'll do data in this module.
    - We'll do models when we get to model engineering.
- Give use cases for where "losing" track of any of these things leads to problems (see C.Huyen's stories in the containers chapter).

References

- <https://dvc.org/#use-cases> (see the 5-minute video for ideas)
:::

## Data Storage

Data storage systems are classified into different types.

![<small>image from [MwML Data Engineering](https://madewithml.com/courses/mlops/data-engineering/#data-systems)</small>](https://madewithml.com/static/images/mlops/data_stack/systems.png){width=65%}

::: {.notes}
- Data storage types
    - Data *lakes* are typically used for *unstructured* data.
        - In this module, we'll use a data lake to store versions our data.
    - *Databases* are typically used for *structured* data.
        - In the next module, we'll use a non-relational DBMS to store data.
        - And you'll definitely see relational DBMSs in other courses (e.g., DATA 654).
    - Data *warehouses* are typically used for aggregated, structured data. We won't cover this in this course.
    - Data *lake-houses*, not to be covered in this course, are a new concept that combines the:
        - structure and ACID properties of data warehouses
        - low-cost, more flexible objects of data lakes.
- MwML addresses the *structured*/*semi-structured*/*unstructured* distinction in this module; DMLS will address it in the next module.

References:

- See DMLS p. 67 & MwML <https://madewithml.com/courses/mlops/data-stack/#data-systems>
:::
