---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## Data Engineering: DataFlow

Keith VanderLinden<br>
Calvin University

::: {.notes}
We focus here on data management from the systems perspective.
:::

## Data Stacks

Modern *data stacks* are designed to handle large datasets and to support the *flow* of that data through an ML system. This generally comprises:

- <span style="color:grey">Data Storage (i.e., sources, formats, models, engines)</span>
- Data Processing
- Data<em>flow</em>

::: {.notes}
- Notes:
    - Data Storage: where/how data is stored - We've covered this already (nb. data engine = DBMS).
    - Data processing: how data is transformed/transacted (aka *compute*)
    - Dataflow: how data is moved/transported from one place to another
- Storing datasets on our local machines is fine for small projects in an academic setting, but it's not a scalable solution for large projects in a production setting.
:::

## Data Processing

Traditional systems distinguish between two types of data processing:

- *Transactional* processing (*OLTP*)
- *Analytical* processing (*OLAP*)

DMLS argues that this distinction is less relevant today.

::: {.notes}
- OLTP: Online Transaction Processing
    - Transaction Models
        - ACID (SQL), BASE (noSQL), CAP
        - row-major
        - data-write heavy
        - real-time
- OLAP: Online Analytical Processing
    - Aggregate analysis
        - column-major
        - data-read heavy
        - batch
- These are outdated somewhat, because:
    - Current DBMSs often support both transactional and analytical processing.
    - We've begun, instead, to decouple storage and *compute*.

References

- <https://aws.amazon.com/compare/the-difference-between-olap-and-oltp/>
:::

## Transaction Models

When performing transactions on data, we set different goals depending on the context.

- *ACID*: Atomic, Consistent, Isolated, Durable
- *BASE*: Basically Available, Soft state, Eventual consistency

These transaction principles are related to [Brewer's CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem).

::: {.notes}
- ACID
    - Typical of traditional SQL databases
    - Properties:
        - *Atomic* - Transactions run all-or-nothing.
        - *Consistent* - Data is always internally valid.
        - *Isolated* - Transactions don't interfere with each other.
        - *Durable* - Data is saved (typical of all DBMSs).
    - Example use cases:
        - Atomic/Isolated - Bank transactions (e.g., [Chase](https://en.wikipedia.org/wiki/Chase_Bank))
        - Consistency - Flight bookings (e.g., [Sabre](https://en.wikipedia.org/wiki/Sabre_(travel_reservation_system)))
- BASE
    - Typical of scaled-up modern databases, often noSQL.
    - Properties:
        - *Basically Available* - The system is always available, but may not be consistent.
        - *Soft state* - The system's state may change over time.
        - *Eventual consistency* - The system will eventually become consistent.
    - Example use cases
        - Social media (e.g., [Facebook](https://en.wikipedia.org/wiki/Facebook))
- CAP: Consistency, Availability, Partition tolerance
    - aka, *Brewer's theorem*
    - C, A, P, choose any two.
    - Discuss the figure on the linked Wikipedia page.

References

- Has basic diagram: <https://en.wikipedia.org/wiki/CAP_theorem>
    - CA = ACID (MySQL, ...)
    - AP = BASE (MongoDB, ...)
- Interesting CAP chart: <https://www.linkedin.com/pulse/acid-cap-base-unveiling-database-consistency-omar-mahmoud/>
- CAP vs BASE: <https://www.baeldung.com/cs/db-base-meaning-cap>
- E. Brewer discusses his CAP theorem: <https://people.eecs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf>
:::

## Data Processing Tasks

Preparing data for analysis requires three traditional data processing tasks:

- Extract
- Transform
- Load

These task flows can be ordered as appropriate for the context: *ETL* vs *ELT*.

::: {.notes}
- Tasks
    - Extract from (raw) data sources.
    - Transform/Wrangle the data into a usable format.
    - Load the wrangled data into a database or data warehouse.
- These tasks can be ordered as appropriate.
- ETL - DMLS Figure 3-7 - traditional
    - Forces you to structure all the data before using it for ML
    - Assumes that the transform step should be batched.
    - MwML mostly ignores this.
- ELT - [MwML Data Engineering Intuition figure](https://madewithml.com/courses/mlops/data-engineering/#intuition) - more modern
    - DMLS says that this is too much processing (raw -> dataset) for big data
    - MwML calls it [best practice](https://madewithml.com/courses/mlops/data-engineering/#best-practices).
- C. Huyen observed that support for hybrid approaches is becoming more common.

References

- [MwML's view of ETL/ELT](https://madewithml.com/static/images/mlops/data_stack/data.png)
- DMLS p. 70-72
- <https://www.ibm.com/blog/elt-vs-etl-whats-the-difference/>
:::

## Dataflow

In production systems, data generally *flows* between separate processes.

- DBMS-based flow
- *Service*-based flow
- *Real-time transport*-based flow

::: {.notes}
- "flows" = "...is transported/passed between..."
- C. Huyen lists these three; there are likely other approaches and combinations that could be used.
    - *DBMS-based flow* - Data is stored/processed in a centralized DBMS hub.
    - *Service-based flow* - Data is served up synchronously via Internet/Web-based services.
    - *Real-time transport-based flow* - Data is requested/transferred asynchronously via a real-time broker.
- Review the *data stack* for this course, placing this module in perspective. We'll be:
    - Pulling data to/from a Data Lake (GoogleDrive) and a DBMS (MongoDB).
    - Serving our ML results up as a REST service.
    - Passing on real-time brokers and the other tools used in MwML.
:::
