Data Engineering: DataFlow

Keith VanderLinden
Calvin University

We focus here on data management from the systems perspective.

Data Stacks

Modern data stacks are designed to handle large datasets and to support the flow of that data through an ML system. This generally comprises:

• Data Storage (i.e., sources, formats, models, engines)
• Data Processing
• Dataflow

• Notes:
  • Data Storage: where/how data is stored - We’ve covered this already (nb. data engine = DBMS).
  • Data processing: how data is transformed/transacted (aka compute)
  • Dataflow: how data is moved/transported from one place to another
• Storing datasets on our local machines is fine for small projects in an academic setting, but it’s not a scalable solution for large projects in a production setting.

Data Processing

Traditional systems distinguish between two types of data processing:

• Transactional processing (OLTP)
• Analytical processing (OLAP)

DMLS argues that this distinction is less relevant today.

• OLTP: Online Transaction Processing
  • Transaction Models
    • ACID (SQL), BASE (noSQL), CAP
    • row-major
    • data-write heavy
    • real-time
• OLAP: Online Analytical Processing
  • Aggregate analysis
    • column-major
    • data-read heavy
    • batch
• These are outdated somewhat, because:
  • Current DBMSs often support both transactional and analytical processing.
  • We’ve begun, instead, to decouple storage and compute.

References

• https://aws.amazon.com/compare/the-difference-between-olap-and-oltp/

Transaction Models

When performing transactions on data, we set different goals depending on the context.

• ACID: Atomic, Consistent, Isolated, Durable
• BASE: Basically Available, Soft state, Eventual consistency

These transaction principles are related to Brewer’s CAP theorem.

• ACID
  • Typical of traditional SQL databases
  • Properties:
    • Atomic - Transactions run all-or-nothing.
    • Consistent - Data is always internally valid.
    • Isolated - Transactions don’t interfere with each other.
    • Durable - Data is saved (typical of all DBMSs).
  • Example use cases:
    • Atomic/Isolated - Bank transactions (e.g., Chase)
    • Consistency - Flight bookings (e.g., Sabre)
• BASE
  • Typical of scaled-up modern databases, often noSQL.
  • Properties:
    • Basically Available - The system is always available, but may not be consistent.
    • Soft state - The system’s state may change over time.
    • Eventual consistency - The system will eventually become consistent.
  • Example use cases
    • Social media (e.g., Facebook)
• CAP: Consistency, Availability, Partition tolerance
  • aka, Brewer’s theorem
  • C, A, P, choose any two.
  • Discuss the figure on the linked Wikipedia page.

References

• Has basic diagram: https://en.wikipedia.org/wiki/CAP_theorem
  • CA = ACID (MySQL, …)
  • AP = BASE (MongoDB, …)
• Interesting CAP chart: https://www.linkedin.com/pulse/acid-cap-base-unveiling-database-consistency-omar-mahmoud/
• CAP vs BASE: https://www.baeldung.com/cs/db-base-meaning-cap
• E. Brewer discusses his CAP theorem: https://people.eecs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf

Data Processing Tasks

Preparing data for analysis requires three traditional data processing tasks:

• Extract
• Transform
• Load

These task flows can be ordered as appropriate for the context: ETL vs ELT.

• Tasks
  • Extract from (raw) data sources.
  • Transform/Wrangle the data into a usable format.
  • Load the wrangled data into a database or data warehouse.
• These tasks can be ordered as appropriate.
• ETL - DMLS Figure 3-7 - traditional
  • Forces you to structure all the data before using it for ML
  • Assumes that the transform step should be batched.
  • MwML mostly ignores this.
• ELT - MwML Data Engineering Intuition figure - more modern
  • DMLS says that this is too much processing (raw -> dataset) for big data
  • MwML calls it best practice.
• C. Huyen observed that support for hybrid approaches is becoming more common.

References

• MwML’s view of ETL/ELT
• DMLS p. 70-72
• https://www.ibm.com/blog/elt-vs-etl-whats-the-difference/

Dataflow

In production systems, data generally flows between separate processes.

• DBMS-based flow
• Service-based flow
• Real-time transport-based flow

• “flows” = “…is transported/passed between…”
• C. Huyen lists these three; there are likely other approaches and combinations that could be used.
  • DBMS-based flow - Data is stored/processed in a centralized DBMS hub.
  • Service-based flow - Data is served up synchronously via Internet/Web-based services.
  • Real-time transport-based flow - Data is requested/transferred asynchronously via a real-time broker.
• Review the data stack for this course, placing this module in perspective. We’ll be:
  • Pulling data to/from a Data Lake (GoogleDrive) and a DBMS (MongoDB).
  • Serving our ML results up as a REST service.
  • Passing on real-time brokers and the other tools used in MwML.