Data Engineering

Keith VanderLinden
Calvin University

In this module, we begin a discussion of data engineering.

Big Data

The Vs of big data:

• Volume
• Variety
• Velocity
• Veracity
• Value

Data engineering is the process of designing, building, and managing the infrastructure for big data.

• Big data is a term that refers to data that is so large, fast, or complex that it’s difficult or impossible to process using traditional methods.
  • This is an older term that’s more commonly addressed by so-called data engineering.
  • This course focuses on using more modern methods to deal with the V’s. E.g., we’re dealing with volume and, to some degree, variety, this week.

References:

• https://en.wikipedia.org/wiki/Big_data
• https://madewithml.com/courses/mlops/data-engineering/

Data Formats

Data are stored in a variety of formats.

• CSV — Comma Separated Values
• TSV — Tab Separated Values
• JSON — JavaScript Object Notation
• ZIP — Compressed Archive
• Pickle — Python-specific binary format

• We’ll see these formats in this course.
  • CSV is a 2-dimensional, textual format that matches closely with spreadsheets and is human-readable. It’s very common but problematic (see Huyen’s discussion, p. 57).
  • TSV is like CSV, but uses tabs instead of commas. With respect to CSV, TSV is:
    • More robust because TABs rarely appear in textual data, whereas commas often do.
    • More efficient because TSV parsers don’t have to escape commas.
    • Less common (than CSV), and thus not the default read format for many tools.
  • JSON is a hierarchical, textual format that’s more flexible than CSV/TSV.
    • See the example in the text or in SLO (in the demo).
  • ZIP/Gzip is a compressed format that’s more space efficient, but isn’t human-readable.
  • Pickle is a binary format that’s more space efficient, but isn’t human-readable.
• The SLO dataset started in JSON, was filtered down to CSV/TSV.

References

• https://stackoverflow.com/questions/11130120/choosing-between-tsv-and-csv

Row vs Column Storage

The distinction between row-major vs column-major format can make a difference in performance.

• CSV/TSV are row-major formats; Parquet is a column-major format.
• Attend to this distinction, it can matter. Huyen’s summary:
  • Row-major is better write-heavy applications (e.g., real-time data collection).
  • Column-major is better for column-based reads (e.g., building a model based on a sub-set of the features).

Versioning

Supporting reproducibility requires that we maintain version histories for everything.

• Code
• Configuration
• Data
• Models

• Mohandas points out that we should version all four of these types of artifacts.
  • We’ve already done code and config.
  • We’ll do data in this module.
  • We’ll do models when we get to model engineering.
• Give use cases for where “losing” track of any of these things leads to problems (see C.Huyen’s stories in the containers chapter).

References

• https://dvc.org/#use-cases (see the 5-minute video for ideas)

Data Storage

Data storage systems are classified into different types.

image from MwML Data Engineering

• Data storage types
  • Data lakes are typically used for unstructured data.
    • In this module, we’ll use a data lake to store versions our data.
  • Databases are typically used for structured data.
    • In the next module, we’ll use a non-relational DBMS to store data.
    • And you’ll definitely see relational DBMSs in other courses (e.g., DATA 654).
  • Data warehouses are typically used for aggregated, structured data. We won’t cover this in this course.
  • Data lake-houses, not to be covered in this course, are a new concept that combines the:
    • structure and ACID properties of data warehouses
    • low-cost, more flexible objects of data lakes.
• MwML addresses the structured/semi-structured/unstructured distinction in this module; DMLS will address it in the next module.

References:

• See DMLS p. 67 & MwML https://madewithml.com/courses/mlops/data-stack/#data-systems