Data Formats and Versioning Tools

Keith VanderLinden
Calvin University

• In this course, we’ll zero in on simple, free resources for data storage, formats and versioning.
• Goal Demo the diversity of file formats and the need for versioning.
  • Demo everything using the SLO project as an example, comparing and contrasting what I did back then (with all those separate data/coding-sets , C:\Users\kvlinden\OneDrive - Calvin University\Documents\projects\slo) with what we’re doing in this course (see the stance-only materials in data562-slo).

Demos

• Data formats
• Versioning using DVC
  • Code/Notebooks and Configuration
  • Data (and models)

The DVC demo is based on E. O’Brian’s video intro found here: Data Versioning.

• Data formats
  • Show the relative sizes of these files - ls -lh data
  • JSON (e.g., SLO data/dataset.csv, too big for VSCode)
    • head data/dataset.json -n 1
    • This was dumped from the (now defunct) Twitter firehose feed collected at CSIRO.
  • CSV/TSV (e.g., SLO dataset.csv, also too big for VSCode)
    • head data/dataset.csv -n 2
    • This is a filtered version of the raw JSON file.
  • TXT (e.g., SLO dataset_norm.txt)
    • head data/dataset.csv -n 2
    • This is raw text for training the embeddings.
  • PKL (e.g., SLO data/model.pkl)
    • This is the trained SVM baseline model store in binary format.
• Versioning
  • Show that https://github.com/CalvinData/data562-slo has:
    • code (notebook) and devcontainer configuration files.
    • DVC files rather than full dataset files.
      • We don’t check datasets (or model) files into GitHub (in general). E.g.,
        • The SLO dataset is 3.5G.
        • The IMDB has 10 million movie titles. We could put that in GitHub, using Git history to log changes, but that’s too large for GitHub (1G limit?) and that’s not what GitHub is for.
    • The DVC files are metafile “pointers” to our data files on our GoogleDrive data lake, here: https://drive.google.com/drive/u/1/folders/1y3CqOpCuPwl7dtV9iuP3tUh0wTkl1Jrs
• Demo DVC’s features (following E. O’Brien’s video).
  • Datafile metafiles
  • Maintenance of the .gitignore file.

Snapshot

See the live demo.

Link a snapshot of the final state of the demo here, with the GitHub-cloud-based VSCode editor dvc-pulling the large SLO dataset files.

OAuth, DVC, and GoogleDrive

There is an issue with OAuth authentication using DVC and GoogleDrive. We’ll demo a work-around that we’ve found.

image from Wikipedia: OAuth

• So far, I’ve demoed the SLO system in WSL2, but I must use the devcontainer to run and deploy it because it was hard to configure the local system properly (q.v., the complexity of .devcontainer).
• Using DVC on GoogleDrive as we’ve done creates issues. GoogleDrive is really a user-oriented system, not an api-oriented system, so there are issues using it as a data lake.
• Demo Ken’s OAuth hack, which is only needed in a container and only works in a DevContainer/Codespace.

References

• https://en.wikipedia.org/wiki/OAuth