Training Data Demos

Keith VanderLinden
Calvin University

Labeling in SLO

• SLO labeling
• Label Studio

• Run the SLO demo in a local VSCode connected to the GitHub running codespace. (I did this by opening the codespace in GitHub and choosing something like “open in local VSCode”.)
  • Show the data/coding/ datasets.
  • Explain how we just used spreadsheets, with the hyperlink to the tweet in context. The context ended up really mattering for SLO, as, indeed, it might for many applications.
  • Show my (local) kappa-computation spreadsheet.
    • Explain the neutral/non-neutral issue we had.
    • Note the low kappa score in this sheet. We ended up publishing at around 0.67.
  • Talk through the SLO auto-labeling (see src/autocoding_preprocessor.py).
    • Show the three rule patterns: PTN_for, PTN_against, PTN_neutral.
  • We worked hard to balance (rare) positive examples with (common) negative examples. Mostly, we used all the positives and sampled the negative tweets for the training set.
    • This might be classified as probabilistic quota sampling.
• Run the Label Studio (LS) demo locally.
  • LS has far more capability that we’ll need, but it’s to see the potential.
    • See the available labeling templates.
  • For the LS TagIfAI project, show the:
    • data, “completed” or not.
    • the labels from the two users/labelers.
    • the labeling interface.
    • the labeling configuration.
      • Show the “code” version of the configuration, which we’ll copy into our repo.
      • Show the sampling setting.
    • the export.
  • This demo is pretty much the lab. So you can follow it closely. See the previous videos for the Kappa computation.

References

• https://labelstud.io/

Snapshot

See the live demo.

Give snapshot of final Label Studio database dashboard.