Model Development

Keith VanderLinden
Calvin University

• This course assumes a basic understanding modeling and opts for simple models.
• If you do, then you probably agree with C. Huyen when she observes that modeling is the fun part.
• It can be, unless you’re joining a project that’s had numerous people building and abandoning numerous models haphazardly over time. That’s a nightmare!

Model Evaluation

Modeling is an iterative process in which we experiment with different models and model configurations. Best practices:

• Simple is better than complex.
• People are biased.
• Time changes things.
• There are no perfect solutions.
• All models are wrong.

• Modeling is an iterative process. These are C. Huyen’s best practices.
• Simple is better than complex.
  • Start simple and proceed iteratively & incrementally.
  • Avoid state-of-the-art, at least at first.
  • Ensemble methods (DMLS p. 156-161) are complex but effective when small gains in performance bring large payoffs.
  • C.f. the observation that the extra work required to get asymptotically close to winning Kaggle competitions probably doesn’t lead to better models, just models that better match particular datasets.
• People are biased.
  • Acknowledge your own biases (q.v., List of cognitive biases).
• Time changes things.
  • Academic work is often short-lived (one-semester-limited); not so for production work.
• There is no perfect solution.
  • Latency and compute requirements are legitimate considerations.
• All models are wrong, but some are useful - G. Box.
  • I wonder if any of the assumptions C. Huyen lists (DMLS p. 155-156) are true.

Experiment Tracking and Versioning

Success in modeling often depends on experimentation with:

• Datasets and Preprocessing
• Model Architectures & Hyperparameters

All leading to different:

• Evaluation results
• Memory and compute requirements

We must be able to compare these experiments (tracking) and to reproduce them (versioning).

• This will be the focus of our class work in this module.
• Tracking maintains a record of the experiment results for comparison.
  • Modern Science is based on this practice.
• Versioning maintains a record of the code and data used in the experiment for reproducibility.
  • Versioning code is standard practice in software development; versioning data is less common.

Distributed Training

In production, the simplifying assumptions we make in academic work are generally not true.

• Datasets can’t be fit into main memory.
• Models can’t be trained in a single machine.

“Big data” is when your workflow breaks. — R. Pruim, MDSR2e

• C. Huyen merely reviews the issues here.
• Datasets don’t fit into main memory; neither do production databases.
  • Data parallelism partitions the data and copies the model, which leads to problems aggregating the model copies.
• Models can’t be trained in a single machine.
  • Model parallelism copies the data and partitions the model, which leads to problems sequencing the model partitions.
  • Pipeline parallelism partitions the model in clever, sequential ways.
• G. Mohandas uses Ray to help address these issues.

AutoML

Much of the success of machine learning has been due to the deployment of relatively simple approaches trained on relatively voluminous datasets and powerful compute engines.

AutoML applies this approach to:

• Hyper-parameter tuning
• Architecture search

• Hyper-parameter tuning, e.g., grid search.
• Architecture search

References

• https://en.wikipedia.org/wiki/Hyperparameter_optimization
• https://en.wikipedia.org/wiki/Neural_architecture_search

Phases of ML Adoption

Huyen presents these phrases of adoption for ML.

1. Before ML
2. Simple models
3. Optimizing
4. Complex models

I found these to be a helpful tonic against ML mania.

• It’s best not to jump directly to phase 4.
• Before ML - Heuristics can be surprisingly effective (cf. scruffy, symbolic AI).
• Simple models - Follow the KISS principle.
• Optimizing - Interestingly, this appears before complex/deep models.
• Complex models - Sometimes, you really do need complex models (q.v., ChatGPT).

Model Metrics

You can’t succeed if you can’t measure success.

• Start with well-understood baseline models.
• Extra-performance metrics
  • Perturbation tests
  • Invariance tests
  • Calibration tests
  • Slice-based tests

• We’ve already talked about the value of baseline models.
• Non-traditional performance metrics (i.e., not accuracy, precision, recall, F1)
  • Perturbation tests - How does the model respond to perturbations of the input signal?
  • Invariance tests - How does the model respond to modifications of input that shouldn’t matter?
  • Calibration tests - How closely does the model’s behavior match its confidence?
  • Slice-based tests - How does the model respond to different sub-sets of the data? (cf. Simpson’s paradox, in which patterns that appear in sub-sets are different from those in the aggregate - e.g., Gerrymandering)