Feature Engineering

Keith VanderLinden
Calvin University

• Feature engineering lies on the boundary between data and models.
• DMLS covers it in chapter 5, and treats it as part of model engineering (sort of, see p. 34).
• MwML does it as part of data preprocessing, e.g., we did a simple feature cross/concatenation of the title and the description fields.
• We may find it useful in the project as well.

Feature Types

Features, important inputs to the modeling process, can be:

• Engineered
  • Handling missing values (cf. MCAR, MAR, MNAR)
  • Feature scaling
  • Feature crossing
  • …
• Learned

C. Huyen argues that we’re not ready to learn all features.

• Engineered features are created by humans.
  • Missing values
    • Handling missing values
      • Deletion
      • Imputation
    • Depends on the feature
      • Missing Completely at Random (MCAR) - You can delete these.
      • Missing at Random (MAR) - You can control for the observed reasons for missingness.
      • Missing not at Random (MNAR) - This one is hard to address because of the un-observed reasons for missingness.
    • The bottom line - know your data and your domain.
  • Scaling - ML systems are sensitive to mis-matching feature scales.
  • Feature crossing - This combines features that might not be predictive in isolation, but are predictive in combination.
  • Discretization - C. Huyen dismisses this one.
  • Feature hashing - The hashing trick is commonly used but poorly understood.
  • Positional embeddings - This is transformer stuff.
• Learned features are created by the model.
  • C. Huyen argues that we just haven’t gotten to the point where we can learn features, even with deep networks. And, by the way, most production models are not deep networks.

References

• Missing data types:
  • https://en.wikipedia.org/wiki/Missing_data#Types
  • https://www.ncbi.nlm.nih.gov/books/NBK493614/
• Feature crosses:
  • https://developers.google.com/machine-learning/crash-course/feature-crosses/encoding-nonlinearity

Data Leakage

Data leakage occurs when information from outside the training set is used to create the model.

• Random selection from correlated sub-sets of the data
  • time-sequenced data
  • grouped data
• Processing data before splitting training and testing data
  • Scaling
  • Imputation

Data leaks are notoriously hard to prevent.

• General Rule: Don’t train on your testing data. Don’t even peek at it!
  • This is easier said than done.
  • Leakage can happen in so many ways that can be subtle, hard to detect, and hard to avoid.
• Random selection from sub-sets of the data with strong correlations
  • Time-sequenced data (Train on past; test on future)
    • SLO failed to handle this with time-sequenced tweets (e.g., events in the test set associated with particular stances found their way into the training set).
  • “Un-controlled-for groups” (so called) of data (Train on one group; test on another)
    • This is such a classic problem (e.g., the [apocryphal] story of tank recognition on sunny and cloudy days).
• Processing data before splitting
  • Scaling (Use training data only to compute the scaling parameters; apply the scaling to both training and test data)
  • Imputation (same here…)
  • Removing duplicates (Remove duplicates so they don’t appear in both training and test data.)
• Data leak is often signaled by results that are “too good to be true”.

References

• https://en.wikipedia.org/wiki/Leakage_(machine_learning)
• The ML Tank story is probably an urban legend, but it’s a good story: https://en.wikipedia.org/wiki/Tank_camouflage https://gwern.net/tank. C. Huyen gives similar, but real, examples in DMLS.

Feature Stores

Feature stores connect data and model engineering workflows, providing these capabilities.

• Feature management
• Feature computation
• Feature consistency

They’re useful when multiple applications share feature definition and construction.

DMLS Figure 7-8

• Feature stores are tools that link data and model engineering workflows.
  • DMLS views them as as part of MLOps (Chapter 10, p. 325-327), i.e., features from a systems point of view.
  • MwML mentions them only in passing as part of data engineering. It used to treat them more explicitly here: data engineering - feature store (still mentioned here: Implementations, hyperlinked here: Data drift, and accessible here: Feature Store).
• The figure (a draft of the one in the text) motivates the need for feature stores.
  • The box shows what we commonly do in ML courses.
  • The streaming aspects are required in real-time inference, where “inference” uses up-to-date feature values as input to the trained model to produce live predictions.
  • The “should be equal” line is a key motivation for feature stores (see the capabilities discussion below).
  • E.g., Netflix or Amazon recommendations are based on:
    • Models, which are trained frequently.
    • Model inputs, which are up-to-date feature values.
• (DMLS) Feature store capabilities:
  • Feature management: definitions, access, sharing
  • Feature computation: ingestion from heterogeneous sources, batch/stream compute, validation, versioning
  • Feature consistency: across training (e.g., constructed using Python) and serving (constructed using a production language)
• (MwML) Use a feature store when you have:
  • Duplication - lots of ML applications, all using the same or similar features, each of which is hard to define and costly to construct.
  • Skew - the risk of using different, incompatible, feature engineering pipelines for training and for serving.
  • Value - the requirement to keep the features up-to-date, which is particularly difficult to do in real-time inference due to the point-in-time correctness problem.
• Conclusions
  • SLO (circa 2017) didn’t use one, but we could have to manage retraining in the context of streaming values.
  • TagIfAI and the team project, as single projects, don’t really need one, so we won’t create a full feature store.
  • For real projects, C. Huyen says that 40% of companies she surveyed in 2022 used feature stores, roughly half built their own, and half used a commercial product.

References

• https://towardsdatascience.com/point-in-time-correctness-in-real-time-machine-learning-32770f322fb1
  • When building training sets from historical data, we need to make sure that the timestamp of the training features precedes the timestamp of the predicted label.If we don’t do this, we’re feeding the answer to the model.
• DMLS, Chapter 10, p. 325-327
• MwML, Feature Store, referenced from the MwML preprocessing-extraction.
• MwML batch vs streaming architectures: https://madewithml.com/courses/mlops/feature-store/#architecture