Systems Engineering for Data Science

Keith VanderLinden
Calvin University

We now address the scope and nature of the design/engineering task for ML systems.

Systems Engineering

Systems engineering conceives of, designs, builds, and deploys systems that satisfy business requirements.

DMLS Figure 1-1

• ML systems comprise more than just ML modeling (i.e., the middle row). We need to engineer the entire system, in context.
  • Business Requirements - Systems must satisfy business goals, which generally include (at least): reliability; scalability; maintainability; adaptability (see DMLS, p. 29-31).
  • Developers/project management - Data science is a team sport. There are many stakeholders: developers; users/UX; …
  • Infrastructure/DevOps - We’ll no longer assume that the systems “are just there” or that everything is done in Jupyter notebooks.
  • Data/Data Engineering - There’s more than just data wrangling here; it’s data engineering: storage; versioning; cleansing; security; tagging; …
  • Modelling/ML Engineering - There’s usually more than one model, so choices must be made.
  • Maintenance - ML systems are never “done”; they’re always in flux.
  • Users - UX matters!; System engineering - The diagram should indicate that the (ML + human) system may need more than just the ML system (cf. IA vs AI).

References:

• MLOps
• Three things required to deploy ChatGPT in a business context: https://www.linkedin.com/pulse/3-things-chatgpt-needs-before-can-deployed-customer-service-ghoche/
• DMLS Chapter 1 case studies.

Machine Learning

Machine learning is an approach to learn complex patterns from existing data and use these patterns to make   predictions on unseen data.

• First, we should ask ourselves whether ML is even the right tool for the job.
• We start with C. Huyen’s definition of ML.
• I found her comments on ML in context very insightful.
  • E.g., she notes that ML is sometimes necessary and/or cost-effective, but never sufficient (p. 3).

Machine Learning: Patterns

Machine learning is an approach to learn complex patterns from existing data and use these patterns to make   predictions on unseen data. To be viable:

The patterns should be sufficiently complicated and changing that they can’t be pre-specified. Distinguish:

• Traditional Programming
  inputs + algorithms = outputs

• Machine learning
  inputs + outputs = patterns

• Compare and contrast these paradigms.
  • Traditional programming works better for computable/consistent phenomena.
  • ML works better for phenomena that are hard to compute.
    • ML may be easier than enumerating all possible cases. Indeed, it may be the only way to solve some particularly complicated problems. (E.g.: SPAM detection; ChatGPT fluency).
    • ML may be easier to update in contexts of rapid change. (E.g.: SPAM detection; SLO’s online capabilities).
• This has been a real paradigm shift. We spent a half-century trying to program “intelligent” systems; now ML is helping us approach those problems more effectively.

References

• P. Norvig’s [Introduction to ML](https://developers.google.com/machine-learning/crash-course/ml-intro]
• DMLS Figure 1-2

Machine Learning: Data

Machine learning is an approach to learn complex patterns from existing data and use these patterns to make  predictions on unseen data. To be viable:

There must be data for learning that’s:

• Appropriate
• Voluminous
• Balanced
• Available
• Unbiased

• Appropriate - Supervised leaning requires input/output pairs.
  • E.g., SLO’s unlabeled tweets
• Voluminous - ML systems need lots of learning examples.
  • E.g., SLO’s limited, 0.75M-tweet dataset
• Balanced - Skewed data produces skewed results.
  • E.g., SLO’s bias toward the negative
• Available - The data must be ethically available.
  • E.g., changes in Twitter data access
• Unbiased - Biased data produces biased results that might be not just wrong, but harmful.
  • E.g., SLO’s reliance on social media data, which is likely biased/unrepresentative

Machine Learning: Predictions

Machine learning is an approach to learn complex patterns from existing data and use these patterns to make   predictions on unseen data. To be viable:

Predictions must be:

• Possible
• Non-Mission-Critical
• Valuable
• Appropriate

• Possible - The past and future must have similar distributions.
  • E.g., SLO’s Twitter data were reasonable in the short term.
• Non-Mission-Critical - We need to be able to deal with incorrect predictions.
  • E.g., SLO is designed to augment existing processes based on surveys and focus groups.
• Valuable - We actually need to care about what’s being predicted.
  • E.g., companies do care about their SLO, at least to some degree.
• Appropriate - This is related to value, but emphasizes human flourishing.
  • E.g., SLO’s results are, admittedly, hard to assess in this regard.
  • I admit that the Friedman doctrine of maximizing profits for shareholders is potentially problematic here.

Software vs ML

Software: Code & Data

• Code/data are separate.
• Only code is versioned.
• Code-bases are small.
• Code is unit-tested.
• Code updates are infrequent.

ML: Datasets and Models

• Data/models are coupled.
• Data/models are versioned.
• Data/models are huge.
• ML is hard to test.
• Data/model updates are frequent.

• If we do choose to pursue an ML solution, we need to be aware of the differences between:
  • Traditional software
  • ML systems
• Notes
  • This course is based, in part, on a course in software engineering.
  • The focus in on the data.
    • E.g., my key contribution to the SLO system was identifying the value of user profile text in the training process.
    • H. Wickham argues that data analysis is not so much of a process as it is a mindset (cf. EDA).
  • Netflix re-deploys thousands of times per day.

Systems Engineering Process

Software Engineering

1. Analysis
2. Design
3. Implementation
4. Testing
5. Deployment &
   Maintenance

ML/Data Engineering

1. Project Scoping
2. Data & Model Engineering

    

    

3. System Deployment
4. System Monitoring
5. Business Analysis

• The software vs ML differences manifest themselves in these different views of the engineering process.
  • Left - Traditional software project lifecycle, known as the waterfall model.
    • This is the model that we’ll use in CS 262.
  • Right - C. Huyen’s ML lifecycle.
    • The comparison/contrast between steps 4–5/4–6 is murky.
    • This will be unpacked throughout the text/course.
• Both must be pursued iteratively and incrementally (cf. Agile development).

References

• DMLS, p. 33-34
• CS 262, traditional vs agile models, https://github.com/kvlinden-courses/cs262/blob/master/materials/02management/class.md