---
format:
    revealjs:
        theme: simple
        chalkboard: true
        auto-stretch: false
---

## Systems Engineering for Data Science

Keith VanderLinden<br>
Calvin University

::: {.notes}
We now address the scope and nature of the design/engineering task for ML systems.
:::

## Systems Engineering

Systems engineering conceives of, designs, builds, and deploys systems that satisfy business requirements.

![DMLS Figure 1-1](images/DMLS-Figure1-1.png){fig-align=center width=60%}

::: {.notes}
- ML systems comprise more than just ML modeling (i.e., the middle row). We need to engineer the entire system, in context.
    - *Business Requirements* - Systems must satisfy business goals, which generally include (at least): reliability; scalability; maintainability; adaptability (see DMLS, p. 29-31).
    - *Developers*/project management - Data science is a team sport. There are many stakeholders: developers; users/UX; ...
    - *Infrastructure*/DevOps - We'll no longer assume that the systems "are just there" or that everything is done in Jupyter notebooks.
    - *Data*/Data Engineering - There's more than just data wrangling here; it's data engineering: storage; versioning; cleansing; security; tagging; ...
    - *Modelling*/ML Engineering - There's usually more than one model, so choices must be made.
    - *Maintenance* - ML systems are never "done"; they're always in flux.
    - *Users* - UX matters!; System engineering - The diagram should indicate that the (ML + human) system may need more than just the ML system (cf. IA vs AI).

References:

- [MLOps](https://en.wikipedia.org/wiki/MLOps)
- Three things required to deploy ChatGPT in a business context: <https://www.linkedin.com/pulse/3-things-chatgpt-needs-before-can-deployed-customer-service-ghoche/>
- DMLS Chapter 1 [case studies](https://github.com/chiphuyen/dmls-book/blob/main/resources.md#chapter-1-overview-of-machine-learning-systems).
:::

## Machine Learning

Machine learning is an approach to *learn complex patterns* from *existing data* and use these patterns to make &nbsp; *predictions on unseen data*.

::: {.notes}
- First, we should ask ourselves whether ML is even the right tool for the job.
- We start with C. Huyen's definition of ML.
- I found her comments on ML in context very insightful.
    - E.g., she notes that ML is sometimes necessary and/or cost-effective, but *never* sufficient (p. 3).
:::

## Machine Learning: Patterns

Machine learning is an approach to ***learn complex patterns*** from *existing data* and use these patterns to make &nbsp; *predictions on unseen data*. To be viable:

The patterns should be sufficiently complicated and changing that they can&rsquo;t be pre-specified. Distinguish:

- Traditional Programming<br>
`inputs + algorithms = outputs`

- Machine learning<br>
`inputs + outputs = patterns`

::: {.notes}
- Compare and contrast these paradigms.
    - *Traditional programming* works better for computable/consistent phenomena.
    - *ML* works better for phenomena that are hard to compute.
        - ML may be easier than enumerating all possible cases. Indeed, it may be the only way to solve some particularly complicated problems. (E.g.: SPAM detection; ChatGPT fluency).
        - ML may be easier to update in contexts of rapid change. (E.g.: SPAM detection; SLO's online capabilities).
- This has been a real paradigm shift. We spent a half-century trying to program "intelligent" systems; now ML is helping us approach those problems more effectively.

References

- P. Norvig's [Introduction to ML](https://developers.google.com/machine-learning/crash-course/ml-intro]
- DMLS Figure 1-2
:::

## Machine Learning: Data

Machine learning is an approach to *learn complex patterns* from ***existing data*** and use these patterns to make &nbsp;*predictions on unseen data*. To be viable:

There must be data for learning that&rsquo;s:

- Appropriate
- Voluminous
- Balanced
- Available
- Unbiased

::: {.notes}
- *Appropriate* - Supervised leaning requires input/output pairs.
    - E.g., SLO's unlabeled tweets
- *Voluminous* - ML systems need lots of learning examples.
    - E.g., SLO's limited, 0.75M-tweet dataset
- *Balanced* - Skewed data produces skewed results.
    - E.g., SLO's bias toward the negative
- *Available* - The data must be *ethically* available.
    - E.g., changes in Twitter data access
- *Unbiased* - Biased data produces biased results that might be not just wrong, but harmful.
    - E.g., SLO's reliance on social media data, which is likely biased/unrepresentative
:::

## Machine Learning: Predictions

Machine learning is an approach to *learn complex patterns* from *existing data* and use these patterns to make &nbsp; ***predictions on unseen data***. To be viable:

Predictions must be:

- Possible
- Non-Mission-Critical
- Valuable
- Appropriate

::: {.notes}
- *Possible* - The past and future must have similar distributions.
    - E.g., SLO's Twitter data were reasonable in the short term.
- *Non-Mission-Critical* - We need to be able to deal with incorrect predictions.
    - E.g., SLO is designed to augment existing processes based on surveys and focus groups.
- *Valuable* - We actually need to care about what&rsquo;s being predicted.
    - E.g., companies do care about their SLO, at least to some degree.
- *Appropriate* - This is related to value, but emphasizes human flourishing.
    - E.g., SLO's results are, admittedly, hard to assess in this regard.
    - I admit that the Friedman doctrine of maximizing profits for shareholders is potentially problematic here.
:::

## Software vs ML

:::: {.columns}
::: {.column}
*Software: Code & Data*

- Code/data are separate.
- Only code is versioned.
- Code-bases are small.
- Code is unit-tested.
- Code updates are infrequent.
:::
::: {.column}
*ML: Datasets and Models*

- Data/models are coupled.
- Data/models are versioned.
- Data/models are huge.
- ML is hard to test.
- Data/model updates are frequent.
:::
::::

::: {.notes}
- If we do choose to pursue an ML solution, we need to be aware of the differences between:
    - Traditional software
    - ML systems
- Notes
    - This course is based, in part, on a course in software engineering.
    - The focus in on the data.
        - E.g., my key contribution to the SLO system was identifying the value of user profile text in the training process.
        - H. Wickham argues that data analysis is not so much of a process as it is a mindset (cf. EDA).
    - Netflix re-deploys thousands of times per day.
:::

## Systems Engineering Process

:::: {.columns}

::: {.column}
*Software Engineering*

1. Analysis
2. Design
3. Implementation
4. Testing
5. Deployment &amp;<br>Maintenance
:::

::: {.column}
*ML/Data Engineering*

1. Project Scoping
2. Data &amp; Model Engineering
<p>&nbsp;</p><p>&nbsp;</p>
4. System Deployment
5. System Monitoring
6. Business Analysis
:::
::::

::: {.notes}
- The software vs ML differences manifest themselves in these different views of the engineering process.
    - Left - Traditional software project lifecycle, known as the *waterfall model*.
        - This is the model that we'll use in CS 262.
    - Right - C. Huyen's ML lifecycle.
        - The comparison/contrast between steps 4--5/4--6 is murky.
        - This will be unpacked throughout the text/course.
- Both must be pursued iteratively and incrementally (cf. Agile development).

References

- DMLS, p. 33-34
- CS 262, traditional vs agile models, <https://github.com/kvlinden-courses/cs262/blob/master/materials/02management/class.md>
:::
