Development Environments

Keith VanderLinden
Calvin University

Infrastructure for ML Systems

Layers of infrastructure required to support ML systems:

• Development Environment
• ML Platform
• Resource Management
• Storage & Compute

From DMLS Figure 10-2; cf. SaaS-PaaS-IaaS.

• We’ll focus mostly on the dev environment and storage/compute provided on our local machines (and, next week, in the cloud).
  • Storage & Compute
    • Physical or virtual hardware: CPUs; GPUs; Memory; Disk; Networking
  • Development Environment
    • Local or cloud-based software: operating systems; programming languages; Integrated Development Environments (IDEs); versioning tools (Git already covered in the previous modules)
  • These are generally provided for us, either in our classroom or virtual labs (e.g., Google Colab, Kaggle, RStudio Server, …). In this course, we’d like to build our own, so we can understand how they work.
• We’ll get to the ML platform and resource management later in the course.
  • Resource Management - tools that automate work(flows)
  • ML Platform - tools that manage ML artifacts and processes
• We’ll need the storage/compute + development environment now.
• Cf. the Azure diagram in the link above.
  • IaaS e.g., AWS, Azure, GCP, …
  • PaaS e.g., AWS/Azure/GCP app services, …
  • SaaS e.g., Office365, G*(mail|docs|sheets|slides|…), …

References

• https://www.ibm.com/topics/iaas-paas-saas
• https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-paas/
• https://cloud.google.com/learn/paas-vs-iaas-vs-saas

Command-Line Interfaces

Command-Line Interfaces (CLIs) are text-based interfaces to computer systems that:

• were an early form of user interaction with computers.
• were largely superseded by more intuitive Graphical User Interfaces (GUIs).
• are still heavily used for system administration and scripting.

Examples: Bash for Linux and Windows Subsystem for Linux (WSL2), PowerShell for Windows, Zsh for OSx, …

• We’ll be moving from cloud-based notebooks to local CLI scripts.
  • Notebooks are for people; scripts are for computers. The cloud is a computer. ChatGPT isn’t a notebook.
  • Cloud services just gives us stuff (IaaS + PaaS + SaaS?); we’ll need to manually configure our local machines.
• Go through CLI slide.
  • CLIs are more powerful that GUIs.
  • Introductory students often seek to avoid CLIs. In this class, we won’t avoid them; they’re too important. E.g., consider the Git CLI, which is my go-to tool for hard Git problems.
• Start building a deployment diagram (saving space for the full diagram).
  • product.ipynb on cloud GoogleColab
  • hello.py on local machine, in WSL2 (for Windows users)
• WSL2 is a Linux OS running in Windows; it’s not a dual-boot or an emulator, not CygWin or GitBash.
  • I’ve been much happier with WSL2 than with previous attempts to run Linux in Windows.
  • In my experience, DS development starts with OSx & Linux, and gets to Windows later, mostly.
  • I haven’t use Macs for a while, but my impression is that they’re better than Windows for DS development.

References

• https://en.wikipedia.org/wiki/Command-line_interface
• https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
• https://www.thomasmaurer.ch/wp-content/uploads/2019/06/WSL-2-Architecture.jpg

Python

• Python is a high-level, general-purpose programming language.
• Python virtual environments bundle the libraries required for separate tasks in separable modules.
• The set of Python libraries for a virtual environment can be specified using requirements.txt.

• Python and R are the standard programming languages in DS.
• Virtual environments allow us to build separate environments for different tasks.
  • E.g., we might have one environment for web development, another for DS, another for ML, …
  • This allows us to keep our libraries separate, so that we can use different versions of the same library for different tasks.
  • requirements.txt is a text file that lists the libraries required for a Python virtual environment.
    • We build the venv using: python3 -m pip install -r requirements.txt (see https://madewithml.com/courses/mlops/setup/)
  • I often keep my venv/ sub-directory in the root of my Git repo, but I add it to the .gitignore file, so that I don’t accidentally commit it to my repo.
  • This is standard practice.
• Add the Python-venv to the deployment diagram.

References

• https://2.bp.blogspot.com/-dDJh1jd8afc/WMbjQWZz3UI/AAAAAAAACB8/KBUg5Gngr38l843Cg2nnnGrf-ReHg1oMACLcB/s1600/python-virtual-env.png

Integrated Development Environments

Integrated Development Environments (IDEs) are software tools that integrate multiple development tools, including:

• Document editing and rendering
• Code development integration, e.g., programming, debugging, testing, …
• File management
• DVCS integration
• CLI integration
• LLM integration

• E.g., VSCode, Intellij, Colab, Kaggle, RStudio, …
• We’ll be using VSCode in this course because it’s a very common IDE for CS/DS, it has good support for Python and Git, and there are lots of engaged VSCode users building plugins.
• Add VSCode to the deployment diagram (n.b., it runs in Windows even with WSL2).
• We’ll demo VSCode in some detail later.

Local Development Environment

See the live walkthrough.

Link a snapshot of the final state of the deployment diagram here, which we’ll revisit during the demo.