376 Unit 5: Multimodal Models and Diffusion

You’ve just taken an AI/ML class. You might get asked: where is AI going? And is it good or bad? Now that you’ve engaged with the details of how some AI systems work, you’re much more qualified than average to answer those questions. But even experts disagree about those questions.

Some people are impressed by AI developments (we’ll call them “fans”). Others are skeptical (we’ll call them “skeptics”). Some people are optimistic about how AI will improve society (“optimists”). Others are worried, to the point of worrying about the future of humanity (“pessimists” or perhaps “concerned”). Many thoughtful people hold several of these views at once.

To be wise, we should consider various points of view: fans and skeptics, optimists and concerned.

Instructions

The Moodle form includes a link to a brief survey about your overall views on AI. Start by filling that out according to your current views.

Then, find two articles that represent different perspectives on the future of AI. (More on sources below.) Read with hospitality: you’ll need to be able to articulate the other side’s point of view.

1. For your first article:
   • Provide a well-formatted link (that indicates the source without having to click it).
   • State the overall stance in a keyword or two, e.g., [skeptical, optimistic] or [fan, concerned]
   • Summarize your first article in a few sentences. Make your summary convince someone who holds a different view to at least open the article and read it. (Yeah, you can use ChatGPT, but give it some guidance and edit its response.)
2. Repeat for your second article.
3. Articulate your own nuanced position, drawing on both articles. What stance should we take?
   • You are highly encouraged, but not required, to draw on what you have learned about Reformed Christian perspectives on technology in this and other classes. Examples:
     • Creation (unfolding latent potential possibilities, the Image of God, work as good)
     • the Fall (and its effect on relationships, work, and technology), idolatry, Mammon, etc.
     • Redemption and Restoration (Jesus reconciling everything), shalom (right relationships, flourishing, peace, rest, justice)

Responses

Read a few of your classmates’ responses to learn about their articles and positions. Respond to at least one of them.

Rubric

• Two articles are provided, each with a well-formatted link and a keyword indicating the overall stance.
• The first article is summarized compellingly, accurately, and briefly.
• The second article is summarized compellingly, accurately, and briefly.
• The student’s nuanced position is articulated clearly and thoughtfully.
• The student’s position demonstrates critical thinking by drawing substantively on both articles (beyond “I agree”) and, if applicable, on Reformed Christian perspectives on technology.
• The student’s response to a classmate’s post is thoughtful and engaging.
• The student’s writing is clear, concise, and free of grammatical errors.

Ideas for Sources

I won’t try to list every possible source here, especially because there are new ones all the time. Instead, here are some ideas of kinds of articles to look for (with a few examples if you’re lazy).

• Two from Nicholas Carlini:
  • My Thoughts on the Future of “AI”
  • Machines of Ruthless Efficiency
• capabilities: The Era of Experience: “A new generation of agents will acquire superhuman capabilities by learning predominantly from experience.” (see press coverage)
• AI as Normal Technology by Arvind Narayanan & Sayash Kapoor
• economic impacts:
  • The Anthropic Economic Index: Insights from Claude 3.7 Sonnet \ Anthropic
  • GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models | Abstract (note the date: March 2023)
• Why AI Will Save the World | Andreessen Horowitz
• Dario Amodei — Machines of Loving Grace
• Announcements of new AI capabilities
  • e.g., from tech companies (whether big ones like Meta, Google, OpenAI, or smaller ones like DataBricks, Anthropic, Mistral, etc.)
  • Academic from AI researchers (look at an aggregator like PapersWithCode)
  • Techie YouTube channels, like Two Minute Papers
  • or papers like Sparks of Artificial General Intelligence: Early experiments with GPT-4 | Abstract or The AI Revolution: The Road to Superintelligence (note: this is a rabbit hole, especially if you go on to part 2)
• Articles about the societal impact of AI
  • from journalists (note that Calvin has a site license to the New York Times)
  • from institutions like the Berkman Klein Center for Internet & Society at Harvard, AI Now Institute at NYU, AlgorithmWatch, Center for Humane Technology, etc.
• Reward Hacking in Reinforcement Learning | Lil’Log
• Articles from people who have been publicly skeptical of current claims of AI progress:
  • Gary Marcus
  • AI Snake Oil
  • two people named M. Mitchell
    • Margaret Mitchell
    • Melanie Mitchell
      • AI’s challenge of understanding the world | Science
      • How do we know how smart AI systems are? | Science
  • Emily Bender
    • On the Dangers of Stochastic Parrots | Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
    • You Are Not a Parrot And a chatbot is not a human. And a linguist named Emily M. Bender is very worried what will happen when we forget this.
  • others, misc
    • The Fallacy of AI Functionality | Abstract
    • How AI Fails Us
    • Enchanted Determinism: Power without Responsibility in Artificial Intelligence

Open this notebook. You can run it on Kaggle or Colab, or run it on a lab machine (but there’s some configuration issues to resolve if you do that.)

On a lab machine, start by opening a terminal and running the following command to set up the environment:

/home/cs/376/setup-cs376.sh

Then, log out and log back in. Then open a terminal and run the following command to start Jupyter Lab:

activate_376
jupyter lab

Download the following notebook, find it in the Jupyter Lab file browser, and open it:

• Stable Diffusion Deep Dive (name: u12n1-stable-diffusion.ipynb; show preview, open in Colab)

Your goal today is to be creative and see what you can make! The notebook includes a variety of things to try, but you can also try your own ideas.

For your write-up, you should include:

1. What does the “random seed” do in first diffusion main loop? What happens when you change it or remove it?
2. What happens when you set the guidance scale to 0 (again in the first diffusion loop)? Follow the flow of data through the code to see what’s happening.
3. What happens when you stop the diffusion early (e.g., if i == 10: break)? What do you notice about the output?
4. Now going to the image2image model: what is the role of adding noise to the latent image? What happens when you change the amount of noise? (change start_step; refer to num_inferenece_steps for its max value)
5. Something creative that you did with the Stable Diffusion model: what did you try, what results did you get, and what did you learn?

Some ideas of things to try:

• Change the prompt. Try more or less descriptive prompts. This is especially interesting for the image-to-image models.
• Under “messing with embeddings”, try changing the two prompts to different texts. Or change the mix factor. What happens?
• Try different ways of mixing things together:
  • Try different text chimeras
  • Try starting with an image latent composed of half of one image and half of another (horizontal or vertical split, or perhaps grab the middle from one and the edges from another)
• Try outpainting: scale the latents to half the size, replace the middle part of a random latent with those (each iteration), and diffuse.