Lecture Schedule

Tuesday, Aug 29
Building Blocks of Modern LLMs
Readings: Jurasky and Martin, Chapter 10, Illustrated Transformer
Thursday, Aug 31
Transformer Architecture, LLM Pretraining Methods
Slides 1 Slides 2
Readings: Dai and Le, Radford et al., Lewis et al.
Tuesday, Sep 5
LLM Pretraining Data HW1 OUT
Readings: ClueWeb22, Documenting C4
Thursday, Sep 7
In-Context Learning, Prompt Engineering, and Alignment
Readings: GPT-2 paper, Prompt Engineering
Tuesday, Sep 12
PROJECT OUT Interpretability
Readings: Edge Probing, Loss Landscape
Thursday, Sep 14
Generation-based Automatic Evaluation Methods
Readings: GEM Benchmark, Evaluation Challenges
Tuesday, Sep 19
Human Evaluation Methods
Readings: Evaluation of Text Generation: A Survey
Thursday, Sep 21
Parameter-Efficient Tuning Methods
Readings: Prompt Tuning, PETM blog post
Tuesday, Sep 26
Generation before LLMs, Discuss HW1 HW1 DUE PROJECT PROPOSALS DUE
Readings: Building NLG Systems, chapter 3, Two Decades of Statistical Language Modeleing
Thursday, Sep 28
In-class Project Proposals HW2 OUT
Tuesday, Oct 3
Industry Talk:; Chatbots and AI Agents
Readings: Diplomacy agents, Generative Agents: Interactive Simulacra of Human Behavior
Thursday, Oct 5
LLM for Search Engines
Readings: ANCE, SentenceT5
Tuesday, Oct 10
Readings: None
Thursday, Oct 12
LLMs for Music Generation (Guest Lecture from Chris Donahue)
Readings: AudioLM, Oore+ 18
Tuesday, Oct 17
Fall break (no class)
Readings: Enjoy Fall Break
Thursday, Oct 19
Fall break (no class)
Readings: Enjoy Fall Break
Tuesday, Oct 24
Anonymous Feedback for the Class. LLM for Search Engines Continued HW2 DUE
Readings: Scaling Laws, Chinchilla
Thursday, Oct 26
Paradigmns of Visual Representation Learning (Guest Lecture from Xinlei Chen)
Readings: SimCLR, MAE
Tuesday, Oct 31
Midpoint Project Presentations PROJECT MIDPOINT DUE
Project Midpoint Slides
Thursday, Nov 2
Scaling Up: Scaling Laws
Readings: Adam, 8-bit Adam
Tuesday, Nov 7
Democracy day (no class)
Readings: None
Thursday, Nov 9
Industry Talk: Anthropic; Scaling Up: Optimizer Basis
Readings: GPipe, ZeRO
Tuesday, Nov 14
HW3 OUT Industry Talk: LLaMA2; Scaling Up: Parallel Optimization
Readings: ZeRO Offload, LLaMA2
Thursday, Nov 16
Retrieval Augmentation
Readings: RAG, REALM
Tuesday, Nov 21
Efficient Serving (Guest Lecture from Beidi Chen)
Readings: H2O, StreamLB
Thursday, Nov 23
Happy Thanksgiving (no class)
Readings: Enjoy break
Tuesday, Nov 28
LLMs for Code and Tool Use (Guest Lecture from Daniel Fried)
Readings: TODO
Thursday, Nov 30
HW3 DUE; Industry Talk: Cohere; Legal Considerations (Guest Lecture from A Feder Cooper)
Readings: AI Supply Chain
Tuesday, Dec 5
Ethical Considerations and Model Bias (Guest Lecture from Maarten Sap)
Readings: TODO
Thursday, Dec 7
Final Project Presentations
Readings: TODO