Carnegie Mellon University
Large Language Models: Methods and Applications
Fall 2024

Schedule

  • Event
    Date
    Description
    Course Material
  • Lecture
    08/27/2024
    Tuesday
    Building blocks of modern LLMs
    [] [slides]
  • Assignment
    08/27/2024
    Tuesday
    Assignment #1 released
    [Implementing a Transformer from scratch and doing inference with the HuggingFace API]
  • Lecture
    08/29/2024
    Thursday
    Transformer architecture and pre-training learning objectives
    [] [slides]
  • Lecture
    09/03/2024
    Tuesday
    Pre-training data curation and tokenization
    [] [slides]
  • Lecture
    09/05/2024
    Thursday
    Architecture advancements since 2019: mixture-of-experts, layer norms, relative attention, RoPE embeddings
    [] [slides]
  • Due
    09/05/2024 14:00
    Thursday
    Assignment #1 due
  • Lecture
    09/10/2024
    Tuesday
    Automatic evaluation of LLMs
    [] [slides]
  • Lecture
    09/12/2024
    Thursday
    Task-oriented finetuning and parameter-efficient tuning methods
    [] [slides]
  • Assignment
    09/12/2024
    Thursday
    Assignment #2 released
    [Preparing and Exploring Pre-Training Data]
  • Lecture
    09/17/2024
    Tuesday
    Instruction tuning and chain-of-thought reasoning
    [] [slides]
  • Lecture
    09/19/2024
    Thursday
    Alignment, RLHF, jail-breaking
    [] [slides]
  • Lecture
    09/24/2024
    Tuesday
    Retrieval-augmented generation
    [] [slides]
  • Due
    09/24/2024 14:00
    Tuesday
    Assignment #2 due
  • Lecture
    09/26/2024
    Thursday
    Guest lecture
    [] [slides]
  • Assignment
    09/26/2024
    Thursday
    Assignment #3 released
    [Retrieval-augmented generation and tool-use]
  • Lecture
    10/01/2024
    Tuesday
    Tool use
    [] [slides]
  • Lecture
    10/03/2024
    Thursday
    Popular benchmarks and their brittleness
    [] [slides]
  • Lecture
    10/08/2024
    Tuesday
    Interpretability methods
    [] [slides]
  • Due
    10/08/2024 14:00
    Tuesday
    Assignment #3 due
  • Lecture
    10/10/2024
    Thursday
    Human evaluation
    [] [slides]
  • 10/15/2024 00:00
    Tuesday
    Fall break - no class
  • 10/17/2024 00:00
    Thursday
    Fall break - no class
  • Exam
    10/22/2024 00:00
    Tuesday
  • Assignment
    10/22/2024
    Tuesday
    Assignment #4 released
    [Comparing models and mitigating bias]
  • Lecture
    10/24/2024
    Thursday
    Bias and ethical issues
    [] [slides]
  • Lecture
    10/29/2024
    Tuesday
    Scaling laws
    [] [slides]
  • Lecture
    10/31/2024
    Thursday
    Optimization and parallel training
    [] [slides]
  • Assignment
    10/31/2024
    Thursday
    Assignment #5 released
    [Improving training efficiency]
  • Due
    10/31/2024 14:00
    Thursday
    Assignment #4 due
  • Lecture
    11/05/2024
    Tuesday
    Democracy day - no class
    [] [slides]
  • Lecture
    11/07/2024
    Thursday
    Long-sequence models
    [] [slides]
  • Lecture
    11/12/2024
    Tuesday
    Flash attention and sparse models
    [] [slides]
  • Lecture
    11/14/2024
    Thursday
    Chatbots and AI agents
    [] [slides]
  • Assignment
    11/14/2024
    Thursday
    Assignment #6 released
    [Mini-project]
  • Due
    11/14/2024 14:00
    Thursday
    Assignment #5 due
  • Lecture
    11/19/2024
    Tuesday
    Embedding learning
    [] [slides]
  • Lecture
    11/21/2024
    Thursday
    Synthetic data generation, self-play, models teaching models
    [] [slides]
  • Lecture
    11/26/2024
    Tuesday
    Legal considerations
    [] [slides]
  • 11/28/2024 00:00
    Thursday
    Thanksgiving - no class
  • Lecture
    12/03/2024
    Tuesday
    guest lecture
    [] [slides]
  • Lecture
    12/05/2024
    Thursday
    guest lecture
    [] [slides]
  • Due
    12/05/2024 14:00
    Thursday
    Assignment #6 due