Schedule

Readings:

• Jurasky and Martin, Chapter 10
• Illustrated Transformer

Readings:

• What Language Model Architecture and Pretraining Objective Work Best

Readings:

• Documenting C4
• Decline of AI Data Commons
• ClueWeb22

Readings:

• Activation Functions
• Layer norms
• Grouped-Query Attention
• RoPE embeddings

Readings:

• GEM Benchmark
• BIG-Bench, chapters 1 and 2
• Holistic Evaluation of Language Models
• Evaluation Challenges

Readings:

• Prompt Tuning
• Blog post on param-efficient tuning
• Really detailed survey paper

Readings:

• Chain-of-thought prompting
• What makes in-context learning work?
• Are aligned models adversarially aligned?

Readings:

• Dense Retrieval Training
• What are tools anyway
• Toolformer

Readings:

• Scaling laws
• Chinchilla

Readings:

• GPipe
• DeepSpeed
• Adam

Readings:

• Edge Probing
• Loss Landscape

Readings:

• How does LLM safety training fail
• Extracting training data from production models

Readings:

• Redteaming LLMs

Suggested Readings:

• Review of MoE
• Switch Transformer
• MatFormer

Readings:

• Speculative Decoding
• FlashAttention
• PagedAttention

– Readings:

• Chatbot Arena
• Evaluating human evaluation

Readings:

• Lost in the middle
• ProLong

Readings:

• MATES
• Montessori-Instruct