Administrative Information
- Time/Place: Mondays 12:30–2:30 PM in CIWW 317
- Instructor: Marshall Ball, Office Hours: TBD
Fall 2026 · CSCI-GA 3033-137 · MATH-GA 2830-004
Updated coursework: Homework contributes 30% of the course grade, quizzes 30%, and the project 40%. The first homework will trial an AI-assisted feedback process with opportunities to revise; instructors will grade the final submissions. See Coursework for details.
Mathematical background: Please use the mathematical background packet (student version; PDF) to refresh the prerequisites and course terminology. Its practice exercises are optional and are not to be submitted for grading.
In the late 1940s, Claude Shannon introduced a mathematical theory of information, quantifying information in terms of uncertainty. Since Shannon's pioneering work on the limits of data compression and reliable/secure communication, intuitions and techniques from information theory have impacted not just our modern communication infrastructure but also a diverse array of other scientific endeavors, including theoretical computer science. In this course, we will begin by covering the foundations of information theory (entropy, mutual information, KL-divergence, etc) before branching off to explore various applications, primarily in theoretical computer science. Potential application topics include: channel and source coding, error correcting codes, communication complexity, hardness amplification, data structures, Kolmogorov complexity, information-theoretic cryptography, pseudoentropy, the Lovasz Local Lemma, and applications in combinatorics. While there are no specific prerequisites, fluency in basic probability and mathematical maturity are required.
Topics will draw from the following:
Assignment instructions, revision deadlines, quiz dates, and project details will be announced during the semester. Please contact the instructor with questions or concerns.
The primary prerequisite is mathematical maturity. You should be comfortable reading and writing proofs. Some familiarity with the basics of algorithms, the theory of computation, and probability is expected.
If you are unsure about whether this class is suitable for you, please contact the instructor via email.
We will not follow a single textbook. The references below complement the lectures; suggested sections and notes appear in the lecture plan. They are alternative and supplementary explanations, not a requirement to read every listed source.
Updated October 8: Lecture 1 covered material through §1.5, Lecture 2 through §2.6, and Lecture 3 through §3.6, including Pinsker. Lecture 4 covers average encoding, typicality, and fixed-length compression, including achievability and the strong converse. Notes for Lectures 1–4 are available below. Future topics remain provisional; December 14 is TBD.
Calendar: The September 21 meeting was held on Wednesday, September 23. There is no class on Monday, October 12. Class 5 meets on Wednesday, October 14, following NYU’s Monday schedule. See the NYU academic calendar. Quiz 1 is scheduled for Monday, October 26, 12:30–1:00 p.m., during the first thirty minutes of class. Quiz 2 remains planned for the first thirty minutes of November 30. Project presentations will take place outside regular lecture time.
| Class | Date | Topic | Suggested Reading | Lecture notes |
|---|---|---|---|---|
| 1 | Sep 14 | Entropy and information — covered. Expected surprise; joint and conditional entropy; chain rules; mutual and conditional mutual information, XOR, and data processing. Notes through §1.5. | Entropy and information portions of CT Chapter 2. | Lecture 1 (PDF) |
| 2 | Sep 23 (Wed) | Lossless coding, divergence, and counting — covered. Prefix codes and Kraft’s inequality; one-shot and iid expected-length source coding; KL divergence and Gibbs’ inequality; mismatch and information identities; entropy counting. Notes through §2.6; Fano was deferred to Lecture 3. | CT §§5.1–5.4 and Chapter 2 for divergence. PW §§2.2, 3.1–3.4, 10.3; §8.1 for counting. Haitner Lesson 1. | Lecture 2 (PDF) |
| 3 | Sep 28 | Information, inference, and testing — covered through §3.6. Fano; coordinate-recovery and storage lower bounds; KL chain rule and data processing; total variation and optimal testing; Pinsker. Average encoding appears in Lecture 4. | CT §§2.7–2.10. PW §§2.5, 3.5, 6.3, 6.5, 7.3–7.6. Haitner Lesson 2 and Haitner Lesson 7. | Lecture 3 (PDF) |
| 4 | Oct 5 | Average encoding and the compression threshold. Average encoding; AEP and typical sets; fixed-length compression allowing error, achievability, and strong converse. | CT §§3.1–3.3. PW §11.1. Haitner Lesson 4. Average encoding applies Pinsker to the joint and product distributions. | Lecture 4 (PDF) Simpler Pinsker proof (weaker constant, PDF) |
| — | Oct 12 | Fall break — no class. | The Monday meeting is held on Wednesday, October 14. | |
| 5 | Oct 14 (Wed) | Channels and capacity. Finite memoryless channels; BSC and BEC; capacity converse; information density and complete random-coding achievability. Expurgation and source–channel separation as extensions. | CT §§7.1–7.7, 7.9, 7.13. PW §§17.1–17.4, 18.1–18.2, 19.1–19.2. Haitner Lesson 5. PW gives the information-density route; CT gives a typicality-based alternative. | |
| 6 | Oct 19 | Codes, distance, and constructions. Rate and distance; packing, Singleton, and greedy bounds; linear codes; a worked Hamming-code construction and decoder. Concatenation as an extension. Finish channel coding first if needed. | Vadhan Chapter 5 for code parameters and constructions. CMU ITCS: February 19–21 for supplementary constructions. | |
| 7 | Oct 26 (Mon) | Quiz 1 (12:30–1:00 p.m.); Hadamard, list, and local decoding. Hadamard distance, a Parseval list-size bound, and local correction. Distinguish a small list from an efficient reconstruction algorithm. Reed–Solomon decoding as an extension. | Vadhan Chapter 5 for codes and list decoding; Chapter 7 for the later coding-to-hardness connection. CMU Coding: List Decoding and Reed–Solomon decoding as supplementary reading. | |
| 8 | Nov 2 | Entropy in combinatorics. Weighted and conditional Shearer; Loomis–Whitney and database-join counting. Independent sets in regular bipartite graphs as an extension. | PW §§1.5, 8.1–8.4. Haitner Lesson 3 and Haitner Lesson 6. CMU ITCS: March 19 and March 21. | |
| 9 | Nov 9 | Communication complexity I. Protocol models and rectangles; one-way indexing; transcript factorization; internal information cost and its communication upper bound. Discrepancy and inner product as an extension. | CMU ITCS: March 28, April 2–4, and April 9. TIFR: Lectures 1–5 for models and basic lower-bound methods. | |
| 10 | Nov 16 | Communication complexity II. Complete randomized set-disjointness lower bound: Hellinger distance, cut-and-paste, conditional information budgets, and an explicit embedding. One-way message simulation under pointwise domination as an extension. | CMU ITCS: April 11 and April 16. TIFR: Lectures 14–17 on information complexity and compression. | |
| 11 | Nov 23 | Parallel repetition and direct products. Precise two-prover theorem and model; conditioning and information-budget lemmas; a complete bounded-message recovery direct product. Hamming-error recovery as an extension. This slot may be used for unfinished core material. | Haitner Lesson 9 (interactive arguments). CMU ITCS: April 30 and Lecture 25/Lecture 26 (two-prover games). | |
| 12 | Nov 30 | Quiz 2 (first 30 minutes); extractors and privacy amplification. Min-entropy, universal hashing, and a complete strong leftover-hash proof. Classical side information and leakage as an extension. | Vadhan §§6.1–6.3. Expander and code connections are further reading. | |
| 13 | Dec 7 | Hardness and pseudorandomness — one reconstruction mechanism. Default: Goldreich–Levin reconstruction and hardcore bits, with a one-way-permutation PRG extension. Alternative: Nisan–Wigderson reconstruction and derandomization, with polynomial-graph designs as the extension. | Vadhan Chapter 7. Haitner Lesson 10 for hardcore predicates. | |
| 14 | Dec 14 | TBD |
Optional reading and project directions: Kolmogorov complexity, description length, pseudoentropy, and cryptographic applications remain possible directions. The December 14 topic is TBD. For algorithmic information, see CT §§14.1–14.5 and Haitner Lesson 8. Detailed extensions are optional; they are not prerequisites for later core lectures.
Homework should be submitted in PDF form in Gradescope. We prefer homework submissions typeset in LaTeX. If you are not familiar with LaTeX, it is a great skill to learn. Overleaf provides a simple web interface for writing and compiling LaTeX (as well as extensive documentation). We will provide LaTeX source for you to edit. You are encouraged to insert scanned figures or illustrations where appropriate. Scanned handwritten submissions will only be graded if perfectly legible. If you are unsure about your handwriting, I strongly suggest you type your solutions.
An important part of this class is about learning to communicate your mathematical ideas and proofs clearly and concisely. Accordingly, you will be graded not simply for correctness, but also clarity.
We strongly encourage you to discuss assignments with up to 3 peers, but you must (a) list the names of your discussion partners on your submission, and (b) you must write up your solution on your own. You may not look at the written solutions of any other student before submitting your own solution. If you do not list the names of your collaborators, you will be penalized.
Revisions may be submitted before the final deadline specified for each assignment. Late final submissions will not be accepted; the lowest homework score will be dropped.
Acknowledge any external resources consulted in your homework. You must write your own proofs. External tools to obtain homework solutions, consulting solutions from other students or elsewhere, or asking an LLM to solve an assigned problem are not allowed.
Your work should be your own. Students should be aware of the CS Department's Policy on Academic Integrity. Violations of academic integrity will not be tolerated.
As a nonsectarian, inclusive institution, NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations. The policy and principles to be followed by students and faculty may be found in the University Calendar Policy on Religious Holidays.
Academic accommodations are available to any student with a chronic, psychological, visual, mobility, learning disability, or who is deaf or hard of hearing. Students should please register with the Moses Center for Students with Disabilities.