Administrative Information
- Time/Place: Mondays 12:30–2:30 PM in CIWW 317
- Instructor: Marshall Ball, Office Hours: TBD
Fall 2026 · CSCI-GA 3033-137 · MATH-GA 2830-004
Lecture 1 notes: The revised Draft PDF is available in the new Lecture notes column of the schedule. It covers the material from our September 14 meeting.
Updated coursework: Homework contributes 30% of the course grade, quizzes 30%, and the project 40%. The first homework will trial an AI-assisted feedback process with opportunities to revise; instructors will grade the final submissions. See Coursework for details.
Mathematical background: Please use the mathematical background packet (student version; PDF) to refresh the prerequisites and course terminology. Its practice exercises are optional and are not to be submitted for grading.
In the late 1940s, Claude Shannon introduced a mathematical theory of information, quantifying information in terms of uncertainty. Since Shannon's pioneering work on the limits of data compression and reliable/secure communication, intuitions and techniques from information theory have impacted not just our modern communication infrastructure but also a diverse array of other scientific endeavors, including theoretical computer science. In this course, we will begin by covering the foundations of information theory (entropy, mutual information, KL-divergence, etc) before branching off to explore various applications, primarily in theoretical computer science. Potential application topics include: channel and source coding, error correcting codes, communication complexity, hardness amplification, data structures, Kolmogorov complexity, information-theoretic cryptography, pseudoentropy, the Lovasz Local Lemma, and applications in combinatorics. While there are no specific prerequisites, fluency in basic probability and mathematical maturity are required.
Topics will draw from the following:
Assignment instructions, revision deadlines, quiz dates, and project details will be announced during the semester. Please contact the instructor with questions or concerns.
The primary prerequisite is mathematical maturity. You should be comfortable reading and writing proofs. Some familiarity with the basics of algorithms, the theory of computation, and probability is expected.
If you are unsure about whether this class is suitable for you, please contact the instructor via email.
We will not follow a single textbook. The references below complement the lectures; suggested sections and notes appear in the lecture plan. They are alternative and supplementary explanations, not a requirement to read every listed source.
Updated September 16: The first meeting covered entropy and information through §1.5 of the notes. The next meeting resumes with prefix coding at §1.6. The opening five meetings develop the foundations, followed by applications in coding, combinatorics, complexity, and cryptography. Future coverage remains provisional.
Calendar: Class 2 moves from Monday, September 21 to Wednesday, September 23, 12:30–2:30 PM, in CIWW 202. There is no class on Monday, October 12. Class 5 meets on Wednesday, October 14, following NYU’s Monday schedule. See the NYU academic calendar. Quiz and end-of-course project presentation dates will be announced separately; lecture pacing may be adjusted to accommodate them.
| Class | Date | Topic | Suggested Reading | Lecture notes |
|---|---|---|---|---|
| 1 | Sep 14 | Entropy and information — covered. Expected surprise and entropy; joint and conditional entropy; chain rules; mutual information, conditional information, XOR, and data processing. Notes through §1.5; prefix coding begins next time. | Entropy and information portions of CT Chapter 2. | Lecture 1 (PDF) |
| 2 | Sep 23 (Wed) | Lossless coding, divergence, and the start of inference. Resume at §1.6: prefix codes and Kraft’s inequality; one-shot and iid expected-length source coding; KL divergence, Gibbs’ inequality, and information identities; entropy counting. Begin Fano’s inequality if time permits. | CT §§5.1–5.4 and Chapter 2 for divergence and Fano. PW §§2.2, 3.1–3.4, 10.3; §8.1 for counting. Haitner Lesson 1 and Lesson 7. | |
| 3 | Sep 28 | Information, inference, and indistinguishability. Complete or recall Fano; coordinate-recovery and storage lower bounds; KL chain rule and data processing; total variation and testing; Pinsker, average encoding, and a testing application. | CT §§2.7–2.10. PW §§2.5, 3.5, 6.3, 6.5, 7.3–7.6. Haitner Lesson 2 and Haitner Lesson 7. Average encoding applies Pinsker to the joint and product distributions. | |
| 4 | Oct 5 | Typicality, types, and the compression threshold. AEP and typical sets; fixed-length compression with error, achievability and strong converse; type classes and divergence exponents. Universal compression as supplementary reading. | CT §§3.1–3.3, 11.1–11.3. PW §11.1; §13.2 for optional universal compression. Haitner Lesson 4. | |
| — | Oct 12 | Fall break — no class. | The Monday meeting is held on Wednesday, October 14. | |
| 5 | Oct 14 (Wed) | Channels, capacity, and information converses. Finite memoryless channels; BSC and BEC; repetition coding; capacity converse; information density and random coding; maximal error and source–channel separation. | CT §§7.1–7.7, 7.9, 7.13. PW §§17.1–17.4, 18.1–18.2, 19.1–19.2. Haitner Lesson 5. PW gives the information-density route; CT gives a typicality-based alternative. | |
| 6 | Oct 19 | Coding at the Shannon/Hamming interface. Linear codes, concatenation, and polar codes. | PW §§11.2, 18.6. CMU ITCS: February 19–21 and February 26. CMU Coding: Polar Codes, Parts 1 and 2. | |
| 7 | Oct 26 | List decoding and local decoding. Reed–Solomon and folded Reed–Solomon codes; locally decodable and locally list-decodable codes. | Vadhan §§5.1–5.2, 7.5–7.6. CMU Coding: List Decoding and Reed–Solomon List Decoding. | |
| 8 | Nov 2 | Entropy in combinatorics. Shearer’s lemma, graph entropy, and entropy compression. | PW §§1.5, 8.1–8.4. Haitner Lesson 3 and Haitner Lesson 6. CMU ITCS: March 19 and March 21. | |
| 9 | Nov 9 | Communication complexity I. Deterministic, randomized, and distributional models; discrepancy and indexing lower bounds. | CMU ITCS: March 28, April 2–4, and April 9. TIFR: Lectures 1–5 for models and basic lower-bound methods. | |
| 10 | Nov 16 | Communication complexity II. Set disjointness, information cost, direct sums, and protocol compression. | CMU ITCS: April 11 and April 16. TIFR: Lectures 14–17 on information complexity and compression. | |
| 11 | Nov 23 | Parallel repetition and direct products. Information-theoretic proof methods for repetition. | Haitner Lesson 9 (interactive arguments). CMU ITCS: April 30 and Lecture 25/Lecture 26 (two-prover games). | |
| 12 | Nov 30 | Randomness extractors. Min-entropy, the leftover hash lemma, expanders, and connections between extractors and codes. | Vadhan §§6.1–6.3; §§4.1 and 5.3 for expander and code connections. | |
| 13 | Dec 7 | Hardness amplification and pseudorandomness. Worst-case to average-case reductions and pseudorandom generators from hardness. | Vadhan §§7.1–7.6. Haitner Lesson 10 (hardcore predicates). | |
| 14 | Dec 14 | Cryptographic entropy and synthesis. Pseudoentropy, accessible entropy, and commitments; connections across the course. | Haitner Lesson 11 and Haitner Lesson 12. Vadhan §7.2 for background on cryptographic PRGs. |
Optional reading and project direction: Kolmogorov complexity, description length, and incompressibility remain part of the course resources rather than a separate scheduled meeting. See CT §§14.1–14.5 and Haitner Lesson 8. Min-entropy will be introduced in the extractors meeting. The extra foundations meeting preserves the two coding meetings and both communication-complexity meetings.
Homework should be submitted in PDF form in Gradescope. We prefer homework submissions typeset in LaTeX. If you are not familiar with LaTeX, it is a great skill to learn. Overleaf provides a simple web interface for writing and compiling LaTeX (as well as extensive documentation). We will provide LaTeX source for you to edit. You are encouraged to insert scanned figures or illustrations where appropriate. Scanned handwritten submissions will only be graded if perfectly legible. If you are unsure about your handwriting, I strongly suggest you type your solutions.
An important part of this class is about learning to communicate your mathematical ideas and proofs clearly and concisely. Accordingly, you will be graded not simply for correctness, but also clarity.
We strongly encourage you to discuss assignments with up to 3 peers, but you must (a) list the names of your discussion partners on your submission, and (b) you must write up your solution on your own. You may not look at the written solutions of any other student before submitting your own solution. If you do not list the names of your collaborators, you will be penalized.
Revisions may be submitted before the final deadline specified for each assignment. Late final submissions will not be accepted; the lowest homework score will be dropped.
Acknowledge any external resources consulted in your homework. You must write your own proofs. External tools to obtain homework solutions, consulting solutions from other students or elsewhere, or asking an LLM to solve an assigned problem are not allowed.
Your work should be your own. Students should be aware of the CS Department's Policy on Academic Integrity. Violations of academic integrity will not be tolerated.
As a nonsectarian, inclusive institution, NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations. The policy and principles to be followed by students and faculty may be found in the University Calendar Policy on Religious Holidays.
Academic accommodations are available to any student with a chronic, psychological, visual, mobility, learning disability, or who is deaf or hard of hearing. Students should please register with the Moses Center for Students with Disabilities.