Information Theory and Applications in Computer Science — Fall 2026

Announcements

Updated coursework: Homework contributes 30% of the course grade, quizzes 30%, and the project 40%. The first homework will trial an AI-assisted feedback process with opportunities to revise; instructors will grade the final submissions. See Coursework for details.

Mathematical background: Please use the mathematical background packet (student version; PDF) to refresh the prerequisites and course terminology. Its practice exercises are optional and are not to be submitted for grading.

Administrative Information

Time/Place: Mondays 12:30–2:30 PM in CIWW 317
Instructor: Marshall Ball, Office Hours: TBD

Course Description

In the late 1940s, Claude Shannon introduced a mathematical theory of information, quantifying information in terms of uncertainty. Since Shannon's pioneering work on the limits of data compression and reliable/secure communication, intuitions and techniques from information theory have impacted not just our modern communication infrastructure but also a diverse array of other scientific endeavors, including theoretical computer science. In this course, we will begin by covering the foundations of information theory (entropy, mutual information, KL-divergence, etc) before branching off to explore various applications, primarily in theoretical computer science. Potential application topics include: channel and source coding, error correcting codes, communication complexity, hardness amplification, data structures, Kolmogorov complexity, information-theoretic cryptography, pseudoentropy, the Lovasz Local Lemma, and applications in combinatorics. While there are no specific prerequisites, fluency in basic probability and mathematical maturity are required.

Topics will draw from the following:

Fundamentals: Entropy, Conditional Entropy, Mutual Information, KL Divergence, Source Coding, Channels
Elements of Coding Theory: Concatenated Codes, Polar Codes, Reed-Muller Codes
Communication Complexity: Lower bounds, Information Complexity, Compression
Application: Lower Bounds for Data Structures and Cryptography
Application: Hardness Amplification (Worst-case to average-case reductions, Parallel Repetition, Derandomization)
Algorithmic Information: Kolmogorov Complexity
Pseudorandomness and pseudoinformation variants: Cryptographic Applications
Randomness Extractors

Coursework and Grading

Homework — 30%. The goal is to understand the material, develop proofs, and write them clearly and precisely. On the first assignment, we will try an experimental AI-assisted feedback process to provide faster, more detailed feedback on your writing. You may submit multiple revisions and flag a particular problem for more detailed instructor feedback. The final submission will be graded by the instructors. We will revise the process or return to traditional feedback if the trial is unsuccessful.
Quizzes — 30%. These will test basic knowledge of the course material. Doing the homework, attending class, and briefly reviewing your notes should prepare you well.
Project — 40%. A short written manuscript and a brief in-person presentation at the end of the course. You may work individually or in a group. You will develop the topic and work in consultation with the instructors. Further details will be provided later in the semester.

Assignment instructions, revision deadlines, quiz dates, and project details will be announced during the semester. Please contact the instructor with questions or concerns.

Prerequisites

The primary prerequisite is mathematical maturity. You should be comfortable reading and writing proofs. Some familiarity with the basics of algorithms, the theory of computation, and probability is expected.

If you are unsure about whether this class is suitable for you, please contact the instructor via email.

Resources

We will not follow a single textbook. The references below complement the lectures; suggested sections and notes appear in the lecture plan. They are alternative and supplementary explanations, not a requirement to read every listed source.

Background packet. Mathematical Background for Information Theory and Applications in Computer Science (PDF). Prerequisite review and optional practice exercises.
CT. Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, second edition (2006). References below use the second edition; chapter numbering differs from the first edition. Publisher’s table of contents (PDF).
PW. Yury Polyanskiy and Yihong Wu, Information Theory: From Coding to Learning (October 2022 draft, PDF). Section numbers below refer to this freely available draft, not the later published edition.
Vadhan. Salil Vadhan, Pseudorandomness (2012). Codes, extractors, and hardness versus randomness; the author provides the monograph and individual chapters.
Haitner. Iftach Haitner, Information Theory and Applications to Computer Science (Fall 2015). Course notes on entropy methods, parallel repetition, and cryptographic applications.
CMU ITCS. Venkatesan Guruswami and Mahdi Cheraghchi, Information Theory and its Applications in Theory of Computation (Spring 2013). Coding, graph entropy, communication complexity, and parallel repetition.
CMU Coding. Venkatesan Guruswami, Coding Theory (Fall 2018). Notes on polar codes, Reed–Solomon codes, and list decoding.
TIFR. Prahladh Harsha, Meena Mahajan, and Jaikumar Radhakrishnan, Communication Complexity (2011–12). Lower bounds, information cost, direct sums, and protocol compression.

Lecture Plan (Subject to Change)

The opening four lectures develop the tools used throughout the course: finite-source compression; information bounds on inference; asymptotic compression; and reliable communication. The later lectures apply these tools in coding, combinatorics, complexity, and cryptography.

Calendar: There is no class on Monday, October 12. Class 5 meets on Wednesday, October 14, following NYU’s Monday schedule. See the NYU academic calendar. Quiz and end-of-course project presentation dates will be announced separately; lecture pacing may be adjusted to accommodate them.

Planned topics and suggested readings. CT uses the second edition; PW uses the October 2022 draft linked above.
Class	Date	Topic	Suggested Reading
1	Sep 14	Entropy, divergence, and coding. Entropy and KL divergence; Gibbs’ inequality; Kraft’s inequality and prefix coding; conditional entropy and information chain rules; entropy counting.	CT §§2.1–2.6, 5.1–5.4. PW §§1.1, 2.2, 3.1–3.4, 10.3; §8.1 for the counting application. Haitner Lesson 1 and Haitner Lesson 7.
2	Sep 21	Information, inference, and indistinguishability. Data processing; Fano’s inequality; coordinate-recovery and storage lower bounds; total variation, Pinsker’s inequality, and average encoding; a testing application.	CT §§2.7–2.10. PW §§2.5, 3.5, 6.3, 6.5, 7.3–7.6. Haitner Lesson 2 and Haitner Lesson 7. Average encoding applies Pinsker to the joint and product distributions.
3	Sep 28	Typicality, types, and the compression threshold. AEP and typical sets; fixed-length compression with error, achievability and strong converse; type classes and divergence exponents. Universal compression as supplementary reading.	CT §§3.1–3.3, 11.1–11.3. PW §11.1; §13.2 for optional universal compression. Haitner Lesson 4.
4	Oct 5	Channels, capacity, and information converses. Finite memoryless channels; BSC and BEC; repetition coding; capacity converse; information density and random coding; maximal error and source–channel separation.	CT §§7.1–7.7, 7.9, 7.13. PW §§17.1–17.4, 18.1–18.2, 19.1–19.2. Haitner Lesson 5. PW gives the information-density route; CT gives a typicality-based alternative.
—	Oct 12	Fall break — no class.	The Monday meeting is held on Wednesday, October 14.
5	Oct 14 (Wed)	Coding at the Shannon/Hamming interface. Linear codes, concatenation, and polar codes.	PW §§11.2, 18.6. CMU ITCS: February 19–21 and February 26. CMU Coding: Polar Codes, Parts 1 and 2.
6	Oct 19	List decoding and local decoding. Reed–Solomon and folded Reed–Solomon codes; locally decodable and locally list-decodable codes.	Vadhan §§5.1–5.2, 7.5–7.6. CMU Coding: List Decoding and Reed–Solomon List Decoding.
7	Oct 26	Entropy in combinatorics. Shearer’s lemma, graph entropy, and entropy compression.	PW §§1.5, 8.1–8.4. Haitner Lesson 3 and Haitner Lesson 6. CMU ITCS: March 19 and March 21.
8	Nov 2	Communication complexity I. Deterministic, randomized, and distributional models; discrepancy and indexing lower bounds.	CMU ITCS: March 28, April 2–4, and April 9. TIFR: Lectures 1–5 for models and basic lower-bound methods.
9	Nov 9	Communication complexity II. Set disjointness, information cost, direct sums, and protocol compression.	CMU ITCS: April 11 and April 16. TIFR: Lectures 14–17 on information complexity and compression.
10	Nov 16	Parallel repetition and direct products. Information-theoretic proof methods for repetition.	Haitner Lesson 9 (interactive arguments). CMU ITCS: April 30 and Lecture 25/Lecture 26 (two-prover games).
11	Nov 23	Kolmogorov complexity and other entropy measures. Description length, incompressibility, and the relationship to Shannon entropy.	CT §§14.1–14.5. Haitner Lesson 8.
12	Nov 30	Randomness extractors. Min-entropy, the leftover hash lemma, expanders, and connections between extractors and codes.	Vadhan §§6.1–6.3; §§4.1 and 5.3 for expander and code connections.
13	Dec 7	Hardness amplification and pseudorandomness. Worst-case to average-case reductions and pseudorandom generators from hardness.	Vadhan §§7.1–7.6. Haitner Lesson 10 (hardcore predicates).
14	Dec 14	Cryptographic entropy and synthesis. Pseudoentropy, accessible entropy, and commitments; connections across the course.	Haitner Lesson 11 and Haitner Lesson 12. Vadhan §7.2 for background on cryptographic PRGs.

Course Policies

Homework

Homework should be submitted in PDF form in Gradescope. We prefer homework submissions typeset in LaTeX. If you are not familiar with LaTeX, it is a great skill to learn. Overleaf provides a simple web interface for writing and compiling LaTeX (as well as extensive documentation). We will provide LaTeX source for you to edit. You are encouraged to insert scanned figures or illustrations where appropriate. Scanned handwritten submissions will only be graded if perfectly legible. If you are unsure about your handwriting, I strongly suggest you type your solutions.

An important part of this class is about learning to communicate your mathematical ideas and proofs clearly and concisely. Accordingly, you will be graded not simply for correctness, but also clarity.

Collaboration

We strongly encourage you to discuss assignments with up to 3 peers, but you must (a) list the names of your discussion partners on your submission, and (b) you must write up your solution on your own. You may not look at the written solutions of any other student before submitting your own solution. If you do not list the names of your collaborators, you will be penalized.

Late Policy

Revisions may be submitted before the final deadline specified for each assignment. Late final submissions will not be accepted; the lowest homework score will be dropped.

External Resources

Acknowledge any external resources consulted in your homework. You must write your own proofs. External tools to obtain homework solutions, consulting solutions from other students or elsewhere, or asking an LLM to solve an assigned problem are not allowed.

Academic Integrity

Your work should be your own. Students should be aware of the CS Department's Policy on Academic Integrity. Violations of academic integrity will not be tolerated.

Religious Observance

As a nonsectarian, inclusive institution, NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations. The policy and principles to be followed by students and faculty may be found in the University Calendar Policy on Religious Holidays.

Disability Disclosure

Academic accommodations are available to any student with a chronic, psychological, visual, mobility, learning disability, or who is deaf or hard of hearing. Students should please register with the Moses Center for Students with Disabilities.