Göm meny

Information theory

Course manager: Harald Nautsch

Suggested credits: 8 hp

Course literature

The course will be based on the book Elements of Information Theory by Thomas Cover & Joy Thomas. The most recent edition is the 2nd edition, with ISBN 0471241954, but it is also possible to use the 1st edition, with ISBN 0471062596.

Organisation

Lectures will be given by Harald and by the course participants.

Examination

In order to pass, each participant should give one lecture. Subjects will be assigned before the first lecture.

After each lecture a number of problems to solve will be given. The problems should be solved individually by each participant, but it's ok to discuss the problems with each other. You should turn in your solutions no later than a week after they are given.

Lectures

Preliminary lecture schedule:
No Lecturer Subject Chapter(s) Time Room
1 Harald Introduction, entropy, mutual information, AEP 1-3 Tue 9/4 13-15 Algoritmen
2 Harald Entropy rates of stochastic processes, differential entropy 4, 8 Tue 16/4 13-15 Algoritmen
3 Jonathan Data compression 5 Tue 23/4 13-15 Algoritmen
4 Michael Channel capacity, channel coding theorem I 2.10, 7 Wed 24/4 10-12 Systemet
5 Asad Channel capacity, channel coding theorem II 2.10, 7 Fri 3/5 10-12 Algoritmen
6 Jens Gaussian channel 9 Wed 8/5 10-12 Systemet
7 André/George Information theory and statistics 11 Thu 16/5 10-12 Algoritmen
8 Tohid Maximum entropy 12 Wed 22/5 10-12 Algoritmen
9 Tai Network information theory I 15 (channel parts) Tue 28/5 13-15 Algoritmen
10 Fahim Network information theory II 15 (channel parts) Wed 29/5 10-12 Algoritmen
11 Gustaf Rate-distortion theory 10 Thu 30/5 13-15 Algoritmen
12 Harald Network source coding, universal source coding 13, 15.4, 15.5, 15.8, 15.9 Tue 4/6 13-15 Algoritmen
Chapter numbers are given according to the second edition of the book. The first edition has a slightly different chapter structure (see below).

Course material

Extra course material, such as lecture slides, will be published here.
Slides from lecture 1 PDF
Slides from lecture 2 PDF
Slides from lecture 3 PDF
Slides from lecture 4 PDF
Slides from lecture 5 PDF
Slides from lecture 6 PDF
Slides from lecture 7, part 1 PDF
Slides from lecture 7, part 2 PDF
Slides from lecture 7, part 3 PDF
Slides from lecture 8 PDF
Slides from lecture 9 PDF
Slides from lecture 10 PDF
Slides from lecture 12 PDF

Problems

Problems will be published here after each lecture. All book references refer to the second edition. In order to pass the course you should solve at least 75% of the total problems and at least 60% of the problems for each subject.
Lecture(s) Problems
1 2.4, 2.5, 2.6, 2.12, 2.35, 3.13ab (NOTE: The table on page 69 is wrong!)
2 4.2, 4.6, 4.7, 8.1, 8.4
3 5.4 (also calculate the expected code length for (c)), 5.6, 5.8, 5.12, 5.25, 5.30
4-5 7.4, 7.8, 7.13, 7.16, 7.19
6 9.3, 9.4 (assume Xi>=0), 9.6, 9.9
7 11.12, 11.14, 11.15, 11.18
8 12.1, 12.3, 12.4
9-10,12 15.2, 15.8, 15.13, 15.17, 15.18
11 10.2, 10.5, 10.14

Comparison between book editions

The chapters in the two editions of the book correspond to each other as follows
2nd ed. 1st ed.
1 1
2 2
3 3
4 4
5 5.1-5.9, 5.11-5.12
6 6
7 8
8 9
9 10
10 13
11 12.1-12.9, 12.11
12 11
13 5.10, 12.10 + new material on Lempel-Ziv coding
14 7
15 14
16 15
17 16

Informationsansvarig: Harald Nautsch
Senast uppdaterad: 2013-06-25