Mathematical Foundations of Information Theory – Khinchin

In this post, we will see the book Mathematical foundations of Information Theory by A. I. Khinchin.

About the book

The book is a translation of two papers written by the Russian mathematician, A. I. Khinchin, for the expository journal Uspekhi. These papers present the mathematical foundations of information theory. While completely rigorous, the flavour of the engineering applications which led to the theory runs throughout and very much helps the intuition. Khinchin has here reformulated basic concepts and presents for the first time rigorous proofs of certain fundamental theorems in the subject.

The first paper discusses the concept of entropy and gives one major application to coding. The only stochastic processes used are Markov chains. This paper would serve as a valuable supplement to an introductory probability course.

The second and longer paper uses more advanced topics from probability theory, for example, stationary processes and martingales. However, the treatment is quite complete and the non specialist would not suffer thanks to Khinchin’s amazing expository ability. It is a tribute to Shannon’s theory that a rigorous treatment only enhances the elegance of the basic theorems.

The book was translated from Russian by R. A. Silverman  and M. D. Friedman was published in  1957.

Credits to original uploader.

You can get the book here.

Follow us on The Internet Archive:

Follow Us On Twitter:

Write to us:

Fork us at GitLab:

Add new entries to the detailed book catalog here.



The Entropy Concept in Probability Theory

# 1. Entropy of Finite Schemes 2

# 2. The Uniqueness Theorem 9

# 3. Entropy of Markov chains 13

# 4. Fundamental Theorems 16

#5. Application to Coding Theory 23

On the Fundamental Theorems of Information Theory


CHAPTER I. Elementary Inequalities 34

# 1. Two generalizations of Shannon’s inequality 34

# 2. Three inequalities of Feinstein 39

CHAPTER II. Ergodic Sources 44

# 3. Concept of a source. Stationarity. Entropy. 44

# 4. Ergodic Sources 49

#5. The E property. McMillan’s theorem. 54

# 6. The martingale concept. Doob’s theorem. 58

% 7. Auxiliary propositions 64

# 8. Proof of McMillan’s theorem 70

CHAPTER III. Channels and the sources driving them 75

# 9. Concept of channel. Noise. Stationarity. Anticipation 75

#10. Connection of the channel to the source 78

#11. The ergodic case 85

CHAPTER IV. Feinstein’s Fundamental Lemma 90

#12. Formulation of the problem 90

#13. Proof of the lemma 93

CHAPTER V. Shannon’s Theorems 102

# 14. Coding 102

# 15. The first Shannon theorem 104

#16. The second Shannon theorem 109




About The Mitr

I am The Mitr, The Friend
This entry was posted in books, engineering, mathematics, physics, statistics, technology and tagged , , , , , , , , , , , , , . Bookmark the permalink.

1 Response to Mathematical Foundations of Information Theory – Khinchin

  1. Pingback: Mathematical Foundations of Information Theory – Khinchin | Chet Aero Marine

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.