In this post, we will see the book Mathematical foundations of Information Theory by A. I. Khinchin.

About the book

The book is a translation of two papers written by the Russian mathematician, A. I. Khinchin, for the expository journal Uspekhi. These papers present the mathematical foundations of information theory. While completely rigorous, the flavour of the engineering applications which led to the theory runs throughout and very much helps the intuition. Khinchin has here reformulated basic concepts and presents for the first time rigorous proofs of certain fundamental theorems in the subject.

The first paper discusses the concept of entropy and gives one major application to coding. The only stochastic processes used are Markov chains. This paper would serve as a valuable supplement to an introductory probability course.

The second and longer paper uses more advanced topics from probability theory, for example, stationary processes and martingales. However, the treatment is quite complete and the non specialist would not suffer thanks to Khinchin’s amazing expository ability. It is a tribute to Shannon’s theory that a rigorous treatment only enhances the elegance of the basic theorems.

The book was translated from Russian by R. A. Silverman  and M. D. Friedman was published in  1957.

Credits to original uploader.

You can get the book here.

Follow us on The Internet Archive: https://archive.org/details/@mirtitles

Follow Us On Twitter: https://twitter.com/MirTitles

Write to us: mirtitles@gmail.com

Fork us at GitLab: https://gitlab.com/mirtitles/

Add new entries to the detailed book catalog here.

Contents

The Entropy Concept in Probability Theory

# 1. Entropy of Finite Schemes 2

# 2. The Uniqueness Theorem 9

# 3. Entropy of Markov chains 13

# 4. Fundamental Theorems 16

#5. Application to Coding Theory 23

On the Fundamental Theorems of Information Theory

INTRODUCTION: 30

CHAPTER I. Elementary Inequalities 34

# 1. Two generalizations of Shannon’s inequality 34

# 2. Three inequalities of Feinstein 39

CHAPTER II. Ergodic Sources 44

# 3. Concept of a source. Stationarity. Entropy. 44

# 4. Ergodic Sources 49

#5. The E property. McMillan’s theorem. 54

# 6. The martingale concept. Doob’s theorem. 58

% 7. Auxiliary propositions 64

# 8. Proof of McMillan’s theorem 70

CHAPTER III. Channels and the sources driving them 75

# 9. Concept of channel. Noise. Stationarity. Anticipation 75

#10. Connection of the channel to the source 78

#11. The ergodic case 85

CHAPTER IV. Feinstein’s Fundamental Lemma 90

#12. Formulation of the problem 90

#13. Proof of the lemma 93

CHAPTER V. Shannon’s Theorems 102

# 14. Coding 102

# 15. The first Shannon theorem 104

#16. The second Shannon theorem 109

CONCLUSION 111

REFERENCES 120