In this post, we will see the book *Mathematical foundations of Information Theory* by A. I. Khinchin.

# About the book

The book is a translation of two papers written by the Russian mathematician, A. I. Khinchin, for the expository journal Uspekhi. These papers present the mathematical foundations of information theory. While completely rigorous, the flavour of the engineering applications which led to the theory runs throughout and very much helps the intuition. Khinchin has here reformulated basic concepts and presents for the first time rigorous proofs of certain fundamental theorems in the subject.

The first paper discusses the concept of entropy and gives one major application to coding. The only stochastic processes used are Markov chains. This paper would serve as a valuable supplement to an introductory probability course.

The second and longer paper uses more advanced topics from probability theory, for example, stationary processes and martingales. However, the treatment is quite complete and the non specialist would not suffer thanks to Khinchin’s amazing expository ability. It is a tribute to Shannon’s theory that a rigorous treatment only enhances the elegance of the basic theorems.

The book was translated from Russian by R. A. Silverman and M. D. Friedman was published in 1957.

Credits to original uploader.

You can get the book here.

Follow us on The Internet Archive: https://archive.org/details/@mirtitles

Follow Us On Twitter: https://twitter.com/MirTitles

Write to us: mirtitles@gmail.com

Fork us at GitLab: https://gitlab.com/mirtitles/

Add new entries to the detailed book catalog here.

# Contents

# The Entropy Concept in Probability Theory

## # 1. Entropy of Finite Schemes 2

## # 2. The Uniqueness Theorem 9

## # 3. Entropy of Markov chains 13

## # 4. Fundamental Theorems 16

## #5. Application to Coding Theory 23

# On the Fundamental Theorems of Information Theory

## INTRODUCTION: 30

## CHAPTER I. Elementary Inequalities 34

## # 1. Two generalizations of Shannon’s inequality 34

## # 2. Three inequalities of Feinstein 39

## CHAPTER II. Ergodic Sources 44

## # 3. Concept of a source. Stationarity. Entropy. 44

## # 4. Ergodic Sources 49

## #5. The E property. McMillan’s theorem. 54

## # 6. The martingale concept. Doob’s theorem. 58

## % 7. Auxiliary propositions 64

## # 8. Proof of McMillan’s theorem 70

## CHAPTER III. Channels and the sources driving them 75

## # 9. Concept of channel. Noise. Stationarity. Anticipation 75

## #10. Connection of the channel to the source 78

## #11. The ergodic case 85

## CHAPTER IV. Feinstein’s Fundamental Lemma 90

## #12. Formulation of the problem 90

## #13. Proof of the lemma 93

## CHAPTER V. Shannon’s Theorems 102

## # 14. Coding 102

## # 15. The first Shannon theorem 104

## #16. The second Shannon theorem 109

CONCLUSION 111

REFERENCES 120

Pingback: Mathematical Foundations of Information Theory – Khinchin | Chet Aero Marine