Fundamentals of Information Theory and Coding Design

In a clear, concise, and modular format, this book introduces the fundamental concepts and mathematics of information and coding theory. The authors emphasize how a code is designed and discuss the main properties and characteristics of different coding algorithms along with strategies for selecting the appropriate codes to meet specific requirements. They provide comprehensive coverage of source and channel coding, address arithmetic, BCH, and Reed-Solomon codes and explore some more advanced topics such as PPM compression and turbo codes. Worked examples and sets of basic and advanced exercises in each chapter reinforce the text's clear explanations of all concepts and methodologies.

Books on information theory and coding have proliferated over the last few years, but few succeed in covering the fundamentals without losing students in mathematical abstraction. Even fewer build the essential theoretical framework when presenting algorithms and implementation details of modern coding systems.Without abandoning the theoretical foundations, Fundamentals of Information Theory and Coding Design presents working algorithms and implementations that can be used to design and create real systems. The emphasis is on the underlying concepts governing information theory and the mathematical basis for modern coding systems, but the authors also provide the practical details of important codes like Reed-Solomon, BCH, and Turbo codes. Also setting this text apart are discussions on the cascading of information channels and the additivity of information, the details of arithmetic coding, and the connection between coding of extensions and Markov modelling.Complete, balanced coverage, an outstanding format, and a wealth of examples and exercises make this an outstanding text for upper-level students in computer science, mathematics, and engineering and a valuable reference for telecommunications engineers and coding theory researchers.

ENTROPY AND INFORMATION StructureStructure in RandomnessFirst Concepts of Probability Theory Surprise and EntropyUnits of EntropyThe Minimum and Maximum Values of Entropy A Useful Inequality Joint Probability Distribution FunctionsConditional Probability and Bayes' Theorem Conditional Probability Distributions and Conditional Entropy Information Sources Memoryless Information Sources Markov Sources and n-Gram Models Stationary Distributions The Entropy of Markov Sources Sequences of Symbols The Adjoint Source of a Markov Source Extensions of Sources Infinite Sample Spaces INFORMATION CHANNELS What Are Information Channels? BSC and BEC ChannelsMutual Information Noiseless and Deterministic Channels Cascaded Channels Additivity of Mutual Information Channel Capacity: Maximum Mutual InformationContinuous Channels and Gaussian Channels Information Capacity Theorem Rate Distortion Theory SOURCE CODING Introduction Instantaneous CodesThe Kraft Inequality and McMillan's TheoremAverage Length and Compact CodesShannon's Noiseless Coding Theorem Fano CodingHuffman CodingArithmetic Coding Higher-Order ModellingDATA COMPRESSION Introduction Basic Concepts of Data Compression Run-Length Coding The CCITT Standard for Facsimile Transmission Block-sorting CompressionDictionary Coding Statistical CompressionPrediction by Partial Matching Image CodingFUNDAMENTALS OF CHANNEL CODING Introduction Code RateDecoding Rules Hamming Distance Bounds on M, Maximal Codes and Perfect CodesError Probabilities Shannon's Fundamental Coding Theorem ERROR-CORRECTING CODES Introduction Groups Rings and FieldsLinear SpacesLinear Spaces over the Binary FieldLinear CodesEncoding and DecodingCodes Derived from Hadamard Matrices CYCLIC CODES Introduction Rings of PolynomialsCyclic CodesEncoding and Decoding of Cyclic Codes Encoding and Decoding Circuits for Cyclic CodesThe Golay CodeHamming CodesCyclic Redundancy Check Codes Reed-Muller Codes BURST-CORRECTING CODES Introduction Finite FieldsIrreducible PolynomialsConstruction of Finite FieldsBursts of Errors Fire Codes Minimum Polynomials Bose-Chaudhuri-Hocquenghem CodesOther Fields Reed-Solomon Codes CONVOLUTIONAL CODES Introduction ASimple Example Binary Convolutional CodesDecoding Convolutional CodesThe Viterbi Algorithm Sequential DecodingTrellis Modulation Turbo CodesINDEXEach chapter also contains a section of exercises and a section of references.