Information theory reading group at Duke

To shake things up a bit, I have started an information theory reading group in my department at Duke. We will meet every Monday to read and discuss papers on information theory, both recent and the classics, particularly those pertaining to estimation and inference, learning, decision-making, and control.

We’ve had our first meeting yesterday, where it was decided that the theme for the fall semester of 2010 will be Control. We have also settled on the following papers (all available for download from the reading group website):

  1. J. Massey, “Causality, feedback and directed information”
  2. H. Touchette and S. Lloyd, “Information-theoretic approach to the study of control systems”
  3. J. Walrand and P. Varaiya, “Optimal causal coding-decoding problems”
  4. N. Elia, “When Bode meets Shannon: control-oriented feedback communication schemes”
  5. N. Martins and M. Dahleh, “Feedback control in the presence of noisy channels: ‘Bode-like’ fundamental limitations of performance”
  6. V. Anantharam and V. Borkar, “An information-theoretic view of stochastic resonance”

Most likely, as the meetings continue, I will be posting various blurbs, notes and musings on these papers and any related matters. We may also shuffle things around a bit — for example, spend some time discussing the basics of stochastic control for the benefit of those not familiar with it.

Stochastic kernels of the world, interconnect!

This is an expository post on a particularly nice way of thinking about stochastic systems in information theory, control, statistical learning and inference, experimental design, etc. I will closely follow the exposition in Chapter 2 of Sekhar Tatikonda‘s doctoral thesis, which in turn builds on ideas articulated by Hans Witsenhausen and Roland Dobrushin.

A general stochastic system consists of smaller interconnected subsystems that influence one another’s behavior. For example, in a communications setting we have sources, encoders, channels and decoders; in control we have plants, sensors, and actuators. The formalism I am about to describe will allow us to treat all these different components on an equal footing. Continue reading “Stochastic kernels of the world, interconnect!”