Disclaimer: I am not a biologist, but I have become interested in biology and related matters over the past couple of years. One reason is obviously the pandemic, so the talk of biology, viruses, mRNA, and the like is everywhere. The other, main, reason is that I think we will not get anywhere interesting in AI unless we understand the concepts of autonomy, self-directedness, integration, and adaptation in even very simple biological systems.
We are increasingly employing information as an explanation of phenomena outside the world of culture and technology — as the central metaphor with which to talk about the nature of life and mind. Molecular biology, for instance, tells us how genetic information is transferred from one generation to the next, and from one cell to the next. And neuroscience is trying to tell us how information from the external world and the body percolates through the brain, influencing behavior and giving rise to conscious experience.
But do we really know what information is in the first place? And is it really a helpful way to think about biological phenomena? I’d like to argue that explanations of natural phenomena that involve information make inappropriate use of our latent, unexamined intuitions about inter-personal communication, blurring the line between what we understand and what we don’t quite have a grip on yet.
Similar sentiments are quoted by Carl Bergstrom and Martin Rosvall:
Biologists think in terms of information at every level of investigation. Signaling pathways transduce information, cells process information, animal signals convey information. Information flows in ecosystems, information is encoded in the DNA, information is carried by nerve impulses. In some domains the utility of the information concept goes unchallenged: when a brain scientist says that nerves transmit information, nobody balks. But when geneticists or evolutionary biologists use information language in their day-to-day work, a few biologists and many philosophers become anxious about whether this language can be justified as anything more than facile metaphor.
Yohan argues that information theory is, on the whole, not an appropriate framework with which to reason about biological information. Carl and Martin argue otherwise, but propose their own framework, what they refer to as the transmission sense of information, which purportedly resolves the issues that trouble “a few biologists and many philosophers.” My goal in this series of posts is to argue that information theory can indeed be applied to biology, but that its proper application needs to be built up from first principles, starting with a serious engagement with its entire conceptual framework. Moreover, I agree with Yohan that digital communication is not the right conceptual schema; instead, we should be talking about control, programmability, and behaviors.
1. Operationalization has to come first
In one big way, Yohan is right — one has to search far and wide for applications of information theory to biology that rise above the level of facile metaphors. What he does not talk about is the reason, which is essentially the same reason why many attempts to apply of information theory outside its original domain of communication systems largely fail. This has to do with the fact that, in information theory, the starting point is always an operational formulation in terms of various performance criteria that are not phrased in information-theoretic language. For instance, if we talk about data compression, then we bring in the considerations of code length, probability of error, compression ratio, etc.; if we talk about channel coding, then we are interested in the maximal rate of transmission subject to a given level of bit or block errors; if we talk about source coding, then we care about expected distortion, codebook size, etc. Note that none of these criteria are phrased in terms of information-theoretic quantities like entropy, conditional entropy, or mutual information. This is not accidental, and the whole enterprise of information theory is to express the fundamental limits of communication systems quantitatively in terms of these quantities. In other words, the pragmatics of communication comes first, the information-theoretic analysis comes second.
When it comes to applying information theory to biology, this important point somehow gets lost in translation, although some biologists do emphasize it — see, e.g., this paper by Olivier Rivoire. Bergstrom and Rosvall recognize this as well, but, in my opinion, do not explore the implications deeply enough. So, if we do want to get anywhere, we have to start with an operational formulation. In the context of molecular and evolutionary biology, which is primarily what I am interested in here, we will need to formulate our operational criteria at the level of an organism as the subject and object of evolution.
2. Genotypes as behaviors, phenotypes as constrained trajectories
The temptation to apply information theory to molecular and evolutionary biology is understandable: The term “genetic code” all but forces one to think in terms of information in the sense of Shannon! However, before we begin, we need to question the communication framework itself. Even granting the language of codes, we need to identify the problem to which the genetic code is a solution. Bergstrom and Rosvall argue that the problem is one of reliable and maximally efficient transmission of the genome from one generation to the next. However, if that were the only problem, this does not answer C.H. Waddington’s question “why animals should have evolved all sorts of highly adaptive structures to do unlikely things instead of simply being reduced to bags of eggs and sperm like certain parasitic worms.” In other words, we need to discuss the genotype and the phenotype. Once we bring the phenotype into the picture, the communication metaphor dissolves in favor of one of control and homeorhesis. In this interpretation, again due to Waddington, the genome is not just a message to be transmitted through time and space, it is a set of instructions (or a program) which, together with some aspects of the organism’s environment and developmental noise, will be used to construct (or realize) the organism’s phenotype out of the set of potential phenotypes. Indeed, biological organisms are highly constrained, programmable open systems which, throughout the course of their development and life, maintain certain relations with their environment. This includes not only the environment affecting the organism, but also the organism affecting and constraining its environment (niche construction).
This suggests thinking about genomes as specifying a behavior in the sense of Jan Willems. To get the main point across, it suffices to paint a very broad picture. Let denote the genome, the configuration of the organism, the configuration of the environment, and the developmental noise. Then the role of the genome is to act as the organism’s formal cause, i.e., to constrain the relation between all the other variables. In other words, treating everything except the genome as a trajectory in time, we can formalize the relations between the organism and its environment as a certain subset of the product space
where, e.g., denotes the set of all trajectories through the configuration space of the organism over its lifetime (here I am, of course, drastically oversimplifying things, ignoring, for instance, the fact that the lifetime of the organism is generally determined by its trajectory and its interaction with the environment). To be a bit more concrete, we may say that a tuple of trajectories is in if there exist suitably defined functions , , and , such that
(i.e., the configuration of the organism is determined by its genome, by its environment, and by developmental noise, possibly with feedback from the organism, subject to appropriate causality restrictions) and
(i.e., some aspects of the environment are shaped by the organism by interaction or coupling via the phenotype but not directly through the genotype). Then the phenotype is any trajectory that is compatible with these constraints, i.e., which could arise in this fashion for some realization of the environment and the developmental noise . Notice that this description introduces mutual constraints between the organism and its environment. So, if one were to think of the genome as a message to be transmitted in space and in time, then this message is a program (formal cause) of the organism, but it does not (indeed, cannot!) contain all the information about the phenotype. The missing information is contained in the environment and in the developmental noise.
Note, by the way, that there is nothing specifically biological in the above description. Indeed, all this time I could have been talking about a Turing machine, where describes its state transition and output functions, is its internal state, is the content of its tape, and is any randomness in the state transitions. In order to bring the biology back into the picture, we could follow Robert Rosen‘s characterization of biological organisms in terms of closure with respect to efficient causation, i.e., the organism’s internal ability to maintain certain relations and constraints with its environment (which includes itself). However, this illustration, I think, provides a correction to the naive picture of the genome “computing” the organism just like a Turing machine computing a function, rightly criticized by Yohan John. A better picture is the one above: the genome prescribes the operation of the Turing machine, the environment provides the initial input and records the output of the Turing machine, and the state sequence generated epigenetically for that particular environment is the phenotype.
This also addresses, I believe, Waddington’s objection to the use of information theory in biology on the grounds of the data processing inequality:
It seems quite obvious to common sense that a rabbit running around in a field contains a much greater `amount of variety’ than a newly fertilized rabbit’s egg. How are we to deal with the situation in terms of an `information theory’ whose basic tenet is that information cannot be gained?
The amount of information contained in the phenotype is, of course, finite, but it only acts as the formal cause that determines the entire behavior ; it cannot by itself select a particular trajectory in the behavior. Hence, if we wish to operationalize the problem of reliable transmission of genetic information, we need to phrase it in terms of criteria pertainig to preserving the behavior as a whole. I will take this up, in a simplified form, in subsequent posts; the next post, however, will mostly be about the Bergstrom-Rosvall paper on the transmission sense of information.
- Yohan John, “How informative is the concept of biological information?”
- Carl Bergstrom and Martin Rosvall, “The transmission sense of information”
- Olivier Rivoire, “Information in models of evolutionary dynamics”
- Richard Lewontin, “The organism as the subject and object of evolution”
- Conrad H. Waddington, “The basic ideas of biology”
- Yuuki Matsushita and Kunihiko Kaneko, “Homeorhesis in Waddington’s landscape by epigenetic feedback regulation”
- Jan Willems, “The behavioral approach to open and interconnected systems”
- Robert Rosen, Life Itself: A Comprehensive Inquiry into the Nature, Origin and Fabrication of Life