On the (information) value of having whale steak for dinner
A great example of academic snark in the abstract of Ronald Howard‘s classic paper “Information value theory” (IEEE Transactions on Systems Science and Cybernetics, vol. SSC-2, no. 1, pp. 22-26, 1966):
The information theory developed by Shannon was designed to place a quantitative measure on the amount of information involved in any communication. The early developers stressed that the information measure was dependent only on the probabilistic structure of the communication process. For example, if losing all your asset in the stock market and having whale steak for supper have the same probability, then the information associated with the occurrence of either event is the same. Attempts to apply Shannon’s information theory to problems beyond communications have, in the large, come to grief. The failure of these attempts could have been predicted because no theory that involves just the probabilities of outcomes without considering their consequences could possibly be adequate in describing the importance of uncertainty to a decision maker.
Incidentally, to this day there is no completely convincing approach to quantifying the value of information in a decision-making scenario. The obvious idea — namely, looking at the reduction in the expected loss due to acquiring additional information — goes only so far, since more or less the only property this putative measure has is the monotonicity with respect to increasing knowledge. Now compare and contrast this with the elaborate additivity properties of the Shannon-theoretic quantities! On the other hand, it seems that the work of Grünwald and Dawid that I had mentioned in a previous post may be a glimmer of hope. We will have to see.