> If someone tells you a fact you already know, then it could be communicating several different things.
You are alluding to the "meaning of communication" while information and communication theory is strictly about bits that are stored somewhere or/and send over the channel. If you know with 100% certainty that the next bit will be zero then receiving this bit gives you no information beyond what you already knew. Of course, if your certainty is less than 100% then the bit delivered you some fractional quantity of information. Mathematically, communication channel couples together random process X at the transmitter with random process Y at the receiver. The game is to describe X while observing only Y. Mutual information is a quantitative measure of how much Y "knows" about X. There is no human meaning attached to it whatsoever.
You are alluding to the "meaning of communication" while information and communication theory is strictly about bits that are stored somewhere or/and send over the channel. If you know with 100% certainty that the next bit will be zero then receiving this bit gives you no information beyond what you already knew. Of course, if your certainty is less than 100% then the bit delivered you some fractional quantity of information. Mathematically, communication channel couples together random process X at the transmitter with random process Y at the receiver. The game is to describe X while observing only Y. Mutual information is a quantitative measure of how much Y "knows" about X. There is no human meaning attached to it whatsoever.