Information flow (information theory)

Information flow in an information theoretical context is the transfer of information from a variable "h" to a variable "l" in a given process. The measure of information flow, "p", is defined as the uncertainty before the process started minus the uncertainty after the process has terminated. This can be quantified as

:$H(h|l)\; -\; H(h|l\text{'})\; stackrel\{mathrm\{def\{=\}\; (H(h,l)\; -\; H(l))\; -\; (H(h,l\text{'})\; -\; H(l\text{'})),!$

where $H(h|l)$ is the conditional entropy (equivocation) of variable "h" (before the process started) given the variable "l" (before the process started), and $H(h|l\text{'})$ is the conditional entropy (equivocation) of variable "h" (before the process started) given the variable "l"' (the value of variable "l" after the process finished).

$H(X,Y)$ is the joint entropy, and can be calculated as follows:

:$H(X,Y)\; =\; -sum\_\{x,y\}\; p\_\{x,y\}\; log(p\_\{x,y\})\; !$