Dual total correlation

Venn diagram of information theoretic measures for three variables x, y, and z. The dual total correlation is represented by the union of the three mutual informations and is shown in the diagram by the yellow, magenta, cyan, and gray regions.

For a set of n random variables ${\displaystyle \{X_{1},\ldots ,X_{n}\}}$, the dual total correlation ${\displaystyle D(X_{1},\ldots ,X_{n})}$ is given by

${\displaystyle D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),}$

where ${\displaystyle H(X_{1},\ldots ,X_{n})}$ is the joint entropy of the variable set ${\displaystyle \{X_{1},\ldots ,X_{n}\}}$ and ${\displaystyle H(X_{i}\mid \cdots )}$ is the conditional entropy of variable ${\displaystyle X_{i}}$, given the rest.

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value ${\displaystyle H(X_{1},\ldots ,X_{n})}$,

${\displaystyle ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.}$

Dual total correlation is non-negative and bounded above by the joint entropy ${\displaystyle H(X_{1},\ldots ,X_{n})}$.

${\displaystyle 0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).}$

Secondly, Dual total correlation has a close relationship with total correlation, ${\displaystyle C(X_{1},\ldots ,X_{n})}$. In particular,

${\displaystyle {\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).}$

In measure theoretic terms, by the definition of dual total correlation:

${\displaystyle D(X_{1},\ldots ,X_{n})=\mu \left(\bigcup _{i}{\tilde {X}}_{i}\setminus \left(\bigcup _{j}{\tilde {X}}_{j}\setminus \bigcup _{k\neq j}{\tilde {X}}_{k})\right)\right)}$

which is equal to the union of the pairwise mutual informations:

${\displaystyle D(X_{1},\ldots ,X_{n})=\mu \left(\bigcup _{i}\bigcup _{j\neq i}\left({\tilde {X}}_{i}\cap {\tilde {X}}_{j}\right)\right)}$

Han (1978) originally defined the dual total correlation as,

${\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\;.\end{aligned}}}$

However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:

${\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\\={}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]+(1-n)\;H(X_{1},\ldots ,X_{n})\\={}&H(X_{1},\ldots ,X_{n})+\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})-H(X_{1},\ldots ,X_{n})\right]\\={}&H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right)\;.\end{aligned}}}$

• Mutual information
• Total correlation

• Han T. S. (1978). Nonnegative entropy measures of multivariate symmetric correlations, Information and Control 36, 133–156.
• Fujishige Satoru (1978). Polymatroidal Dependence Structure of a Set of Random Variables, Information and Control 39, 55–72. doi:10.1016/S0019-9958(78)91063-X.
• Dubnov S. (2006). Spectral anticipations, Computer Music Journal, 30(2):63–83.
• Olbrich, E. and Bertschinger, N. and Ay, N. and Jost, J. (2008). How should complexity scale with system size?, The European Physical Journal B - Condensed Matter and Complex Systems. doi:10.1140/epjb/e2008-00134-9.
• Abdallah S. A. and Plumbley, M. D. (2010). A measure of statistical complexity based on predictive information, ArXiv e-prints. arXiv:1012.1890v1.
• Nihat Ay, E. Olbrich, N. Bertschinger (2001). A unifying framework for complexity measures of finite systems. European Conference on Complex Systems. pdf.