Compound probability distribution

A compound probability distribution is the probability distribution that results from assuming that a random variable ${\displaystyle X}$ is distributed according to some parametrized distribution ${\displaystyle F}$ with an unknown parameter ${\displaystyle \theta }$ that is again distributed according to some other distribution ${\displaystyle G}$. The resulting distribution ${\displaystyle H}$ is said to be the distribution that results from compounding ${\displaystyle F}$ with ${\displaystyle G}$. The parameter's distribution ${\displaystyle G}$ is also called the mixing distribution or latent distribution. Technically, the unconditional distribution ${\displaystyle H}$ results from marginalizing over ${\displaystyle G}$, i.e., from integrating out the unknown parameter(s) ${\displaystyle \theta }$. Its probability density function is given by:

${\displaystyle p_{H}(x)={\displaystyle \int \limits p_{F}(x|\theta )\,p_{G}(\theta )\operatorname {d} \!\theta }}$

The same formula applies analogously if some or all of the variables are vectors.

From the above formula, one can see that a compound distribution essentially is a special case of a marginal distribution: The joint distribution of ${\displaystyle x}$ and ${\displaystyle \theta }$ is given by ${\displaystyle p(x,\theta )=p(x|\theta )p(\theta )}$, and the compound results as its marginal distribution: ${\displaystyle {\textstyle p(x)=\int p(x,\theta )\operatorname {d} \!\theta }}$. If the domain of ${\displaystyle \theta }$ is discrete, then the distribution is again a special case of a mixture distribution.

A compound distribution ${\displaystyle H}$ resembles in many ways the original distribution ${\displaystyle F}$ that generated it, but typically has greater variance, and often heavy tails as well. The support of ${\displaystyle H}$ is the same as the support of the ${\displaystyle F}$, and often the shape is broadly similar as well. The parameters of ${\displaystyle H}$ include any parameters of ${\displaystyle G}$ or ${\displaystyle F}$ that have not been marginalized out.

The compound distribution's first two moments are given by

${\displaystyle \operatorname {E} _{H}[X]=\operatorname {E} _{G}{\bigl [}\operatorname {E} _{F}[X|\theta ]{\bigr ]}}$

and

${\displaystyle \operatorname {Var} _{H}(X)=\operatorname {E} _{G}{\bigl [}\operatorname {Var} _{F}(X|\theta ){\bigr ]}+\operatorname {Var} _{G}{\bigl (}\operatorname {E} _{F}[X|\theta ]{\bigr )}}$

(Law of total variance).

Compound distributions derived from exponential family distributions often have a closed form. If analytical integration is not possible, numerical methods may be necessary.

Compound distributions may relatively easily be investigated using Monte Carlo methods, i.e., by generating random samples. It is often easy to generate random numbers from the distributions ${\displaystyle p(\theta )}$ as well as ${\displaystyle p(x|\theta )}$ and then utilize these to perform collapsed Gibbs sampling to generate samples from ${\displaystyle p(x)}$.

A compound distribution may usually also be approximated to a sufficient degree by a mixture distribution using a finite number of mixture components, allowing to derive approximate density, distribution function etc.[1]

Parameter estimation (maximum-likelihood or maximum-a-posteriori estimation) within a compound distribution model may sometimes be simplified by utilizing the EM-algorithm.[2]

• Gaussian scale mixtures:[3]
  • Compounding a normal distribution with variance distributed according to an inverse gamma distribution (or equivalently, with precision distributed as a gamma distribution) yields a non-standardized Student's t-distribution.[4] This distribution has the same symmetrical shape as a normal distribution with the same central point, but has greater variance and heavy tails.
  • Compounding a Gaussian distribution with variance distributed according to an exponential distribution (or with standard deviation according to a Rayleigh distribution) yields a Laplace distribution.
  • Compounding a Gaussian distribution with variance distributed according to an exponential distribution whose rate parameter is itself distributed according to a gamma distribution yields a Normal-exponential-gamma distribution. (This involves two compounding stages. The variance itself then follows a Lomax distribution; see below.)
  • Compounding a Gaussian distribution with standard deviation distributed according to a (standard) inverse uniform distribution yields a Slash distribution.
• other Gaussian mixtures:
  • Compounding a Gaussian distribution with mean distributed according to another Gaussian distribution yields (again) a Gaussian distribution.
  • Compounding a Gaussian distribution with mean distributed according to a shifted exponential distribution yields an exponentially modified Gaussian distribution.

• Compounding a binomial distribution with probability of success distributed according to a beta distribution yields a beta-binomial distribution. It possesses three parameters, a parameter ${\displaystyle n}$ (number of samples) from the binomial distribution and shape parameters ${\displaystyle \alpha }$ and ${\displaystyle \beta }$ from the beta distribution.[5][6]
• Compounding a multinomial distribution with probability vector distributed according to a Dirichlet distribution yields a Dirichlet-multinomial distribution.
• Compounding a Poisson distribution with rate parameter distributed according to a gamma distribution yields a negative binomial distribution.[7][8]
• Compounding an exponential distribution with its rate parameter distributed according to a gamma distribution yields a Lomax distribution.[9]
• Compounding a gamma distribution with inverse scale parameter distributed according to another gamma distribution yields a three-parameter beta prime distribution.[10]
• Compounding a half-normal distribution with its scale parameter distributed according to a Rayleigh distribution yields an exponential distribution. This follows immediately from the Laplace distribution resulting as a normal scale mixture; see above. The roles of conditional and mixing distributions may also be exchanged here; consequently, compounding a Rayleigh distribution with its scale parameter distributed according to a half-normal distribution also yields an exponential distribution.
• A Gamma(k=2,θ) - distributed random variable whose scale parameter θ again is uniformly distributed marginally yields an exponential distribution.

• Mixture distribution
• Marginal distribution
• Conditional distribution, Joint distribution
• Compound Poisson distribution, Compound Poisson process
• Convolution
• Overdispersion
• EM-algorithm

• Lindsay, B. G. (1995), Mixture models: theory, geometry and applications, NSF-CBMS Regional Conference Series in Probability and Statistics, 5, Hayward, CA, USA: Institute of Mathematical Statistics, pp. i–163, ISBN 978-0-940600-32-4, JSTOR 4153184
• Seidel, W. (2010), "Mixture models", in Lovric, M. (ed.), International Encyclopedia of Statistical Science, Heidelberg: Springer, pp. 827–829, doi:10.1007/978-3-642-04898-2_368, ISBN 978-3-642-04898-2
• Mood, A. M.; Graybill, F. A.; Boes, D. C. (1974), "III.4.3 Contagious distributions and truncated distributions", Introduction to the theory of statistics (3rd ed.), New York: McGraw-Hill, ISBN 978-0-07-042864-5
• Johnson, N. L.; Kemp, A. W.; Kotz, S. (2005), "8 Mixture distributions", Univariate discrete distributions, New York: Wiley, ISBN 978-0-471-27246-5