Free energy principle

Active inference is closely related to the good regulator theorem[13] and related accounts of self-organisation,[14][15] such as self-assembly, pattern formation, autopoiesis[16] and practopoiesis[17]. It addresses the themes considered in cybernetics, synergetics[18] and embodied cognition. Because free energy can be expressed as the expected energy of observations under the variational density minus its entropy, it is also related to the maximum entropy principle.[19] Finally, because the time average of energy is action, the principle of minimum variational free energy is a principle of least action.

The objective is to maximise model evidence ${\displaystyle p(s\mid m)}$ or minimise surprise ${\displaystyle -\log p(s\mid m)}$. This generally involves an intractable marginalisation over hidden states, so surprise is replaced with an upper variational free energy bound.[9] However, this means that internal states must also minimise free energy, because free energy is a function of sensory and internal states:

${\displaystyle a(t)={\underset {a}{\operatorname {arg\,min} }}\{F(s(t),\mu (t))\}}$

${\displaystyle \mu (t)={\underset {\mu }{\operatorname {arg\,min} }}\{F(s(t),\mu ))\}}$

${\displaystyle {\underset {\mathrm {free-energy} }{\underbrace {F(s,\mu )} }}={\underset {\mathrm {energy} }{\underbrace {E_{q}[-\log p(s,\psi \mid m)]} }}-{\underset {\mathrm {entropy} }{\underbrace {H[q(\psi \mid \mu )]} }}={\underset {\mathrm {surprise} }{\underbrace {-\log p(s\mid m)} }}+{\underset {\mathrm {divergence} }{\underbrace {D_{\mathrm {KL} }[q(\psi \mid \mu )\parallel p(\psi \mid s,m)]} }}\geq {\underset {\mathrm {surprise} }{\underbrace {-\log p(s\mid m)} }}}$

This induces a dual minimisation with respect to action and internal states that correspond to action and perception respectively.

Free energy minimisation has been proposed as a hallmark of self-organising systems when cast as random dynamical systems.[20] This formulation rests on a Markov blanket (comprising action and sensory states) that separates internal and external states. If internal states and action minimise free energy, then they place an upper bound on the entropy of sensory states

${\displaystyle \lim _{T\to \infty }{\frac {1}{T}}{\underset {\text{free-action}}{\underbrace {\int _{0}^{T}F(s(t),\mu (t))\,dt} }}\geq \lim _{T\to \infty }{\frac {1}{T}}\int _{0}^{T}{\underset {\text{surprise}}{\underbrace {-\log p(s(t)\mid m)} }}\,dt=H[p(s\mid m)]}$

This is because – under ergodic assumptions – the long-term average of surprise is entropy. This bound resists a natural tendency to disorder – of the sort associated with the second law of thermodynamics and the fluctuation theorem.

All Bayesian inference can be cast in terms of free energy minimisation; e.g.,.[21] When free energy is minimised with respect to internal states, the Kullback–Leibler divergence between the variational and posterior density over hidden states is minimised. This corresponds to approximate Bayesian inference – when the form of the variational density is fixed – and exact Bayesian inference otherwise. Free energy minimisation therefore provides a generic description of Bayesian inference and filtering (e.g., Kalman filtering). It is also used in Bayesian model selection, where free energy can be usefully decomposed into complexity and accuracy:

${\displaystyle {\underset {\text{free-energy}}{\underbrace {F(s,\mu )} }}={\underset {\text{complexity}}{\underbrace {D_{\mathrm {KL} }[q(\psi \mid \mu )\parallel p(\psi \mid m)]} }}-{\underset {\mathrm {accuracy} }{\underbrace {E_{q}[\log p(s\mid \psi ,m)]} }}}$

Models with minimum free energy provide an accurate explanation of data, under complexity costs (c.f., Occam's razor and more formal treatments of computational costs[22]). Here, complexity is the divergence between the variational density and prior beliefs about hidden states (i.e., the effective degrees of freedom used to explain the data).

Variational free energy is an information theoretic functional and is distinct from thermodynamic (Helmholtz) free energy.[23] However, the complexity term of variational free energy shares the same fixed point as Helmholtz free energy (under the assumption the system is thermodynamically closed but not isolated). This is because if sensory perturbations are suspended (for a suitably long period of time), complexity is minimised (because accuracy can be neglected). At this point, the system is at equilibrium and internal states minimise Helmholtz free energy, by the principle of minimum energy.[24]

Free energy minimisation is equivalent to maximising the mutual information between sensory states and internal states that parameterise the variational density (for a fixed entropy variational density).[11] This relates free energy minimization to the principle of minimum redundancy[25] and related treatments using information theory to describe optimal behaviour.[26][27]

Free energy minimisation formalises the notion of unconscious inference in perception[7][9] and provides a normative (Bayesian) theory of neuronal processing. The associated process theory of neuronal dynamics is based on minimising free energy through gradient descent. This corresponds to generalised Bayesian filtering (where ~ denotes a variable in generalised coordinates of motion and ${\displaystyle D}$ is a derivative matrix operator):[30]

${\displaystyle {\dot {\tilde {\mu }}}=D{\tilde {\mu }}-\partial _{\mu }F(s,\mu ){\Big |}_{\mu ={\tilde {\mu }}}}$

Usually, the generative models that define free energy are non-linear and hierarchical (like cortical hierarchies in the brain). Special cases of generalised filtering include Kalman filtering, which is formally equivalent to predictive coding[31] – a popular metaphor for message passing in the brain. Under hierarchical models, predictive coding involves the recurrent exchange of ascending (bottom-up) prediction errors and descending (top-down) predictions[32] that is consistent with the anatomy and physiology of sensory[33] and motor systems.[34]

In predictive coding, optimising model parameters through a gradient ascent on the time integral of free energy (free action) reduces to associative or Hebbian plasticity and is associated with synaptic plasticity in the brain.

Optimising the precision parameters corresponds to optimising the gain of prediction errors (c.f., Kalman gain). In neuronally plausible implementations of predictive coding,[32] this corresponds to optimising the excitability of superficial pyramidal cells and has been interpreted in terms of attentional gain.[35]

Simulation of the results achieved from a selective attention task carried out by the Bayesian reformulation of the SAIM entitled PE-SAIM in multiple objects environment. The graphs show the time course of the activation for the FOA and the two template units in the Knowledge Network.

Concerning the top-down vs bottom-up controversy that has been addressed as a major open problem of attention, a computational model has succeeded in illustrating the circulatory nature of reciprocation between top-down and bottom-up mechanisms. Using an established emergent model of attention, namely, SAIM, the authors suggested a model called PE-SAIM that in contrast to the standard version approaches the selective attention from a top-down stance. The model takes into account the forwarding prediction errors sent to the same level or a level above to minimize the energy function indicating the difference between data and its cause or in other words between the generative model and posterior. To enhance validity, they also incorporated the neural competition between the stimuli in their model. A notable feature of this model is the reformulation of the free energy function only in terms of predictions error in the course of task performance.

${\displaystyle {\dfrac {\partial E^{total}(Y^{VP},X^{SN},x^{CN},y^{KN})}{\partial y_{mn}^{SN}}}=x_{mn}^{CN}-b^{CN}\varepsilon _{nm}^{CN}+b^{CN}\sum _{k}(\varepsilon _{knm}^{KN})}$

where, ${\displaystyle E^{total}}$ is the total energy function of the neural networks entail, and ${\displaystyle \varepsilon _{knm}^{KN}}$ is the prediction error between the generative model (prior) and posterior changing over time.[36]) Comparing the two models reveals a notable similarity between the results as well as a promising finding, in that, in the standard version of SAIM, the model architecture consists of excitatory connections whereas in the PE-SAIM inhibitory connections will be leveraged in the course of Bayesian inference. The model has also been shown fit to predict the EEG and fMRI data drawn from human experiments.

Active inference is related to optimal control by replacing value or cost-to-go functions with prior beliefs about state transitions or flow.[38] This exploits the close connection between Bayesian filtering and the solution to the Bellman equation. However, active inference starts with (priors over) flow ${\displaystyle f=\Gamma \cdot \nabla V+\nabla \times W}$ that are specified with scalar ${\displaystyle V(x)}$ and vector ${\displaystyle W(x)}$ value functions of state space (c.f., the Helmholtz decomposition). Here, ${\displaystyle \Gamma }$ is the amplitude of random fluctuations and cost is ${\displaystyle c(x)=f\cdot \nabla V+\nabla \cdot \Gamma \cdot V}$. The priors over flow ${\displaystyle p({\tilde {x}}\mid m)}$ induce a prior over states ${\displaystyle p(x\mid m)=\exp(V(x))}$ that is the solution to the appropriate forward Kolmogorov equations.[39] In contrast, optimal control optimises the flow, given a cost function, under the assumption that ${\displaystyle W=0}$ (i.e., the flow is curl free or has detailed balance). Usually, this entails solving backward Kolmogorov equations.[40]

Optimal decision problems (usually formulated as partially observable Markov decision processes) are treated within active inference by absorbing utility functions into prior beliefs. In this setting, states that have a high utility (low cost) are states an agent expects to occupy. By equipping the generative model with hidden states that model control, policies (control sequences) that minimise variational free energy lead to high utility states.[41]

Neurobiologically, neuromodulators like dopamine are considered to report the precision of prediction errors by modulating the gain of principal cells encoding prediction error.[42] This is closely related to – but formally distinct from – the role of dopamine in reporting prediction errors per se[43] and related computational accounts.[44]

Active inference has been used to address a range of issues in cognitive neuroscience, brain function and neuropsychiatry, including: action observation,[45] mirror neurons,[46] saccades and visual search,[47][48] eye movements,[49] sleep,[50] illusions,[51] attention,[35] action selection,[42] consciousness,[52][53] hysteria[54] and psychosis.[55]