Expected free energy¶
The scalar each candidate action is scored by, G = pragmatic − epistemic. The pragmatic term pulls toward preferred outcomes. The epistemic term rewards actions that sharpen the belief. Lower G is better.
In plain terms
An agent choosing an action asks two questions about it. Where will it leave me
relative to what I want to see? How much will the next reading tell me about where I
am? expected_free_energy answers both and returns them as one number per action.
Lower is better.
The first answer is the pragmatic term. It is a cost. It grows with the distance between the reading the agent expects and the reading it wants, and with how blurred that expected reading is. The second answer is the epistemic term. It is a value, and it measures how much the reading will sharpen the agent's belief about its state. The score is the cost minus the value, so an action can win by landing near the goal or by being informative.
The goal is a reading. The agent says what it wants to observe, and that is a state
only when the sensor reads the state directly. Pass that reading as a Preference.
An ObservationGoal you already hold hands over its own as goal.preference.
Fix the sensor's noise and the dynamics' noise, and every action sharpens the belief by the same amount. The epistemic term is then a constant. The best action is the one a classic regulator picks, and that regulator is worked out once, with no candidates to score. The information term matters only when the noise depends on where the action takes the state.
expected_free_energy ¶
expected_free_energy(
model: LinearGaussianModel,
belief: Belief,
action: Float64[Array, p],
preference: Preference,
) -> EFEResult
Expected Free Energy of taking action from belief, and its split.
The locked linear-Gaussian definition (ADR-005), with the sensor read at the predicted mean::
predict: μ⁺ = A·μ + B·a Σ⁺ = A·Σ·Aᵀ + Q
sense: (C, R) at μ⁺, o⁺ = C·μ⁺, S = C·Σ⁺·Cᵀ + R
pragmatic: ½·(o⁺ − o*)ᵀ·Λ·(o⁺ − o*) + ½·tr(Λ·S)
epistemic: ½·(ln det S − ln det R)
G = pragmatic − epistemic
μ and Σ are the belief's mean and covariance, and a is the action. A, B and Q are the dynamics, control and dynamics-noise matrices. C and R are the sensor's observation matrix and noise. μ⁺ and Σ⁺ are the predicted state moments, o⁺ the predicted reading and S its covariance. o* is the preferred reading and Λ its precision.
Pure jnp, so it composes under jit, vmap and grad, in particular
over a batch of candidate action vectors with model, belief and
preference held fixed, which is how EFESelector
searches. Take jax.grad of a function that returns result.g, and rank a
batch by its g too. The Expected free energy page of the API reference opens
the same account in plain terms.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
LinearGaussianModel
|
The generative model. Must have a control matrix (an action has no
meaning without one). Its |
required |
belief
|
Belief
|
The current belief, mean μ and covariance Σ. |
required |
action
|
Float64[Array, p]
|
The candidate action a, shape |
required |
preference
|
Preference
|
The goal as an observation-space
|
required |
Returns:
| Type | Description |
|---|---|
EFEResult
|
An |
EFEResult
|
field and its two non-negative terms, |
EFEResult
|
is preferred. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the model has no control matrix. |
Source code in src/cpomdp/efe.py
EFEResult
dataclass
¶
Expected free energy and the two terms it is made of.
expected_free_energy,
policy_efe and
policy_efe_ffg return one.
PolicyEFETrace has the same three fields, one entry
per step. Fields are read by name. A record does not index or unpack, as the search
results do not.
A registered JAX pytree, so a vmap over candidate actions returns one record
with every field batched. Rank the batch by its g, as jnp.argmin(batch.g).
Differentiate g too. Three fields are not the (value, aux) pair that
jax.value_and_grad(..., has_aux=True) expects, so a function differentiated
that way returns (result.g, result).
Attributes:
| Name | Type | Description |
|---|---|---|
g |
Float64[Array, '']
|
The expected free energy |
pragmatic |
Float64[Array, '']
|
The goal term, a cost. |
epistemic |
Float64[Array, '']
|
The information the observation carries about the state, or about the targeted node where one is set, a value. |
__array__ ¶
Scoring a plan over several steps¶
In plain terms
expected_free_energy looks one step ahead. Some actions pay off later. A push
changes a velocity first and a reading only on the step after, so a one-step look
sees no difference between pushes. policy_efe scores a whole sequence of actions
instead. It steps the belief forward through the sequence and adds up each step's
score.
Between steps it carries the belief the agent expects to hold. The mean follows the action. The uncertainty shrinks as if each expected reading had arrived. Under a Gaussian model that shrinkage does not depend on what the reading says, so it can be worked out before any reading exists.
EFESelector above horizon 1 and the flat exhaustive searches call it once per
candidate sequence. That is the work each of their decisions costs. policy_efe_trace runs the same steps and keeps every step's
numbers, for inspecting one sequence rather than ranking many.
policy_efe ¶
policy_efe(
model: LinearGaussianModel,
belief: Belief,
policy: Float64[Array, "H p"],
preference: Preference,
) -> EFEResult
Summed EFE of a horizon-H policy from belief.
EFESelector above horizon 1 and the flat exhaustive
searches in cpomdp.enumeration score with it. A lax.scan over the
policy rows sums each step's G while propagating the belief predict-only
between steps. The mean follows the prediction. The covariance contracts by the
Kalman update at the moments that step's score already predicted. H is
policy.shape[0], and at H = 1 it reduces exactly to
expected_free_energy. Composes under jit,
vmap and grad. The Expected free energy page of the API reference opens
the rollout in plain terms.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
LinearGaussianModel
|
The generative model. Must have a control matrix. |
required |
belief
|
Belief
|
The belief the rollout starts from. |
required |
policy
|
Float64[Array, 'H p']
|
The |
required |
preference
|
Preference
|
The observation-space |
required |
Returns:
| Type | Description |
|---|---|
EFEResult
|
An |
EFEResult
|
horizon. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the model has no control matrix. |
Source code in src/cpomdp/efe.py
policy_efe_trace ¶
policy_efe_trace(
model: LinearGaussianModel,
belief: Belief,
policy: Float64[Array, "H p"],
preference: Preference,
) -> PolicyEFETrace
Per-step trace of the H-step rollout, the diagnostic sibling of policy_efe.
Runs the same scan as policy_efe and keeps the whole
per-step record instead of summing it away. The arithmetic is shared, so
jnp.sum of each returned scalar column equals the matching policy_efe
field bit-for-bit. Composes under jit, vmap and grad.
Not on the selector hot path. Stacking the H×n×n covariances is the memory cost,
so EFESelector stays on policy_efe and this is called
only for diagnostics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
LinearGaussianModel
|
The generative model. Must have a control matrix. |
required |
belief
|
Belief
|
The belief the rollout starts from. |
required |
policy
|
Float64[Array, 'H p']
|
The |
required |
preference
|
Preference
|
The observation-space |
required |
Returns:
| Type | Description |
|---|---|
PolicyEFETrace
|
A |
PolicyEFETrace
|
leading H axis. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the model has no control matrix. |
Source code in src/cpomdp/efe.py
PolicyEFETrace
dataclass
¶
PolicyEFETrace(
g: Float64[Array, H],
pragmatic: Float64[Array, H],
epistemic: Float64[Array, H],
predicted_mean: Float64[Array, "H n"],
predicted_cov: Float64[Array, "H n n"],
posterior_cov: Float64[Array, "H n n"],
predicted_observation_cov: Float64[Array, "H m m"],
)
One H-step rollout's per-step record, each field stacked along a leading H axis.
The first three fields are the per-step EFEResult, and
each sums to the matching field of the rollout's result. The rest are the moments
the rollout carries between steps, kept for inspection: the covariance a policy
carries, and the matrices whose conditioning the rollout depends on.
A registered JAX pytree. lax.scan emits it as its ys, and it survives
jit, vmap and grad, gaining any outer vmap axis too.
Predicted means after the dynamics and the action and before any observation. Posterior means after the predict-only Kalman update the rollout carries to the next step.
Attributes:
| Name | Type | Description |
|---|---|---|
g |
Float64[Array, H]
|
Per-step EFE |
pragmatic |
Float64[Array, H]
|
Per-step goal term, shape |
epistemic |
Float64[Array, H]
|
Per-step information gain about the state, shape |
predicted_mean |
Float64[Array, 'H n']
|
The predicted state mean, μ⁺ = |
predicted_cov |
Float64[Array, 'H n n']
|
The predicted state covariance, Σ⁺ = |
posterior_cov |
Float64[Array, 'H n n']
|
The state covariance after the predicted observation, Σ_post,
shape |
predicted_observation_cov |
Float64[Array, 'H m m']
|
The predicted observation covariance,
S = |
__array__ ¶
On a factor graph¶
The same two rollouts over a CouplingGraphBackend, with the epistemic term aimed at one node's block of the joint state.
policy_efe_ffg ¶
policy_efe_ffg(
backend: GraphBackend,
belief: Belief,
policy: Float64[Array, "H p"],
preference: Preference,
*,
info_node: SupportsIndex
| Sequence[SupportsIndex]
| None = None,
) -> EFEResult
Summed EFE of a horizon-H policy over a branching FFG backend.
The graph counterpart of policy_efe. Each step scores
the backend's predicted joint, with the epistemic term aimed at info_node's
state or the whole joint state. A lax.scan over
the policy rows sums each step's G while propagating the joint belief
predict-only between steps. The mean carries as the coupling-resolved predicted
mean. The covariance contracts by the Kalman update at the sensor noise read
there. At H = 1 it is one step of the graph score. With no couplings and a
whole-state target it reproduces policy_efe.
The backend is held as a closure constant, since it is not a JAX pytree. This
composes under jit, vmap and grad over the array arguments, the way
FFGEFESelector vmaps its per-candidate score.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
GraphBackend
|
The FFG inference backend. It supplies |
required |
belief
|
Belief
|
The starting joint belief |
required |
policy
|
Float64[Array, 'H p']
|
The |
required |
preference
|
Preference
|
The observation-space |
required |
info_node
|
SupportsIndex | Sequence[SupportsIndex] | None
|
The node whose state the per-step epistemic value is about.
|
None
|
Returns:
| Type | Description |
|---|---|
EFEResult
|
An |
EFEResult
|
horizon. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
InadmissiblePartitionError
|
If the backend's carry partition severs an EFE-load-bearing edge. |
ValueError
|
If |
TypeError
|
If |
Source code in src/cpomdp/efe.py
policy_efe_ffg_trace ¶
policy_efe_ffg_trace(
backend: GraphBackend,
belief: Belief,
policy: Float64[Array, "H p"],
preference: Preference,
*,
info_node: SupportsIndex
| Sequence[SupportsIndex]
| None = None,
) -> PolicyEFETrace
Per-step trace of the FFG rollout, the diagnostic sibling of policy_efe_ffg.
Runs the same scan as policy_efe_ffg and keeps the
whole per-step record instead of summing it. The arithmetic is shared, so
jnp.sum of each returned scalar column equals the matching policy_efe_ffg
field bit-for-bit. The epistemic column is the info gain about the
info_node targeted. The flat
policy_efe_trace carries the whole-state term
there instead. The moment columns are the
coupling-resolved predicted moments and the contracted posterior covariance.
Composes under jit, vmap and grad with the backend held fixed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
GraphBackend
|
The FFG inference backend, as for |
required |
belief
|
Belief
|
The starting joint belief over the backend's joint state. |
required |
policy
|
Float64[Array, 'H p']
|
The |
required |
preference
|
Preference
|
The observation-space |
required |
info_node
|
SupportsIndex | Sequence[SupportsIndex] | None
|
The node whose state the per-step epistemic value is about.
|
None
|
Returns:
| Type | Description |
|---|---|
PolicyEFETrace
|
A |
PolicyEFETrace
|
emits, of arrays stacked along a leading H axis. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
InadmissiblePartitionError
|
If the backend's carry partition severs an EFE-load-bearing edge. |
ValueError
|
If |
TypeError
|
If |