Action selection & goals¶
What the agent wants, and how it picks an action. StateGoal and ObservationGoal are the goals you hand the Agent, the continuous-state answer to pymdp's C. Each one also describes the selector the Agent builds from it. A Preference is the target and precision alone, the part every selector reads. Hand the Agent a Preference together with a selector of your own when no goal describes the search you want. EFESelector is the grid search the Agent builds for an ObservationGoal.
StateGoal
dataclass
¶
StateGoal(
target: ArrayLike,
*,
precision: ArrayLike | None = None,
effort: ArrayLike | None = None,
)
A state-space objective: reach a target state (the LQR / fixed-sensor regime).
The complete spec for the state-tracking path, the target and the LQR cost
weights. precision is the state-cost weight and effort the action-cost
weight, under the names LQRController takes. effort
is left None here because the action dimension is unknown until the Agent pairs
the goal with a model. The Agent then fills the identity and builds an
LQRSelector. Not a pytree. The Agent reads it once at construction and keeps a
Preference for the selector.
The target must hold still under the model's dynamics at zero action,
A · target = target with A the dynamics matrix. LQR regulates the offset
from the target and charges for effort, so a target the dynamics move would take a
constant push it never pays in full, and the state would settle short. The Agent
refuses such a target.
Source code in src/cpomdp/selection.py
ObservationGoal
dataclass
¶
ObservationGoal(
target: ArrayLike,
*,
action_bounds: tuple[float, float],
precision: ArrayLike | None = None,
n_candidates: int = _DEFAULT_N_CANDIDATES,
horizon: int | None = None,
info_node: SupportsIndex
| Sequence[SupportsIndex]
| None = None,
)
An observation-space objective: prefer to observe a target (the EFE regime).
The complete spec for the information-seeking path: the preferred observation, how
sharply it is preferred (precision), and the grid search the Agent builds from
it. action_bounds is the action box, two finite scalars. n_candidates is how
many actions the grid holds in total.
horizon is how many steps each candidate is held and scored over. A candidate
is one action repeated, so a longer horizon looks further along a constant push.
It never scores a plan whose action changes, such as stepping to a cue and then
committing. That search is cpomdp.enumeration.EnumeratedEFESearch, passed to
the Agent as a selector beside a Preference. Left as
None, the horizon is the shortest at which the target moves the scores, which
the Agent reads off the model (ADR-068). On a double integrator observed in
position that is 2. agent.selector.horizon reads it back.
info_node optionally aims the epistemic term at a latent node, or at several
as their joint marginal (the factored-EFE regime on a branching backend, issue
26). None means whole-state info gain. The Agent dispatches an¶
ObservationGoal to an EFESelector, or to an FFGEFESelector on a branching backend.
Not a pytree. The Agent reads it once at construction and keeps its
preference, which it hands to the EFE functions.
Source code in src/cpomdp/selection.py
preference
cached
property
¶
The target and precision, as the preference a score is taken against.
The Preference that
expected_free_energy and the rollouts in
cpomdp.efe take. The search settings stay on the goal. Built once, since
the goal cannot change.
Preference
dataclass
¶
What the agent wants: a target and how sharply it is preferred.
Every selector's select reads one. The Agent builds it from a
goal's target and precision, or takes it as the objective beside a selector
of your own, in the space that selector reads.
It is one Gaussian preference, single-mode since v0.3. The disjunctive mixture
case, visiting one of several goals, is RFC-002 and deferred. A mixture
Preference plugs in here.
precision is unused by LQRSelector, whose controller
bakes its own precision into the Riccati solve at construction. The EFE selectors
read it in the pragmatic term.
Source code in src/cpomdp/selection.py
EFESelector ¶
EFESelector(
model: LinearGaussianModel,
*,
action_bounds: tuple[float, float],
n_candidates: int = _DEFAULT_N_CANDIDATES,
horizon: int = 1,
)
EFE action selection over a front-loaded candidate grid, horizon-aware.
At horizon = 1 (default) it minimises one-step G over the grid. At
horizon > 1 it scores constant-action policies (each grid action held for H
steps) via policy_efe and returns the first action of the best one
(receding-horizon). Per-cycle cost is a single attributable number,
cost_per_cycle = n_candidates * horizon.
Built directly, the selector uses the horizon it is given. On a model where a
push takes more than one step to reach the reading, horizon = 1 leaves the
target out of every comparison. The candidates then differ by their noise alone, or
tie. The Agent reads the horizon for an ObservationGoal off the model and
refuses one too short for the target to move the scores.
Candidates whose scores tie to within roundoff go to the one nearest zero. When no
candidate scores a finite G, select returns NaN rather than an edge of the
box. A state-dependent noise that is not a covariance at the predicted states does
that, and Agent refuses the NaN.
Honest caveat: horizon selects the best constant action, not the best
sequence. A genuinely sequential epistemic policy — move to sense, then exploit —
needs a varying sequence the constant-action family cannot express, so at H > 1 the
selector can still look myopic-ish on such tasks. True varying-sequence search is
EnumeratedEFESearch in cpomdp.enumeration, which enumerates every length-H
sequence of a declared finite action set (ADR-031).
Front-load the candidate grid over the action box.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
LinearGaussianModel
|
The model the scores are computed under. It needs a control matrix with one action dimension. |
required |
action_bounds
|
tuple[float, float]
|
|
required |
n_candidates
|
int
|
How many evenly spaced actions the grid holds, in total. At least 2. |
_DEFAULT_N_CANDIDATES
|
horizon
|
int
|
How many steps each candidate is held and scored over. At least 1. |
1
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the model has no control matrix or more than one action
dimension, if |
Source code in src/cpomdp/selection.py
candidates
property
¶
The actions the grid searches, shape (n_candidates, p), lowest first.
n_candidates
property
¶
How many actions the grid holds in total, each scored once per cycle.
warrant
property
¶
CORROBORATED — the grid samples a continuum, it does not decide it.
The counterpart to EnumeratedEFESearch.warrant (PROVED): the two search
families print in different vocabulary so a sampled result is never read as a
decided one (ADR-031).
select ¶
The grid action minimising G over the horizon (the per-cycle work).
At horizon = 1 one vmap of the one-step kernel + argmin. At
horizon > 1 one vmap of policy_efe over the constant-action policies
+ argmin, returning the first (= constant) action of the best policy. NaN
when no candidate scores a finite G. The first call compiles the work and
later calls reuse it.
Source code in src/cpomdp/selection.py
__repr__ ¶
The horizon and the grid, the numbers that set what each cycle costs.
Source code in src/cpomdp/selection.py
ActionSelector is the protocol every selector satisfies. LQRSelector is the fixed-noise path. With a fixed sensor and fixed dynamics noise, expected free energy provably reduces to LQR 1 (ADR-003). FFGEFESelector is EFESelector's peer for a branching backend.
ActionSelector ¶
Bases: Protocol
Chooses an action from a belief and a preference.
The abstraction wall for action selection. LQRSelector is
the fixed-noise case, a fixed sensor and fixed dynamics noise, where EFE collapses
to LQR (ADR-003). EFESelector is the state-dependent one.
Agent depends only on this, never on a concrete selector.
select ¶
LQRSelector
dataclass
¶
Adapts an LQRController to the ActionSelector interface.
A thin wrapper: it owns no control logic, it just forwards to the front-loaded controller, unpacking the belief's mean and the preference's target. EFE collapses to LQR under a fixed sensor and fixed dynamics noise (ADR-003), so for that regime this is the action selector.
select ¶
Forward to the controller: action(belief.mean, preference.target).
No control logic of its own — the Riccati solve was front-loaded into the
controller at construction. This just unpacks the belief and preference
into the controller's (mean, target) signature.
Source code in src/cpomdp/selection.py
FFGEFESelector ¶
FFGEFESelector(
backend: GraphBackend,
*,
action_bounds: tuple[float, float],
n_candidates: int = _DEFAULT_N_CANDIDATES,
info_node: SupportsIndex
| Sequence[SupportsIndex]
| None = None,
)
EFE action selection over a branching FFG backend (issue #26).
The FFG counterpart of EFESelector: instead of
_efe_step on a flat model it vmaps _ffg_efe_step over the candidate
action grid, reading μ⁺/Σ⁺
from the backend's predicted_belief (structural couplings folded in) and aiming
the epistemic term at info_node, one node or several, or the whole joint state
when it is None. One-step
(horizon = 1); the H-step FFG rollout is a later seam. Conforms to
ActionSelector.
Front-load the candidate grid, after checking the backend and the target.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
GraphBackend
|
The branching backend whose predicted joint belief is scored. |
required |
action_bounds
|
tuple[float, float]
|
|
required |
n_candidates
|
int
|
How many evenly spaced actions the grid holds, in total. At least 2. |
_DEFAULT_N_CANDIDATES
|
info_node
|
SupportsIndex | Sequence[SupportsIndex] | None
|
The node whose state the information term is about. |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
InadmissiblePartitionError
|
If the backend's carry partition severs an EFE-load-bearing edge. |
ValueError
|
If the backend has no control matrix or more than one action
dimension, if |
TypeError
|
If |
Source code in src/cpomdp/selection.py
candidates
property
¶
The actions the grid searches, shape (n_candidates, p), lowest first.
n_candidates
property
¶
How many actions the grid holds in total, each scored once per cycle.
model
property
¶
The backend's model, whose sizes, sensor and control the score reads.
On a coupled graph it is the interface model, which leaves out the couplings
the score reads off backend.
warrant
property
¶
CORROBORATED, as for EFESelector.
The grid samples a continuum of actions and decides nothing about it.
select ¶
The grid action minimising G (the per-cycle work).
For each candidate: predict the joint (predicted_belief → μ⁺, Σ⁺),
score with _ffg_efe_step, argmin over the grid. R is re-read at
μ⁺ per candidate. It is constant under a fixed sensor and moves with the
action under R(x), the dual effect. This is the RFC-001 hot path. Ties
and scores that are not finite go as for
EFESelector, and the first call compiles the work.
Source code in src/cpomdp/selection.py
__repr__ ¶
The horizon, the grid and the node the epistemic aims at.
Source code in src/cpomdp/selection.py
-
Magnus T. Koudahl, Wouter M. Kouw, and Bert de Vries. On epistemics in expected free energy for linear Gaussian state space models. Entropy, 23(12):1565, 2021. URL: https://doi.org/10.3390/e23121565, doi:10.3390/e23121565. ↩