Skip to content

Action selection & goals

What the agent wants, and how it picks an action. StateGoal and ObservationGoal are the goals you hand the Agent, the continuous-state answer to pymdp's C. Each one also describes the selector the Agent builds from it. A Preference is the target and precision alone, the part every selector reads. Hand the Agent a Preference together with a selector of your own when no goal describes the search you want. EFESelector is the grid search the Agent builds for an ObservationGoal.

StateGoal dataclass

StateGoal(
    target: ArrayLike,
    *,
    precision: ArrayLike | None = None,
    effort: ArrayLike | None = None,
)

A state-space objective: reach a target state (the LQR / fixed-sensor regime).

The complete spec for the state-tracking path, the target and the LQR cost weights. precision is the state-cost weight and effort the action-cost weight, under the names LQRController takes. effort is left None here because the action dimension is unknown until the Agent pairs the goal with a model. The Agent then fills the identity and builds an LQRSelector. Not a pytree. The Agent reads it once at construction and keeps a Preference for the selector.

The target must hold still under the model's dynamics at zero action, A · target = target with A the dynamics matrix. LQR regulates the offset from the target and charges for effort, so a target the dynamics move would take a constant push it never pays in full, and the state would settle short. The Agent refuses such a target.

Source code in src/cpomdp/selection.py
def __init__(
    self,
    target: ArrayLike,
    *,
    precision: ArrayLike | None = None,
    effort: ArrayLike | None = None,
) -> None:
    target = jnp.asarray(target, dtype=float)
    object.__setattr__(self, "target", target)
    n = target.shape[0]
    object.__setattr__(
        self,
        "precision",
        jnp.eye(n) if precision is None else jnp.asarray(precision, dtype=float),
    )
    object.__setattr__(
        self,
        "effort",
        None if effort is None else jnp.asarray(effort, dtype=float),
    )
    self._validate()

ObservationGoal dataclass

ObservationGoal(
    target: ArrayLike,
    *,
    action_bounds: tuple[float, float],
    precision: ArrayLike | None = None,
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    horizon: int | None = None,
    info_node: SupportsIndex
    | Sequence[SupportsIndex]
    | None = None,
)

An observation-space objective: prefer to observe a target (the EFE regime).

The complete spec for the information-seeking path: the preferred observation, how sharply it is preferred (precision), and the grid search the Agent builds from it. action_bounds is the action box, two finite scalars. n_candidates is how many actions the grid holds in total.

horizon is how many steps each candidate is held and scored over. A candidate is one action repeated, so a longer horizon looks further along a constant push. It never scores a plan whose action changes, such as stepping to a cue and then committing. That search is cpomdp.enumeration.EnumeratedEFESearch, passed to the Agent as a selector beside a Preference. Left as None, the horizon is the shortest at which the target moves the scores, which the Agent reads off the model (ADR-068). On a double integrator observed in position that is 2. agent.selector.horizon reads it back.

info_node optionally aims the epistemic term at a latent node, or at several as their joint marginal (the factored-EFE regime on a branching backend, issue

26). None means whole-state info gain. The Agent dispatches an

ObservationGoal to an EFESelector, or to an FFGEFESelector on a branching backend. Not a pytree. The Agent reads it once at construction and keeps its preference, which it hands to the EFE functions.

Source code in src/cpomdp/selection.py
def __init__(
    self,
    target: ArrayLike,
    *,
    action_bounds: tuple[float, float],
    precision: ArrayLike | None = None,
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    horizon: int | None = None,
    info_node: SupportsIndex | Sequence[SupportsIndex] | None = None,
) -> None:
    target = jnp.asarray(target, dtype=float)
    object.__setattr__(self, "target", target)
    m = target.shape[0]
    object.__setattr__(
        self,
        "precision",
        jnp.eye(m) if precision is None else jnp.asarray(precision, dtype=float),
    )
    object.__setattr__(self, "action_bounds", _checked_action_bounds(action_bounds))
    object.__setattr__(self, "n_candidates", _checked_n_candidates(n_candidates))
    object.__setattr__(
        self, "horizon", None if horizon is None else _checked_horizon(horizon)
    )
    object.__setattr__(self, "info_node", _stored_info_node(info_node))
    self._validate()

preference cached property

preference: Preference

The target and precision, as the preference a score is taken against.

The Preference that expected_free_energy and the rollouts in cpomdp.efe take. The search settings stay on the goal. Built once, since the goal cannot change.

Preference dataclass

Preference(
    target: ArrayLike, precision: ArrayLike | None = None
)

What the agent wants: a target and how sharply it is preferred.

Every selector's select reads one. The Agent builds it from a goal's target and precision, or takes it as the objective beside a selector of your own, in the space that selector reads.

It is one Gaussian preference, single-mode since v0.3. The disjunctive mixture case, visiting one of several goals, is RFC-002 and deferred. A mixture Preference plugs in here.

precision is unused by LQRSelector, whose controller bakes its own precision into the Riccati solve at construction. The EFE selectors read it in the pragmatic term.

Source code in src/cpomdp/selection.py
def __init__(self, target: ArrayLike, precision: ArrayLike | None = None) -> None:
    target = jnp.asarray(target, dtype=float)
    if target.ndim != 1:
        raise ValueError(f"target must be a 1-D vector, got shape {target.shape}")
    object.__setattr__(self, "target", target)
    n = target.shape[0]
    object.__setattr__(
        self,
        "precision",
        jnp.eye(n) if precision is None else jnp.asarray(precision, dtype=float),
    )
    self._validate()

tree_flatten

tree_flatten()

Leaves: (target, precision), and no static aux. jit and vmap take one.

Source code in src/cpomdp/selection.py
def tree_flatten(self):
    """Leaves: (target, precision), and no static aux. jit and vmap take one."""
    return (self.target, self.precision), None

tree_unflatten classmethod

tree_unflatten(aux_data, children)

Rebuild without re-validating — the leaves may be tracers.

Source code in src/cpomdp/selection.py
@classmethod
def tree_unflatten(cls, aux_data, children):
    """Rebuild without re-validating — the leaves may be tracers."""
    target, precision = children
    obj = object.__new__(cls)
    object.__setattr__(obj, "target", target)
    object.__setattr__(obj, "precision", precision)
    return obj

EFESelector

EFESelector(
    model: LinearGaussianModel,
    *,
    action_bounds: tuple[float, float],
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    horizon: int = 1,
)

EFE action selection over a front-loaded candidate grid, horizon-aware.

At horizon = 1 (default) it minimises one-step G over the grid. At horizon > 1 it scores constant-action policies (each grid action held for H steps) via policy_efe and returns the first action of the best one (receding-horizon). Per-cycle cost is a single attributable number, cost_per_cycle = n_candidates * horizon.

Built directly, the selector uses the horizon it is given. On a model where a push takes more than one step to reach the reading, horizon = 1 leaves the target out of every comparison. The candidates then differ by their noise alone, or tie. The Agent reads the horizon for an ObservationGoal off the model and refuses one too short for the target to move the scores.

Candidates whose scores tie to within roundoff go to the one nearest zero. When no candidate scores a finite G, select returns NaN rather than an edge of the box. A state-dependent noise that is not a covariance at the predicted states does that, and Agent refuses the NaN.

Honest caveat: horizon selects the best constant action, not the best sequence. A genuinely sequential epistemic policy — move to sense, then exploit — needs a varying sequence the constant-action family cannot express, so at H > 1 the selector can still look myopic-ish on such tasks. True varying-sequence search is EnumeratedEFESearch in cpomdp.enumeration, which enumerates every length-H sequence of a declared finite action set (ADR-031).

Front-load the candidate grid over the action box.

Parameters:

Name Type Description Default
model LinearGaussianModel

The model the scores are computed under. It needs a control matrix with one action dimension.

required
action_bounds tuple[float, float]

(lo, hi), the action box the grid spans, two finite scalars with lo < hi.

required
n_candidates int

How many evenly spaced actions the grid holds, in total. At least 2.

_DEFAULT_N_CANDIDATES
horizon int

How many steps each candidate is held and scored over. At least 1.

1

Raises:

Type Description
TypeError

If n_candidates or horizon is not an integer, or model is a coupled graph backend's model, which leaves out the couplings.

ValueError

If the model has no control matrix or more than one action dimension, if action_bounds is not two finite scalars in order, if n_candidates is below 2, or if horizon is below 1.

Source code in src/cpomdp/selection.py
def __init__(
    self,
    model: LinearGaussianModel,
    *,
    action_bounds: tuple[float, float],
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    horizon: int = 1,
) -> None:
    """Front-load the candidate grid over the action box.

    Args:
        model: The model the scores are computed under. It needs a control matrix
            with one action dimension.
        action_bounds: ``(lo, hi)``, the action box the grid spans, two finite
            scalars with ``lo < hi``.
        n_candidates: How many evenly spaced actions the grid holds, in total. At
            least 2.
        horizon: How many steps each candidate is held and scored over. At least
            1.

    Raises:
        TypeError: If ``n_candidates`` or ``horizon`` is not an integer, or
            ``model`` is a coupled graph backend's model, which leaves out the
            couplings.
        ValueError: If the model has no control matrix or more than one action
            dimension, if ``action_bounds`` is not two finite scalars in order, if
            ``n_candidates`` is below 2, or if ``horizon`` is below 1.
    """
    _refuse_graph_interface(model, "EFESelector")
    _require_scalar_control(
        model.control_matrix, "EFESelector", "EnumeratedEFESearch"
    )
    candidates = _candidate_grid(action_bounds, n_candidates)
    horizon = _checked_horizon(horizon)
    self._model = model
    self._horizon = horizon
    self._candidates = candidates
    self._distance_from_zero = jnp.linalg.norm(candidates, axis=1)
    # H>1: each candidate becomes a constant-action policy (held for H steps).
    self._policies = jnp.repeat(candidates[:, None, :], horizon, axis=1)
    # Compiled once, on first use. Belief and Preference are pytree arguments, so
    # every later call reuses the program (ADR-002).
    self._scores = jax.jit(self._score_grid)
    self._choose = jax.jit(self._score_and_choose)

candidates property

candidates: Float64[Array, 'k p']

The actions the grid searches, shape (n_candidates, p), lowest first.

n_candidates property

n_candidates: int

How many actions the grid holds in total, each scored once per cycle.

model property

model: LinearGaussianModel

The model every candidate is scored under.

horizon property

horizon: int

How many steps each candidate's one action is held and scored over.

cost_per_cycle property

cost_per_cycle: int

Per-cycle step-evals = n_candidates * horizon.

warrant property

warrant: Warrant

CORROBORATED — the grid samples a continuum, it does not decide it.

The counterpart to EnumeratedEFESearch.warrant (PROVED): the two search families print in different vocabulary so a sampled result is never read as a decided one (ADR-031).

select

select(
    belief: Belief, preference: Preference
) -> Float64[Array, p]

The grid action minimising G over the horizon (the per-cycle work).

At horizon = 1 one vmap of the one-step kernel + argmin. At horizon > 1 one vmap of policy_efe over the constant-action policies + argmin, returning the first (= constant) action of the best policy. NaN when no candidate scores a finite G. The first call compiles the work and later calls reuse it.

Source code in src/cpomdp/selection.py
def select(self, belief: Belief, preference: Preference) -> Float64[Array, "p"]:
    """The grid action minimising ``G`` over the horizon (the per-cycle work).

    At ``horizon = 1`` one ``vmap`` of the one-step kernel + ``argmin``. At
    ``horizon > 1`` one ``vmap`` of ``policy_efe`` over the constant-action policies
    + ``argmin``, returning the first (= constant) action of the best policy. NaN
    when no candidate scores a finite ``G``. The first call compiles the work and
    later calls reuse it.
    """
    _target_and_precision(preference)  # refused here, not inside the compiled call
    return self._choose(belief, preference)

__repr__

__repr__() -> str

The horizon and the grid, the numbers that set what each cycle costs.

Source code in src/cpomdp/selection.py
def __repr__(self) -> str:
    """The horizon and the grid, the numbers that set what each cycle costs."""
    lo, hi = float(self._candidates[0, 0]), float(self._candidates[-1, 0])
    return (
        f"EFESelector(horizon={self._horizon}, n_candidates={self.n_candidates}, "
        f"action_bounds=({lo}, {hi}))"
    )

ActionSelector is the protocol every selector satisfies. LQRSelector is the fixed-noise path. With a fixed sensor and fixed dynamics noise, expected free energy provably reduces to LQR 1 (ADR-003). FFGEFESelector is EFESelector's peer for a branching backend.

ActionSelector

Bases: Protocol

Chooses an action from a belief and a preference.

The abstraction wall for action selection. LQRSelector is the fixed-noise case, a fixed sensor and fixed dynamics noise, where EFE collapses to LQR (ADR-003). EFESelector is the state-dependent one. Agent depends only on this, never on a concrete selector.

select

select(
    belief: Belief, preference: Preference
) -> Float64[Array, p]

The action to take given the current belief and preference.

Source code in src/cpomdp/selection.py
def select(self, belief: Belief, preference: Preference) -> Float64[Array, "p"]:
    """The action to take given the current ``belief`` and ``preference``."""
    ...

LQRSelector dataclass

LQRSelector(controller: LQRController)

Adapts an LQRController to the ActionSelector interface.

A thin wrapper: it owns no control logic, it just forwards to the front-loaded controller, unpacking the belief's mean and the preference's target. EFE collapses to LQR under a fixed sensor and fixed dynamics noise (ADR-003), so for that regime this is the action selector.

model property

model: LinearGaussianModel

The model the controller's gain was solved for.

select

select(
    belief: Belief, preference: Preference
) -> Float64[Array, p]

Forward to the controller: action(belief.mean, preference.target).

No control logic of its own — the Riccati solve was front-loaded into the controller at construction. This just unpacks the belief and preference into the controller's (mean, target) signature.

Source code in src/cpomdp/selection.py
def select(self, belief: Belief, preference: Preference) -> Float64[Array, "p"]:
    """Forward to the controller: ``action(belief.mean, preference.target)``.

    No control logic of its own — the Riccati solve was front-loaded into the
    controller at construction. This just unpacks the belief and preference
    into the controller's ``(mean, target)`` signature.
    """
    return self.controller.action(belief.mean, preference.target)

FFGEFESelector

FFGEFESelector(
    backend: GraphBackend,
    *,
    action_bounds: tuple[float, float],
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    info_node: SupportsIndex
    | Sequence[SupportsIndex]
    | None = None,
)

EFE action selection over a branching FFG backend (issue #26).

The FFG counterpart of EFESelector: instead of _efe_step on a flat model it vmaps _ffg_efe_step over the candidate action grid, reading μ⁺/Σ⁺ from the backend's predicted_belief (structural couplings folded in) and aiming the epistemic term at info_node, one node or several, or the whole joint state when it is None. One-step (horizon = 1); the H-step FFG rollout is a later seam. Conforms to ActionSelector.

Front-load the candidate grid, after checking the backend and the target.

Parameters:

Name Type Description Default
backend GraphBackend

The branching backend whose predicted joint belief is scored.

required
action_bounds tuple[float, float]

(lo, hi), the action box the grid spans, two finite scalars with lo < hi.

required
n_candidates int

How many evenly spaced actions the grid holds, in total. At least 2.

_DEFAULT_N_CANDIDATES
info_node SupportsIndex | Sequence[SupportsIndex] | None

The node whose state the information term is about. None takes the whole joint state, as on an ObservationGoal.

None

Raises:

Type Description
TypeError

If info_node is not a node index, or a list, tuple, range or 1-D array of them.

InadmissiblePartitionError

If the backend's carry partition severs an EFE-load-bearing edge.

ValueError

If the backend has no control matrix or more than one action dimension, if action_bounds is not two finite scalars in order, if n_candidates is below 2, or if info_node is empty, repeats a node or names one outside the graph.

TypeError

If n_candidates is not an integer.

Source code in src/cpomdp/selection.py
def __init__(
    self,
    backend: GraphBackend,
    *,
    action_bounds: tuple[float, float],
    n_candidates: int = _DEFAULT_N_CANDIDATES,
    info_node: SupportsIndex | Sequence[SupportsIndex] | None = None,
) -> None:
    """Front-load the candidate grid, after checking the backend and the target.

    Args:
        backend: The branching backend whose predicted joint belief is scored.
        action_bounds: ``(lo, hi)``, the action box the grid spans, two finite
            scalars with ``lo < hi``.
        n_candidates: How many evenly spaced actions the grid holds, in total. At
            least 2.
        info_node: The node whose state the information term is about. ``None``
            takes the whole joint state, as on an
            [`ObservationGoal`][cpomdp.ObservationGoal].

    Raises:
        TypeError: If ``info_node`` is not a node index, or a list, tuple,
            range or 1-D array of them.
        InadmissiblePartitionError: If the backend's carry partition severs an
            EFE-load-bearing edge.
        ValueError: If the backend has no control matrix or more than one action
            dimension, if ``action_bounds`` is not two finite scalars in order,
            if ``n_candidates`` is below 2, or if ``info_node`` is empty, repeats a
            node or names one outside the graph.
        TypeError: If ``n_candidates`` is not an integer.
    """
    _require_scalar_control(
        backend.model.control_matrix,
        "FFGEFESelector",
        "EnumeratedEFESearch.over_backend",
    )
    candidates = _candidate_grid(action_bounds, n_candidates)
    # ADR-018 admissibility, enforced on the EFE paths. Pure filtering on any
    # partition is never blocked.
    info_block = _info_block_of(backend, info_node)
    # Front-load (ADR-002): the candidate grid, the info block, the constant C.
    # R is *not* front-loaded — it may be state-dependent, so it is read per
    # candidate at μ⁺ via ``model.observation_noise_at`` (issue #27). A fixed
    # sensor returns the same constant each time.
    self._backend = backend
    self._info_node = _stored_info_node(info_node)
    self._info_block = info_block
    self._observation_matrix = backend.model.observation_matrix  # C (constant)
    self._candidates = candidates
    self._distance_from_zero = jnp.linalg.norm(candidates, axis=1)
    self._choose = jax.jit(self._score_and_choose)  # compiled once, on first use

candidates property

candidates: Float64[Array, 'k p']

The actions the grid searches, shape (n_candidates, p), lowest first.

n_candidates property

n_candidates: int

How many actions the grid holds in total, each scored once per cycle.

model property

model: LinearGaussianModel

The backend's model, whose sizes, sensor and control the score reads.

On a coupled graph it is the interface model, which leaves out the couplings the score reads off backend.

backend property

backend: GraphBackend

The graph backend every candidate is scored on.

horizon property

horizon: int

One-step; the H-step FFG rollout is a deferred seam.

cost_per_cycle property

cost_per_cycle: int

Per-cycle step-evals = n_candidates (horizon is 1).

warrant property

warrant: Warrant

CORROBORATED, as for EFESelector.

The grid samples a continuum of actions and decides nothing about it.

select

select(
    belief: Belief, preference: Preference
) -> Float64[Array, p]

The grid action minimising G (the per-cycle work).

For each candidate: predict the joint (predicted_belief → μ⁺, Σ⁺), score with _ffg_efe_step, argmin over the grid. R is re-read at μ⁺ per candidate. It is constant under a fixed sensor and moves with the action under R(x), the dual effect. This is the RFC-001 hot path. Ties and scores that are not finite go as for EFESelector, and the first call compiles the work.

Source code in src/cpomdp/selection.py
def select(self, belief: Belief, preference: Preference) -> Float64[Array, "p"]:
    """The grid action minimising ``G`` (the per-cycle work).

    For each candidate: predict the joint (``predicted_belief`` → ``μ⁺``, ``Σ⁺``),
    score with ``_ffg_efe_step``, ``argmin`` over the grid. ``R`` is re-read at
    ``μ⁺`` per candidate. It is constant under a fixed sensor and moves with the
    action under ``R(x)``, the dual effect. This is the RFC-001 hot path. Ties
    and scores that are not finite go as for
    [`EFESelector`][cpomdp.EFESelector], and the first call compiles the work.
    """
    _target_and_precision(preference)  # refused here, not inside the compiled call
    return self._choose(belief, preference)

__repr__

__repr__() -> str

The horizon, the grid and the node the epistemic aims at.

Source code in src/cpomdp/selection.py
def __repr__(self) -> str:
    """The horizon, the grid and the node the epistemic aims at."""
    lo, hi = float(self._candidates[0, 0]), float(self._candidates[-1, 0])
    return (
        f"FFGEFESelector(horizon={self.horizon}, n_candidates={self.n_candidates}, "
        f"action_bounds=({lo}, {hi}), info_node={self._info_node})"
    )

  1. Magnus T. Koudahl, Wouter M. Kouw, and Bert de Vries. On epistemics in expected free energy for linear Gaussian state space models. Entropy, 23(12):1565, 2021. URL: https://doi.org/10.3390/e23121565, doi:10.3390/e23121565. ↩