Skip to content

Control

Steady-state LQR action selection, the action-side dual of the Kalman filter. Under a fixed sensor and fixed dynamics noise it is the action expected free energy chooses 1. The Agent builds one when you give it a StateGoal. Build one yourself to hand the Agent an LQRSelector beside a state Preference.

LQRController

LQRController(
    model: LinearGaussianModel,
    *,
    precision: ArrayLike,
    effort: ArrayLike,
    tol: float = 1e-12,
    max_iter: int = 1000,
)

Steady-state LQR action selection — the action-side dual of the filter.

It front-loads action: solve the control Riccati once for L∞, then action = -L∞·(mean − target) on every step. The estimation Riccati is its dual. A steady-state KalmanBackend solves that one once for K∞, and the pair is steady-state LQG. The Agent does not freeze its filter. It recomputes the Kalman gain every step, so its loop is this controller on a per-step filter.

The load-bearing claim (ADR-003) is that LQR is active inference here, not a substitute for it. For a fixed linear-Gaussian sensor and fixed dynamics noise the covariance recursion is control-independent, so Expected Free Energy's epistemic term is identical for every action and drops out of the argmin; EFE-minimising selection reduces to its pragmatic term, and the pragmatic term under a Gaussian preference is a quadratic cost whose optimum is exactly LQR. The epistemic term re-enters once the sensor noise or the dynamics noise depends on the state (ADR-006).

The two cost matrices are named for the preference they encode, not by LQR's traditional letters, which already mean the noise covariances on the model. That is the collision ADR-003 warns about. The names are the ones a StateGoal carries, precision and effort, and an Agent hands a goal's two straight through.

Parameters:

Name Type Description Default
model LinearGaussianModel

The linear-Gaussian model to act in. Must carry a control_matrix matrix — there is nothing to act with otherwise.

required
precision ArrayLike

How sharply the agent prefers the target, an (n, n) matrix, the state-cost weight. It is exactly the precision of the Gaussian preference centred at the target, exp(−½(state−target)ᵀ·precision·(state−target)), so a heavier precision buys a more aggressive controller.

required
effort ArrayLike

How much action costs, a (p, p) matrix, the action-cost weight. A heavier effort buys a gentler controller.

required
tol float

Absolute tolerance on successive cost-to-go iterates; convergence is declared when they stop moving by more than this.

1e-12
max_iter int

Iteration cap before the Riccati recursion is declared to have failed to converge.

1000

Raises:

Type Description
TypeError

If model is a coupled graph backend's model, which leaves out the couplings.

ValueError

If the model has no control_matrix, or a cost matrix does not match the state/action dimensions, is not symmetric, or fails its definiteness requirement (precision PSD, effort PD).

RuntimeError

If the control Riccati does not converge within max_iter, usually because (dynamics_matrix, control_matrix) is not stabilisable.

Source code in src/cpomdp/control.py
def __init__(
    self,
    model: LinearGaussianModel,
    *,
    precision: ArrayLike,
    effort: ArrayLike,
    tol: float = 1e-12,
    max_iter: int = 1000,
) -> None:
    _refuse_graph_interface(model, "LQRController")
    self._model = model
    self._precision, self._effort = _validated_costs(
        model, precision, effort, names=("precision", "effort")
    )
    self._gain = self._converge_to_steady_state(tol, max_iter)

model property

model: LinearGaussianModel

The model the gain was solved for, fixed at construction.

gain property

gain: Float64[Array, 'p n']

The steady-state feedback gain L∞, shape (p, n).

action

action(
    mean: ArrayLike, target: ArrayLike
) -> Float64[Array, p]

The action that drives the estimated state toward target.

One matrix-vector product, -L∞·(mean − target). All the work was front-loaded into L∞ at construction, so there is no optimisation in the loop. The mean − target shift turns the regulator (which drives its state to zero) into a controller that drives the state to target.

Parameters:

Name Type Description Default
mean ArrayLike

The current belief mean — the best estimate of the state, shape (n,).

required
target ArrayLike

The state to steer toward, shape (n,). It must be an equilibrium the dynamics can hold at zero action; aim at a non-equilibrium and a steady-state offset is left behind.

required

Returns:

Type Description
Float64[Array, p]

The action, shape (p,).

Raises:

Type Description
ValueError

If target is not a 1-D vector of length n.

Source code in src/cpomdp/control.py
def action(self, mean: ArrayLike, target: ArrayLike) -> Float64[Array, "p"]:
    """The action that drives the estimated state toward ``target``.

    One matrix-vector product, ``-L∞·(mean − target)``. All the work was
    front-loaded into ``L∞`` at construction, so there is no optimisation in
    the loop. The ``mean − target`` shift turns the regulator (which drives its
    state to zero) into a controller that drives the state to ``target``.

    Args:
        mean: The current belief mean — the best estimate of the state,
            shape ``(n,)``.
        target: The state to steer toward, shape ``(n,)``. It must be an
            equilibrium the dynamics can hold at zero action; aim at a
            non-equilibrium and a steady-state offset is left behind.

    Returns:
        The action, shape ``(p,)``.

    Raises:
        ValueError: If ``target`` is not a 1-D vector of length ``n``.
    """
    # self._gain : (p, n) L∞;  mean, target : (n,);  returns (p,)
    mean = jnp.asarray(mean, dtype=float)
    target = jnp.asarray(target, dtype=float)
    if target.shape != (self.model.n_states,):
        raise ValueError(
            f"target must be a 1-D vector of length {self.model.n_states} "
            f"(the state dimension), got shape {target.shape}"
        )
    return -self._gain @ (mean - target)

__repr__

__repr__() -> str

The steady-state gain, the one number the Riccati solve leaves behind.

Source code in src/cpomdp/control.py
def __repr__(self) -> str:
    """The steady-state gain, the one number the Riccati solve leaves behind."""
    gain = [[round(float(entry), 4) for entry in row] for row in self._gain]
    return f"LQRController(gain={gain})"

The finite-horizon schedule

In plain terms

A controller steers a system toward a goal. LQRController builds one by solving a puzzle that asks how hard to push right now, given where the state is and given forever to reach the goal. It answers by repeating one calculation until the answer stops changing. That final answer is the steady-state gain, and the controller keeps only it.

The intermediate answers were never junk. After one repetition the answer is how hard to push with one step left. After five, with five steps left. lqr_schedule runs the same calculation a chosen number of times, H, and keeps every answer. The result is a schedule: the right push with H steps left, then with H − 1, down to the last one.

The expected-free-energy planner looks a fixed number of steps ahead, applies the first action and plans again. A planner like that is using the "H steps left" rule at every step. The "forever" rule is close to it when H is large and visibly different when H is small. Held against the forever rule, the planner shows a gap that fades as H grows and looks exactly like a bug. first_gain is the matching rule, so no such gap is manufactured.

One convention to know. The cost is charged on the state each action arrives at, and nothing is charged after the last action. The planner's pragmatic term does the same accounting, so the two line up without adjustment.

lqr_schedule

lqr_schedule(
    model: LinearGaussianModel,
    *,
    horizon: int,
    precision: ArrayLike,
    effort: ArrayLike,
) -> LQRSchedule

Run the control Riccati recursion backward over horizon steps.

The same Bellman step LQRController iterates to a fixed point, run a declared number of times from a zero terminal cost and with every gain kept. With j steps remaining and W = precision + P_{j−1} the cost of what remains::

L_j = (effort + Bᵀ W B)⁻¹ (Bᵀ W A)
P_j = Aᵀ W A − (Aᵀ W B) L_j

starting from P_0 = 0. At horizon = 1, W is precision and the gain is the one-step regulator. The Control page of the API reference opens the same account in plain terms.

Parameters:

Name Type Description Default
model LinearGaussianModel

The linear-Gaussian model to act in. Must carry a control_matrix.

required
horizon int

H, how many steps the plan covers. At least one.

required
precision ArrayLike

The stage cost on the state, an (n, n) matrix, as a StateGoal carries it. (LQR's Q.)

required
effort ArrayLike

The stage cost on the action, a (p, p) matrix. (LQR's R.)

required

Returns:

Type Description
LQRSchedule

The schedule, with gains[k] the gain at step k of the plan and

LQRSchedule

costs_to_go[k] the cost still to pay before it, P_{H−k}.

Raises:

Type Description
TypeError

If model is a coupled graph backend's model, which leaves out the couplings.

ValueError

If horizon is below one, or on any of the cost and model conditions LQRController refuses.

Source code in src/cpomdp/control.py
def lqr_schedule(
    model: LinearGaussianModel,
    *,
    horizon: int,
    precision: ArrayLike,
    effort: ArrayLike,
) -> LQRSchedule:
    """Run the control Riccati recursion backward over ``horizon`` steps.

    The same Bellman step ``LQRController`` iterates to a fixed point, run a declared
    number of times from a zero terminal cost and with every gain kept. With ``j``
    steps remaining and ``W = precision + P_{j−1}`` the cost of what remains::

        L_j = (effort + Bᵀ W B)⁻¹ (Bᵀ W A)
        P_j = Aᵀ W A − (Aᵀ W B) L_j

    starting from ``P_0 = 0``. At ``horizon = 1``, ``W`` is ``precision`` and the
    gain is the one-step regulator. The Control page of the API reference opens the
    same account in plain terms.

    Args:
        model: The linear-Gaussian model to act in. Must carry a ``control_matrix``.
        horizon: ``H``, how many steps the plan covers. At least one.
        precision: The stage cost on the state, an ``(n, n)`` matrix, as a
            [`StateGoal`][cpomdp.StateGoal] carries it. (LQR's ``Q``.)
        effort: The stage cost on the action, a ``(p, p)`` matrix. (LQR's ``R``.)

    Returns:
        The schedule, with ``gains[k]`` the gain at step ``k`` of the plan and
        ``costs_to_go[k]`` the cost still to pay before it, ``P_{H−k}``.

    Raises:
        TypeError: If ``model`` is a coupled graph backend's model, which leaves
            out the couplings.
        ValueError: If ``horizon`` is below one, or on any of the cost and model
            conditions ``LQRController`` refuses.
    """
    _refuse_graph_interface(model, "lqr_schedule")
    if horizon < 1:
        raise ValueError(f"horizon must be at least 1, got {horizon}")
    precision, effort = _validated_costs(
        model, precision, effort, names=("precision", "effort")
    )
    dynamics_matrix = model.dynamics_matrix  # A
    assert model.control_matrix is not None  # refused by _validated_costs
    control_matrix = model.control_matrix  # B

    costs = [jnp.zeros_like(precision)]
    gains = []
    for _ in range(horizon):
        cost_to_go, gain = _riccati_step(
            precision + costs[-1], dynamics_matrix, control_matrix, effort
        )
        costs.append(cost_to_go)
        gains.append(gain)
    # Built with the fewest steps remaining first. The plan reads both the other way.
    return LQRSchedule(
        model=model,
        precision=precision,
        effort=effort,
        gains=jnp.stack(gains[::-1]),
        costs_to_go=jnp.stack(costs[::-1]),
    )

LQRSchedule dataclass

LQRSchedule(
    model: LinearGaussianModel,
    precision: Float64[Array, "n n"],
    effort: Float64[Array, "p p"],
    gains: Float64[Array, "H p n"],
    costs_to_go: Float64[Array, "H+1 n n"],
)

The gain schedule and cost-to-go of an H-step regulator.

The stage cost is charged on the state each action arrives at, and nothing is charged after the last one: the terminal cost is zero. That is the sum a receding-horizon planner scores over its lookahead, so first_gain is the gain such a planner applies at every step, and it is what a comparison against one has to use. LQRController.gain is its limit as H grows and differs from it at every finite H, by an amount that shrinks with H and reads as an error when the horizons are not matched.

The schedule carries the model and the two costs it was built from, so anything priced against it reads the same costs and noise the recursion did.

Parameters:

Name Type Description Default
model LinearGaussianModel

The model the schedule regulates.

required
precision Float64[Array, 'n n']

The stage cost on the state it was built with. (LQR's Q.)

required
effort Float64[Array, 'p p']

The stage cost on the action it was built with. (LQR's R.)

required
gains Float64[Array, 'H p n']

gains[k] is the gain applied at step k of the plan, with H − k steps remaining, shape (H, p, n). The action is −gains[k] · state.

required
costs_to_go Float64[Array, 'H+1 n n']

costs_to_go[k] is the matrix of the optimal cost stateᵀ · costs_to_go[k] · state still to pay before acting at step k, shape (H + 1, n, n). It is P_{H−k} in the recursion's count of steps remaining. costs_to_go[0] prices the whole plan, and costs_to_go[H] is zero.

required

horizon property

horizon: int

H, the number of steps planned.

first_gain property

first_gain: Float64[Array, 'p n']

The gain with all H steps remaining, the receding-horizon gain.

kalman_schedule

kalman_schedule(
    model: LinearGaussianModel,
    *,
    horizon: int,
    prior: Belief | None = None,
) -> KalmanSchedule

The exact filter's gain and covariance at each of horizon readings.

The covariance recursion the per-step KalmanBackend runs, run ahead of time and kept, since under fixed noise it never sees a reading.

Parameters:

Name Type Description Default
model LinearGaussianModel

The model the filter runs under, with fixed noise.

required
horizon int

How many readings, at least one.

required
prior Belief | None

The belief before the first reading. Defaults to the model's prior.

None

Returns:

Type Description
KalmanSchedule

The gains and covariances, indexed as the plan indexes its steps.

Raises:

Type Description
TypeError

If model is a coupled graph backend's model, which leaves out the couplings.

ValueError

If horizon is below one, if either noise depends on the state, or if prior is not over the model's state.

Source code in src/cpomdp/control.py
def kalman_schedule(
    model: LinearGaussianModel, *, horizon: int, prior: Belief | None = None
) -> KalmanSchedule:
    """The exact filter's gain and covariance at each of ``horizon`` readings.

    The covariance recursion the per-step ``KalmanBackend`` runs, run ahead of time
    and kept, since under fixed noise it never sees a reading.

    Args:
        model: The model the filter runs under, with fixed noise.
        horizon: How many readings, at least one.
        prior: The belief before the first reading. Defaults to the model's prior.

    Returns:
        The gains and covariances, indexed as the plan indexes its steps.

    Raises:
        TypeError: If ``model`` is a coupled graph backend's model, which leaves
            out the couplings.
        ValueError: If ``horizon`` is below one, if either noise depends on the state,
            or if ``prior`` is not over the model's state.
    """
    _refuse_graph_interface(model, "kalman_schedule")
    if horizon < 1:
        raise ValueError(f"horizon must be at least 1, got {horizon}")
    require_fixed_noise(model, _PRICES_FIXED_NOISE)
    prior = _prior_over(model, prior)
    gains, covariances = [], [prior.cov]
    for _ in range(horizon):
        gain, cov_post = _gain_and_posterior_cov(
            model.dynamics_matrix,
            model.observation_matrix,
            model.dynamics_noise,
            model.observation_noise,
            covariances[-1],
        )
        gains.append(gain)
        covariances.append(cov_post)
    return KalmanSchedule(gains=jnp.stack(gains), covariances=jnp.stack(covariances))

KalmanSchedule dataclass

KalmanSchedule(
    gains: Float64[Array, "H n m"],
    covariances: Float64[Array, "H+1 n n"],
)

The exact filter's gains and error covariances over H readings.

Both are data-independent, so they can be written down before a reading exists. The indexing follows the plan's: action k is chosen from the belief after k readings, and reading k + 1 arrives after it.

Parameters:

Name Type Description Default
gains Float64[Array, 'H n m']

gains[k] folds reading k + 1 into the estimate, shape (H, n, m).

required
covariances Float64[Array, 'H+1 n n']

covariances[k] is the error covariance of the estimate action k is chosen from, shape (H + 1, n, n). covariances[0] is the prior's.

required

horizon property

horizon: int

H, the number of readings scheduled.


  1. Magnus T. Koudahl, Wouter M. Kouw, and Bert de Vries. On epistemics in expected free energy for linear Gaussian state space models. Entropy, 23(12):1565, 2021. URL: https://doi.org/10.3390/e23121565, doi:10.3390/e23121565. ↩