Literature
Display
← All papers

The Geometry of Empowerment

Catherine Ji · Vivek Myers · Sergey Levine · Benjamin Eysenbach

Preprint posted · arXiv (preprint) ↗

Machine learning15 min text · CondensedOriginal paper ↗
Contents

At a glance

Having options and exercising them are different objectives. Potential empowerment measures the maximum control available from a state; effective empowerment measures the information a particular skill policy actually transmits to its future state. The paper relates both to the geometry of reachable state distributions.

High potential empowerment identifies states offering a large information radius. In finite-state settings, achieved skill information lower-bounds average adaptation under a specified Gaussian reward prior. This is not a ranking guarantee: more empowerment can mean worse adaptation, and information can be arbitrarily large while useful reward variation vanishes in continuous spaces or under misaligned reward priors. Experiments are illustrative tabular examples.

1 Introduction

A hiker can either move to a peak that offers many routes or choose a route and descend. Potential-empowerment navigation corresponds to seeking the peak; effective-empowerment skills exercise its options [1] [2]. Confusing these objectives obscures what skill-learning algorithms actually optimize.

Original 1 diagram with all labels
Figure 1: Two trained empowerment-maximizing policies characterize central states in an MDP: Consider an agent (robot) on a hill that is easier to descend than to ascend. In Fig. 1(a), the potential policy ascends the empowerment landscape to the top of the hill, using ℰpot​(s) as an intrinsic reward. In Fig. 1(b), The effective policy π eff maximizes Iπ​(Z;S+∣S0), associating each skill z with a downhill route.

Empowered States have Broad Access.

The paper formalizes earlier conjectures connecting empowerment, centrality, and broad access [3] [5] [6]. Centrality is measured through reachable discounted state distributions, rather than physical location alone. Separate results connect information to repeated-rollout hitting counts and, for an extended objective, temporal distances [7].

Information Geometries and Reward Geometries can Diverge in Continuous-State Markov decision process (MDP) s.

The positive adaptation result requires finite states and a particular reward distribution. Continuous rewards may barely distinguish states that remain perfectly distinguishable informationally; anisotropic priors can value directions that the skills do not control. Thus, information centrality does not generally imply reward adaptability.

2 Related Work

Classical empowerment is the capacity of the channel from an open-loop action sequence to a fixed-horizon future state [1] [8]. The distinction between capacity and information realized by a specific policy predates this work [2]. Earlier research connected empowerment empirically to central or gateway states, while graph-based centrality and power-seeking analyses largely used discrete settings [5] [6] [12].

This paper instead studies reactive skills at geometrically sampled times. Information geometry gives a distributional notion of centrality; temporal-distance geometry extends reachability reasoning beyond discrete graphs [7]. Neither is ordinary Euclidean distance between physical states.

Estimating and Maximizing Empowerment in Practice.

MISL methods learn latent-conditioned policies that make future states informative about the latent [18] [20] [21] [22]. Their direct objective resembles effective empowerment; learning diverse skills from one starting state does not itself learn to navigate toward states with maximal potential empowerment.

The paper supplies theoretical analysis and small tabular demonstrations, not a scalable continuous-control method. High-dimensional potential-empowerment maximization remains an open practical problem.

3 Preliminaries

The analysis represents policies by discounted distributions over visited states. This separates which future occupancies are feasible from which skill labels produce them.

3.1 Discounted State Occupancy

MDP problem setting.

The setting is a fully observed MDP with known transition structure for the illustrative experiments. A policy conditions on the current state, a skill latent, and the skill’s starting state. Fixing the latter two gives an ordinary policy. The theoretical skill source can be continuous even when states and actions are finite.

Reachable state-occupancy polytope.

A DSOM weights state visits by (1−γ)γᵗ. Equivalently, it is the distribution of the state observed at a geometric termination time, including time zero. Averaging over skills gives the skill-marginal occupancy.

For a fixed start state, achievable occupancies form a convex reachable set constrained by Bellman flow. In finite-state/action settings this set is a polytope inside the probability simplex. Its extent depends on controllability, stochastic dynamics, and discount horizon—not just the number of physically nearby states.

Equations 1–3
pπ​(s+=s∣s0,z)≜(1−γ)​∑t=0∞γt​pπ​(st=s∣s0,z),p^{\pi}(s_{+}=s\mid s_{0},z)\triangleq(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}p^{\pi}(s_{t}=s\mid s_{0},z),\addcontentsline{lla}{section}{\numberline q:dsom} (1)
pπ​(s+∣s0)=𝔼p⁡(z∣s0)​[pπ​(s+∣s0,z)],p^{\pi}(s_{+}\mid s_{0})=\mathbb{E}_{p(z\mid s_{0})}\big[p^{\pi}(s_{+}\mid s_{0},z)\big],\addcontentsline{lla}{section}{\numberline q:dsom_{m}arginal} (2)
𝒫⁡(s0)≜{pπ​(s+∣s0):π∈Π}⊆Δ⁡(𝒮),\mathcal{P}(s_{0})\triangleq\bigl\{\,p^{\pi}(s_{+}\mid s_{0})\;:\;\pi\in\Pi\,\bigr\}\;\subseteq\;\Delta(\mathcal{S}),\addcontentsline{lla}{section}{\numberline q:polytope} (3)

3.2 Empowerment Preliminaries

Potential empowerment optimizes over possible skill policies; effective empowerment evaluates one specified skill policy and source.

Potential and Effective Empowerment.

Definitions 3.1–3.3: potential empowerment is the maximum MI between a skill and its geometrically sampled future state, conditioned on the start. A potential policy uses this state-dependent capacity as intrinsic reward over its own trajectory. Effective empowerment is the MI achieved by a particular skill-conditioned policy [1] [2].

These definitions replace classical fixed-horizon action sequences with reactive skills and a discounted horizon. Effective empowerment is an average KL divergence from each skill’s occupancy to the marginal occupancy; potential empowerment maximizes this quantity over policies and, equivalently, sources.

A fixed atomless continuous source loses no expressive generality when policies permit arbitrary measurable relabeling of skills. This does not hold for a fixed finite uniform skill alphabet, nor does it guarantee that a restricted neural policy can learn the needed relabeling. Appendices D–E establish the distinction.

Equations 4
ℰpotH​(s)=maxp⁡(a0H−1)⁡I⁡(A0H−1;SH∣S0=s)\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}_{H}(s)=\max_{p(a_{0}^{H-1})}I(A_{0}^{H-1};S_{H}\mid S_{0}=s)
ℰpot​(s0)\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0}) =maxπ∈Π𝒵⁡Iπ​(Z;S+∣S0=s0).\displaystyle=\max_{\pi\in\Pi_{\mathcal{Z}}}I^{\pi}(Z;S_{+}\mid S_{0}=s_{0}).
πpot∗=arg​maxπ∈Π⁡𝔼π​[∑tγt​ℰpot​(st)].\displaystyle\lx@glossaries@gls@link{main}{pi_{m}athrmpot}{{{}}{\hyperlink{glo:pi_mathrmpot}{\pi_{\mathrm{pot}}}}}^{*}=\argmax_{\pi\in\Pi}\mathbb{E}_{\pi}\left[\sum_{t}\gamma^{t}\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{t})\right].
ℰeff​(π,s0)\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmeff}{{{}}{\hyperlink{glo:ge_mathrmeff}{{\mathcal{E}}_{\mathrm{eff}}}}}(\pi,s_{0}) =Iπ(Z;S+∣S0=s0)=𝔼p⁡(z∣s0)DKL[pπ(s+∣s0,z)∥pπ(s+∣s0)]\displaystyle=I^{\pi}(Z;S_{+}\mid S_{0}=s_{0})=\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}D_{\mathrm{KL}}[p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})]
ℰpot​(s0)=maxπ∈Π𝒵⁡maxp⁡(z∣s0)​Iπ​(Z;S+∣S0=s0).\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\max_{\pi\in\Pi_{\mathcal{Z}}}\;\max_{p(z\mid s_{0})}I^{\pi}(Z;S_{+}\mid S_{0}=s_{0}). (4)

Notation convention.

The paper writes π for an effective skill policy when unambiguous, π_eff when contrasting it with π_pot, and ℰ(s) for potential empowerment at a start state.

4 The Geometry of Empowerment

The results connect empowerment to an information radius, to repeated-rollout reachability, and to a limited downstream adaptation guarantee. Counterexamples then show where reward geometry departs from information geometry.

4.1 Information Geometry of Potential and Effective Empowerment Policies.

Figures 2–3 illustrate the two optimization problems: learn distinguishable skills from each candidate start, then navigate toward starts offering greater capacity.

Refer to caption
Figure 2: Skills measure the centrality of a state. We visualize the potential empowerment field ℰ⁡(s) (a) for each state in a tabular GridWorld. The central state has high potential empowerment, because (b) the central state gives an agent access to a maximal set of distinguishable skills. A potential empowerment policy ascends the potential empowerment landscape to the center state in (c), from which it can execute different skills. See Theorem 4.1 for the formal statement. See Appendix I for tabular experiment details.

Geometric intuition.

Each skill corresponds to a point in the space of state distributions. Maximizing effective empowerment spreads these points over the reachable set. The capacity-achieving mixture is a central occupancy distribution; it is not generally a single environment state. Comparing the resulting KL radii ranks starting states by potential empowerment.

High Empowerment States are Central in Information Geometry.

Theorem 4.1: potential empowerment equals the smallest worst-case KL divergence from reachable occupancies to a center distribution. In the finite case, optimal active skills occupy extreme points on the boundary of this information ball, and their mixture is its center.

The general version assumes jointly measurable policies on standard Borel spaces. Finite capacity gives an attainable unique center; the boundary statement is conditional on attaining the optimum and holds almost everywhere under the optimal skill source. It should not be read as a claim about unused skill labels.

In the key-and-room example, the potential policy first obtains a key, then moves into the room. Holding the key changes reachable futures, so the preferred state differs from a location-only notion of the grid’s center.

Equations 5–6
ℰpot​(s0)=maxπ∈Π𝒵,p⁡(z∣s0)⁡Iπ​(Z;S+∣S0=s0)\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\max_{\pi\in\Pi_{\mathcal{Z}},\,p(z\mid s_{0})}I^{\pi}(Z;S_{+}\mid S_{0}=s_{0}) =minq⁡(s+)∈Δ⁡(𝒮)maxp∈𝒫⁡(s0)DKL(p(s+)∥q(s+)).\displaystyle=\min_{q(s_{+})\in\Delta(\mathcal{S})}\max_{p\in\mathcal{P}(s_{0})}\;D_{\mathrm{KL}}\bigl(p({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr). (5)
DKL(pπ(s+∣s0,z)∥q⋆(s+))\displaystyle D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr) =maxp∈𝒫⁡(s0)DKL(p(s+)∥q⋆(s+))for every skill z.\displaystyle=\max_{p\in\mathcal{P}(s_{0})}D_{\mathrm{KL}}\bigl(p({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)\quad\text{for every skill }z. (6)
Original figure
(a) State occupancy simplex
Original figure
(b) Potential policy finds central states.
Figure 3: Potentially empowered states are central states in the polytope. Tabular intuition for Theorem 4.1. (a) We can use a probability simplex Δ⁡(𝒮) to visualize the discounted state occupancy measures of each skill (circles). Orange circles indicate the best set of skills starting from state s1, and teal circles indicate the best skills starting from state s2. from starting states s1 and s2 in arbitrary MDP. The skills starting from s1 define a smaller polytope (orange points) in the simplex than s2 (teal points) as a result of their transition dynamics, so the potential empowerment is higher at s2 than s1. (b) We use a GridWorld to illustrate the connection between centrality and tool use. An agent in the GridWorld begins at the bottom of the hallway. The agent can set down and pick up a key at the top of the hallway. The key provides the agent access to shaded regions of the MDP. (b, i) A potentially empowered agent picks up the key then navigates to the center of the room, correctly identifying the middle state with the key as the most central state. (b, ii) From the center, the agent can execute the maximal number of distinguishable skills. See §. I.3 for experimental details.

4.2 Empowerment, Bottlenecks, and Temporal Distance

Lemma 4.2: in a discrete MDP, restart independent rollouts from the same start and stop each at a geometric time. Count how many rollouts are required before the terminal state equals a target. Effective empowerment equals the average reduction in log expected rollout count obtained by fixing a skill rather than sampling from the source. Maximizing over skill policies gives potential empowerment.

This is not primitive-step first-passage time or the event of visiting the target anywhere during a rollout. Appendix C separately introduces contiguous skill executions, with another discount factor; that different objective has an exact temporal-distance interpretation applicable to continuous settings.

Grid examples show empowerment concentrating near doorways as the horizon increases. They illustrate a bottleneck effect whose location depends on dynamics and discount, rather than proving that every geometrical bottleneck universally maximizes empowerment.

Equation
ℰpot​(s0)=maxπeff∈Π𝒵⁡𝔼pπeff​(z,s+∣s0)​[log⁡𝔼⁡[Hπeff​(s0→s+)]−log⁡𝔼⁡[Hπeff​(s0→s+∣z)]].\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\max_{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}\in\Pi_{\mathcal{Z}}}\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(z,s_{+}\mid s_{0})}\left[\log\mathbb{E}[H^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{0}\to s_{+})]-\log\mathbb{E}[H^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{0}\to s_{+}\mid z)]\right].
Refer to caption
Refer to caption
Figure 4: Bottleneck states are high potential empowerment states. Consider GridWorlds with a door, or a “bottleneck”. (a) For short horizon behaviors where γ=0.5, the high empowerment regions are the central door (a, left) and the center of the two sides of the room (a, right). (b) Lifting γ=0.95 increases the horizon of possible behaviors, such that the empowerment tracks the location of the door. In §. 4.2 we show that empowerment measures the reduction in temporal distance to future states, which quantifies a bottleneck effect. See §. I.2 for experimental details.

4.3 Is Empowerment Optimal for Downstream Adaptation?

Potential empowerment lower bounds skill adaptation under an uninformed reward prior.

Theorem 4.3: in a finite state space, choose a full-support baseline b and independent mean-zero Gaussian rewards with variance 1/b(s) at each state. A uniform baseline makes the prior isotropic. After seeing the reward, adaptation selects the best pretrained skill; it does not retrain the policy or directly optimize a new occupancy.

Let J be the reward-prior average of that best skill’s expected reward under its normalized discounted occupancy. For any skill set and source, J ≥ I/[c√(2π)], where I is achieved effective empowerment and c = supₓ≥₀ ln(1+x)/√x ≈ 0.8. Potential empowerment enters when the skill set achieves capacity.

J is the Gaussian width of baseline-scaled skill-occupancy offsets. The guarantee averages over the specified reward prior; it is neither a per-task guarantee nor proof that greater capacity gives better adaptation. Illustrative examples show the bound becoming tighter with stronger commitment and looser as the number of target states grows.

Equations
r⁡(s+)∼𝒩⁡(0,1b⁡(s+)).r(s_{+})\ \sim\ \mathcal{N}\!\left(0,\ \frac{1}{b(s_{+})}\right).
J⁡(s0)≜𝔼r​[supz∈𝒵𝔼pπeff​(s+∣s0,z)​[r⁡(s+)]],J(s_{0})\triangleq\mathbb{E}_{r}\left[\sup_{z\in\mathcal{Z}}\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)}[r(s_{+})]\right],
J⁡(s0)≥I⁡(Z;S+∣S0=s0)c​2​π,c≜supx≥0ln⁡(1+x)x≈0.8.J(s_{0})\ \geq\ \frac{I(Z;S_{+}\mid S_{0}=s_{0})}{c\sqrt{2\pi}},\qquad c\triangleq\sup_{x\geq 0}\frac{\ln(1+x)}{\sqrt{x}}\approx 0.8.

The First Limitation of Empowered Adaptation: Gaussian widths versus KL divergence.

First limitation: different geometries can rank starts differently. Two reliably reachable targets can give more MI than three noisier targets, yet the three-target system can have greater Gaussian width and expected adapted reward. Figure 5 supplies a finite-state counterexample even under the isotropic prior.

Theorem 4.4 / Proposition G.2: infinitely many distinct target states can fit in an arbitrarily small metric ball. Skills reaching separate targets can transmit γ log K information for K targets, while their expected rewards differ by at most 2γε for any bounded 1-Lipschitz reward when all targets lie within radius ε. Information can grow without useful reward differentiation.

The formal construction uses distinct absorbing targets, not the Gaussian distributions drawn for intuition in Figure 6. The shrinking plotted spatial width is also different from the Gaussian-width functional used in the adaptation bound.

Refer to caption
Figure 5: Discrepancy between information geometry and reward adaptation geometry in discrete settings. (a) By going left/right from s, an agent can reach node s1 or s2 with high likelihood, so there are two easily separable skills from s. (d) Targeting nodes s1, s2, or s3 can make the agent inadvertently land in other nodes, so there are three somewhat separable skills from s. (b,e) The MDP s in (a) and (d) admit polytopes of feasible state occupancies in the simplex, visualized by the segment and triangle. (b) Axis u1 trades off occupancy of s1 vs. s2. (e) Axis u1 trades off occupying s2 vs. s3; axis u2 trades off occupying s1 vs. s2/s3. Return is linear in the state occupancy, so we can also represent reward functions as vectors in state-occupancy space corresponding to the reward weights. The color indicates the best adapted skill for that reward coordinate. The polytope in (e) has a larger Gaussian Width than that in (b), supporting better adaptation. (c,f) However, increased skill separability in (a) gives larger information radius and MI, where the black contour indicates the maximal MI. Thus, even though empowerment is higher for the 2 state system (a) versus the 3 state system (d), the 2 state system has worse adapted returns. See §. I.4 for details.
Refer to caption
Figure 6: The information ball can collapse in the continuous limit (Proposition G.2). Consider an arbitrary smooth reward function (level curves in gray) and some MDP that admits Gaussian skill state occupancy measures (colored distributions). While the potential empowerment of the starting state s∈𝒮 increases from subfigure (a) → (c), the skills have increasingly identical returns and, thus, provide progressively smaller adaptation benefit. The plotted width is the maximum distance between skill means, not to be confused with Gaussian Width. This width bounds the differences in expected return for the smooth reward and decreases from left to right. Therefore, large channel capacities do not imply adaptation for continuous MDP s and reward functions. See Proposition G.2 for the theoretical result.

The Second Limitation of Empowered Adaptation: Controllable Degrees of Freedom versus Rewarding Degrees of Freedom.

Second limitation: controllable directions may be irrelevant to rewards. If the baseline-whitened Gaussian reward covariance is Σ, the lower bound gains a factor √λ_min(Σ). A positive minimum eigenvalue retains a weaker guarantee; a singular or nearly singular covariance can make it uninformative.

Appendix H constructs finite-state skills with γ log K information but zero adaptation when reward variation is orthogonal to every skill-occupancy difference, even at fixed total covariance trace. Thus, empowerment alone cannot guarantee useful control under arbitrary task preferences.

5 Discussion, Limitations, & Future Work

Potential empowerment measures access to distinguishable future distributions; effective skills realize that access. The geometric characterization clarifies centrality and gives a reward-prior-averaged adaptation lower bound in the tabular setting.

Limitations & Future Work.

The lower bound does not preserve rankings. Continuous smooth rewards and general anisotropic priors can destroy useful alignment with MI. The paper establishes these limits theoretically and tests only small, known-dynamics tabular examples.

Future methods must align information-based control with reward-relevant variation while estimating potential empowerment at scale. Modifying continuous-control or scalable skill-learning methods is a proposed direction, not a demonstrated solution [6] [17] [21] [22] [37] [38].

Acknowledgments.

The authors acknowledge research feedback, support from Google and NSF graduate research funding, and Font Awesome icons adapted under CC BY 4.0.

Generative AI Disclosure Statement.

The authors disclose generative-AI assistance with figures, the paper website, tabular experiment implementation, and minor readability edits.

Glossary

The paper’s main terms are DSOM, MI, KL divergence, MISL, and the potential/effective policy distinction. Definitions are available through the term links and the Terms tab. “Gaussian width” refers to expected maximal projection onto Gaussian reward directions, rather than an ordinary spatial diameter.

Appendix A Preliminaries for Proofs

The proofs use four standard facts [32] [40] [41] [42] [43]: MI is concave in the input source for a fixed channel and convex in the channel for a fixed source; capacity equals a minimax information radius; KL divergence is bounded by log(1+χ²); and an atomless reference random variable can be measurably transformed into a desired distribution on a standard Borel space.

These facts respectively justify source optimization, the occupancy-center characterization, the adaptation inequality, and replacing source learning with skill relabeling. The continuous-source assumption matters for the last step.

Equations
supPX∈Δ⁡(𝒳)I(X;Y)=infq⁡(y)supx∈𝒳DKL(PY|X=x(y)∥q(y)).\sup_{P_{X}\in\Delta(\mathcal{X})}I(X;Y)=\inf_{q({\color[rgb]{0.5,0.5,0.5}y})}\sup_{x\in\mathcal{X}}D_{\mathrm{KL}}\bigl(P_{Y\mid X=x}({\color[rgb]{0.5,0.5,0.5}y})\,\|\,q({\color[rgb]{0.5,0.5,0.5}y})\bigr).
DKL(P∥Q)≤log(1+χ2(P∥Q)).D_{\mathrm{KL}}(P\|Q)\leq\log\bigl(1+\chi^{2}(P\|Q)\bigr).
P=pπeff​(z∣s0,s+)andQ=p⁡(z∣s0).P=p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0},s_{+})\quad\text{and}\quad Q=p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0}).
T⁡(Z~)∼μ.T(\tilde{Z})\sim\mu.

Appendix B Information Geometry of Empowerment

Assumption B.1: states, actions, and skills are standard Borel spaces, with policies jointly measurable in state and skill. This makes the skill-to-occupancy map measurable. Since a skill affects the sampled future only through its occupancy, replacing the latent by that occupancy preserves MI.

Theorem B.2: optimizing the induced channel yields the infimum over center distributions of the supremum KL divergence to reachable occupancies. Both sides may be infinite. With finite capacity the center is unique and attained; if a policy/source attains capacity, its marginal occupancy equals that center.

The proof uses the decomposition of average KL to any center into MI plus the KL of the marginal to that center. At an attained optimum, active skills have maximal radius almost surely; strict convexity excludes non-extreme active occupancies. In finite tabular settings the reachable set is a polytope, so these extreme points are vertices and capacity is at most log|S|.

Equations 7–9
𝒵→𝜋𝒫⁡(s0),z⟼pπ​(s+∣s0,z).\displaystyle\mathcal{Z}\xrightarrow{\ \pi\ }\mathcal{P}(s_{0}),\qquad z\longmapsto p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z).
I⁡(Z;S+∣S0)=I⁡(PZ;S+∣S0).\displaystyle I(Z;S_{+}\mid S_{0})=I(P_{Z};S_{+}\mid S_{0}).
(s,z)↦π(⋅∣s,z)∈Δ(𝒜)(s,z)\mapsto\pi(\cdot\mid s,z)\in\Delta(\mathcal{A})
ℰpot(s0)=infq⁡(s+)supp∈𝒫⁡(s0)DKL(p(s+)∥q(s+)).\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\inf_{q({\color[rgb]{0.5,0.5,0.5}s_{+}})}\sup_{p\in\mathcal{P}(s_{0})}D_{\mathrm{KL}}\bigl(p({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr).
q⋆=pπ​(s+∣s0).q^{\star}=p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0}).
DKL(pπ(s+∣s0,z)∥q⋆(s+))=𝔼ℰpot(s0),D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)=\mathbb{E}\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0}),
pπ​(s+∣s0,z)∈𝒫⁡(s0),p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\in\mathcal{P}(s_{0}),
p∈𝒫⁡(s0),S+∼p.p\in\mathcal{P}(s_{0}),\quad S_{+}\sim p.
ℰpot(s0)=infq⁡(s+)supp∈𝒫⁡(s0)DKL(p(s+)∥q(s+)).\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\inf_{q({\color[rgb]{0.5,0.5,0.5}s_{+}})}\sup_{p\in\mathcal{P}(s_{0})}D_{\mathrm{KL}}\bigl(p({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr).
𝔼p⁡(z∣s0)[DKL(pπ(s+∣s0,z)∥q(s+))]=Iπ(Z;S+∣S0=s0)+DKL(pπ(s+∣s0)∥q(s+))\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left[D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)\right]=I^{\pi}(Z;S_{+}\mid S_{0}=s_{0})+D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})\|q({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)
C\displaystyle C ≥𝔼p⁡(z∣s0)[DKL(pπ(s+∣s0,z)∥q⋆(s+))]\displaystyle\geq\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left[D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)\right] (7)
=C+DKL(pπ(s+∣s0)∥q⋆(s+))\displaystyle=C+D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr) (Golden formula)
≥C.\displaystyle\geq C. (8)
q⋆=pπ​(s+∣s0).q^{\star}=p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0}).
DKL(pπ(s+∣s0,z)∥q⋆(s+))=C\displaystyle D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)=C (9)
pπ​(s+∣s0,z)=λ​p1+(1−λ)​p2.p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)=\lambda p_{1}+(1-\lambda)p_{2}.
DKL(pπ(s+∣s0,z)∥q⋆(s+))\displaystyle D_{\mathrm{KL}}\bigl(p^{\pi}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr) <λDKL(p1(s+)∥q⋆(s+))+(1−λ)DKL(p2(s+)∥q⋆(s+))\displaystyle<\lambda D_{\mathrm{KL}}\bigl(p_{1}({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)+(1-\lambda)D_{\mathrm{KL}}\bigl(p_{2}({\color[rgb]{0.5,0.5,0.5}s_{+}})\|q^{\star}({\color[rgb]{0.5,0.5,0.5}s_{+}})\bigr)
≤C,\displaystyle\leq C,

B.1 Log reachability interpretation of potential empowerment in discrete deterministic settings

Lemma B.3: for the classical fixed-horizon, open-loop definition in a discrete deterministic MDP, empowerment is log of the number of states reachable in exactly H steps. Determinism makes conditional output entropy zero; choosing one action sequence per reachable endpoint with equal probability achieves the entropy maximum.

This is a special-case reachability formula, not a replacement for the paper’s stochastic discounted-skill objective.

Equations
ℰpotH​(s)=maxp⁡(a0H−1)⁡I⁡(A0H−1;SH∣S0=s).\displaystyle\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}_{H}(s)=\max_{p(a_{0}^{H-1})}I(A_{0}^{H-1};S_{H}\mid S_{0}=s).
ℛH​(s)={s+∈𝒮:∃a0H−1∈𝒜H​ that reaches ​SH=s+​ from ​S0=s},\mathcal{R}_{H}(s)=\{s_{+}\in\mathcal{S}:\ \exists\,a_{0}^{H-1}\in\mathcal{A}^{H}\text{ that reaches }S_{H}=s_{+}\text{ from }S_{0}=s\},
ℰpotH​(s)=log⁡|ℛH​(s)|.\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}_{H}(s)=\log|\mathcal{R}_{H}(s)|.
I⁡(A0H−1;SH∣S0=s)=H⁡(SH∣S0=s).I(A_{0}^{H-1};S_{H}\mid S_{0}=s)=H(S_{H}\mid S_{0}=s).

Appendix C Empowerment and Bottleneck States

The appendix distinguishes independent restarted rollouts from a different process that resamples skills along one continuing trajectory.

C.1 Bottleneck states are high-empowerment states

A restarted rollout succeeds only when its geometrically sampled endpoint equals the target. Its rollout count is geometric, so expected counts are reciprocals of the marginal or skill-conditioned occupancy probabilities. Substituting these reciprocals into the MI log ratio proves Lemma 4.2 for any policy; optimizing gives potential empowerment.

Equations 10
Hπ​(s0→s+)≜inf{k≥1:S+(k)=s+},H^{\pi}(s_{0}\to s_{+})\triangleq\inf\left\{k\geq 1:S_{+}^{(k)}=s_{+}\right\},
𝔼⁡[Hπ​(s0→s+)]=1pπ​(s+∣s0),𝔼⁡[Hπ​(s0→s+∣z)]=1pπ​(s+∣s0,z).\mathbb{E}[H^{\pi}(s_{0}\to s_{+})]=\frac{1}{p^{\pi}(s_{+}\mid s_{0})},\quad\mathbb{E}[H^{\pi}(s_{0}\to s_{+}\mid z)]=\frac{1}{p^{\pi}(s_{+}\mid s_{0},z)}.
Iπ​(Z;S+∣S0=s0)\displaystyle I^{\pi}(Z;S_{+}\mid S_{0}=s_{0}) =𝔼pπ​(z,s+∣s0)​log⁡pπ​(s+∣s0,z)pπ​(s+∣s0)\displaystyle=\mathbb{E}_{p^{\pi}(z,s_{+}\mid s_{0})}\log\frac{p^{\pi}(s_{+}\mid s_{0},z)}{p^{\pi}(s_{+}\mid s_{0})}
=𝔼pπ​(z,s+∣s0)​[log⁡𝔼⁡[Hπ​(s0→s+)]−log⁡𝔼⁡[Hπ​(s0→s+∣z)]].\displaystyle=\mathbb{E}_{p^{\pi}(z,s_{+}\mid s_{0})}\left[\log\mathbb{E}[H^{\pi}(s_{0}\to s_{+})]-\log\mathbb{E}[H^{\pi}(s_{0}\to s_{+}\mid z)]\right]. (10)

C.2 Temporal distances over contiguous skill rollouts

For contiguous execution, run a skill until its geometric endpoint, then sample another skill from the current state and continue. γ discounts time within skills; a second factor β discounts the sequence of skill endpoints. The conditional occupancy fixes only the first skill and then follows the source for later skills.

Proposition C.1: MI between the initial skill and this longer-horizon future equals the expected reduction in temporal distance from committing to that first skill. The proof subtracts two negative log occupancy ratios whose shared self-reachability normalization cancels. Optimizing both policy and source defines potential contiguous empowerment.

In discrete spaces, temporal distance can be written as −log E[βᴴ] for appropriate endpoint-process hitting counts, rather than simply E[H]. The distributional formulation extends beyond discrete hitting probabilities under the temporal-distance setup.

Equations 11
Mβπ​(g∣s)≜(1−β)​∑k=0∞βk​pπ​(Xk=g∣X0=s).M_{\beta}^{\pi}(g\mid s)\triangleq(1-\beta)\sum_{k=0}^{\infty}\beta^{k}p^{\pi}(X_{k}=g\mid X_{0}=s).
Mβπ​(g∣s,z)≜(1−β)​∑k=0∞βk​pπ​(Xk=g∣X0=s,Z0=z),M_{\beta}^{\pi}(g\mid s,z)\triangleq(1-\beta)\sum_{k=0}^{\infty}\beta^{k}p^{\pi}(X_{k}=g\mid X_{0}=s,Z_{0}=z),
𝔼z∼p⁡(z∣s)​[Mβπ​(g∣s,z)]=Mβπ​(g∣s).\mathbb{E}_{z\sim p({\color[rgb]{0.5,0.5,0.5}z}\mid s)}\left[M_{\beta}^{\pi}(g\mid s,z)\right]=M_{\beta}^{\pi}(g\mid s).\addcontentsline{lla}{section}{\numberline q:rollout_{s}uccessor_{c}onsistency} (11)
ℰeffcontig​(π,s0)\displaystyle{\mathcal{E}}^{\mathrm{contig}}_{\mathrm{eff}}(\pi,s_{0}) ≜Iπ​(Z;G∣X0=s0),\displaystyle\triangleq I^{\pi}(Z;G\mid X_{0}=s_{0}),
ℰpotcontig​(s0)\displaystyle{\mathcal{E}}^{\mathrm{contig}}_{\mathrm{pot}}(s_{0}) ≜supπ∈Π𝒵,p⁡(z∣s)ℰeffcontig​(π,s0).\displaystyle\triangleq\sup_{\pi\in\Pi_{\mathcal{Z}},\,p(z\mid s)}{\mathcal{E}}^{\mathrm{contig}}_{\mathrm{eff}}(\pi,s_{0}).
dβπ​(s,g)≜−log⁡Mβπ​(g∣s)Mβπ​(g∣g)d^{\pi}_{\beta}(s,g)\triangleq-\log\frac{M_{\beta}^{\pi}(g\mid s)}{M_{\beta}^{\pi}(g\mid g)}
dβπ​(s,g∣z)≜−log⁡Mβπ​(g∣s,z)Mβπ​(g∣g),d^{\pi}_{\beta}(s,g\mid z)\triangleq-\log\frac{M_{\beta}^{\pi}(g\mid s,z)}{M_{\beta}^{\pi}(g\mid g)},
ℰeffcontig​(π,s0)=𝔼Mβπ​(z,g∣s0)​[dβπ​(s0,g)−dβπ​(s0,g∣z)].{\mathcal{E}}^{\mathrm{contig}}_{\mathrm{eff}}(\pi,s_{0})=\mathbb{E}_{M_{\beta}^{\pi}({\color[rgb]{0.5,0.5,0.5}z},{\color[rgb]{0.5,0.5,0.5}g}\mid s_{0})}\left[d_{\beta}^{\pi}(s_{0},g)-d_{\beta}^{\pi}(s_{0},g\mid z)\right].
ℰpotcontig​(s0)=supπ∈Π𝒵,p⁡(z∣s)𝔼Mβπ​(z,g∣s0)​[dβπ​(s0,g)−dβπ​(s0,g∣z)].{\mathcal{E}}^{\mathrm{contig}}_{\mathrm{pot}}(s_{0})=\sup_{\pi\in\Pi_{\mathcal{Z}},\,p(z\mid s)}\mathbb{E}_{M_{\beta}^{\pi}({\color[rgb]{0.5,0.5,0.5}z},{\color[rgb]{0.5,0.5,0.5}g}\mid s_{0})}\left[d_{\beta}^{\pi}(s_{0},g)-d_{\beta}^{\pi}(s_{0},g\mid z)\right].
dβπ​(s0,g)−dβπ​(s0,g∣z)=log⁡Mβπ​(g∣s0,z)Mβπ​(g∣s0).d_{\beta}^{\pi}(s_{0},g)-d_{\beta}^{\pi}(s_{0},g\mid z)=\log\frac{M_{\beta}^{\pi}(g\mid s_{0},z)}{M_{\beta}^{\pi}(g\mid s_{0})}.
Hcontigπ​(s→g)≜inf{k≥0:Xk=g}H_{\mathrm{contig}}^{\pi}(s\to g)\triangleq\inf\{k\geq 0:X_{k}=g\}
dβπ​(s,g)=−log⁡𝔼⁡[βHcontigπ​(s→g)],dβπ​(s,g∣z)=−log⁡𝔼⁡[βHcontigπ​(s→g∣z)]for ​g≠s.d_{\beta}^{\pi}(s,g)=-\log\mathbb{E}\left[\beta^{H_{\mathrm{contig}}^{\pi}(s\to g)}\right],\quad d_{\beta}^{\pi}(s,g\mid z)=-\log\mathbb{E}\left[\beta^{H_{\mathrm{contig}}^{\pi}(s\to g\mid z)}\right]\quad\text{for }g\neq s.

Note on different empowerment expressions.

Contiguous empowerment is mathematically different from the main single-skill objective. Its channel also depends on the source used for later skills, so experiments optimize a parameterized source as well as candidate policies. Figure 7 shows similar bottleneck patterns, without establishing equality of the objectives.

Refer to caption
Figure 7: Empowerment over contiguous rollouts concentrates at bottlenecks (Proposition C.1). The figures show the MI of the contiguous process at each start state on the 8×8 grids from Fig. 4, with (a) a center opening and (b) a corner opening. The bottleneck openings have high potential contiguous empowerment.

Appendix D Skill Selection: Fixed Continuous Uniform Priors are General

Lemma D.1: a fixed atomless continuous source can represent any discrete or continuous skill distribution on a standard Borel space through a measurable map. Composing the policy with that map preserves the joint behavior and MI, so optimizing over unrestricted measurable policies removes the need to optimize the source separately.

The skill is sampled once and held fixed during the rollout. This is an expressivity result: it does not assert that finite categorical priors, limited parameterizations, or practical training preserve capacity.

Equations
supπ∈Π𝒵,p⁡(z∣s0)∈Δ⁡(𝒵)Iπ​(Z;S+∣S0=s0)=supπ~∈Π𝒵,Z~∼UIπ~​(Z~;S+∣S0=s0),\sup_{\pi\in\Pi_{\mathcal{Z}},\,p(z\mid s_{0})\in\Delta(\mathcal{Z})}I^{\pi}(Z;S_{+}\mid S_{0}=s_{0})=\sup_{\tilde{\pi}\in\Pi_{\mathcal{Z}},\,\tilde{Z}\sim U}I^{\tilde{\pi}}(\tilde{Z};S_{+}\mid S_{0}=s_{0}),
π~​(a∣s,z~;s0)=π⁡(a∣s,Ts0​(z~);s0).\tilde{\pi}(a\mid s,\tilde{z};s_{0})=\pi(a\mid s,T_{s_{0}}(\tilde{z});s_{0}).
Iπ~​(Z~;S+∣S0=s0)=Iπ~​(Ts0​(Z~);S+∣S0=s0)=Iπ​(Z;S+∣S0=s0).I^{\tilde{\pi}}(\tilde{Z};S_{+}\mid S_{0}=s_{0})=I^{\tilde{\pi}}\bigl(T_{s_{0}}(\tilde{Z});S_{+}\mid S_{0}=s_{0}\bigr)=I^{\pi}(Z;S_{+}\mid S_{0}=s_{0}).

Appendix E Continuous versus Discrete Skill Spaces

Lemma E.1: a finite uniform skill prior can strictly lose capacity. In the two-skill construction, one action always reaches one target while the other reaches the alternative with probability β. For β=1/2, the optimal probability of the second skill is 2/5, rather than 1/2.

Convexity puts the optimal channel at an extreme action assignment; optimizing the source then shows a strict gain over the best uniform-prior channel. Thus, channel optimization alone does not fix the finite-uniform-source restriction.

Equations
p⁡(s1∣s0,a)=β​a,p⁡(s2∣s0,a)=1−β​a,0<β<1.p(s_{1}\mid s_{0},a)=\beta a,\qquad p(s_{2}\mid s_{0},a)=1-\beta a,\qquad 0<\beta<1.
I⁡(Z;S+∣S0=s0)=h⁡(p⁡(Z=1∣s0)​β)−p⁡(Z=1∣s0)​h​(β).I(Z;S_{+}\mid S_{0}=s_{0})=h\bigl(p(Z=1\mid s_{0})\,\beta\bigr)-p(Z=1\mid s_{0})\,h(\beta).
∂I⁡(Z;S+∣S0=s0)∂p⁡(Z=1∣s0)=β​log⁡1−p⁡(Z=1∣s0)​βp⁡(Z=1∣s0)​β−h⁡(β),\frac{\partial\,I(Z;S_{+}\mid S_{0}=s_{0})}{\partial\,p(Z=1\mid s_{0})}=\beta\log\frac{1-p(Z=1\mid s_{0})\,\beta}{p(Z=1\mid s_{0})\,\beta}-h(\beta),
p⋆​(Z=1∣s0)=1β⁡(1+exp⁡(h⁡(β)/β)).p^{\star}(Z=1\mid s_{0})=\frac{1}{\beta(1+\exp(h(\beta)/\beta))}.
h⁡(1/4)−12​h​(1/2),h(1/4)-\frac{1}{2}h(1/2),
h⁡(1/5)−25​h​(1/2).h(1/5)-\frac{2}{5}h(1/2).

Comment on VIC [ 18 ] and DIAYN [ 20 ] .

VIC and DIAYN implementations use uniform categorical skills [18] [20]. The finite counterexample identifies an expressivity restriction; it does not invalidate their empirical results. Atomless continuous priors avoid this restriction in the unrestricted policy class, and several scalable methods already use them [21] [22].

Appendix F Empowerment Lower Bounds Tabular Adaptation

Subtract the marginal occupancy from each skill occupancy and divide coordinatewise by √b to obtain offsets u_z. Whiten rewards as g=√b·r. The common marginal return averages to zero, so expected best-skill return becomes the Gaussian width E_g sup_z〈g,u_z〉.

The 1/b reward-variance scaling preserves return distributions when states and their occupancy mass are proportionally subdivided. Figure 8 demonstrates this on two states split into four; unit-variance rewards would instead shrink return variation.

The proof first bounds Gaussian width below using the root-mean-square offset norm. It then bounds MI above by c times that norm, applying the KL–χ² comparison and Cauchy–Schwarz. Combining the inequalities yields Theorem 4.3. This is a finite-state result for the specified independent Gaussian prior.

Figures 9–10 vary control commitment and target count. For those illustrative constructions the bound tightens with stronger commitment but loosens as more distinct targets become available; they do not establish a general tightness theorem.

Equations 12
J⁡(s0)=𝔼g​supz∈𝒵⟨g,uz⟩,J(s_{0})=\mathbb{E}_{g}\sup_{z\in\mathcal{Z}}\,\bigl\langle g,\ u_{z}\bigr\rangle,\addcontentsline{lla}{section}{\numberline q:J_{w}idth} (12)
𝔼pπeff​(s+∣s0,z)​[r⁡(s+)]=p1​r​(1)+p2​r​(2)∼𝒩⁡(0, 2​p12+2​p22).\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)}[r(s_{+})]=p_{1}\,r(1)+p_{2}\,r(2)\ \sim\ \mathcal{N}\!\left(0,\ 2p_{1}^{2}+2p_{2}^{2}\right).
p12​(r′​(1​a)+r′​(1​b))+p22​(r′​(2​a)+r′​(2​b))∼𝒩⁡(0,p124⋅8+p224⋅8)=𝒩⁡(0, 2​p12+2​p22),\tfrac{p_{1}}{2}\bigl(r^{\prime}(1a)+r^{\prime}(1b)\bigr)+\tfrac{p_{2}}{2}\bigl(r^{\prime}(2a)+r^{\prime}(2b)\bigr)\ \sim\ \mathcal{N}\!\left(0,\ \tfrac{p_{1}^{2}}{4}\cdot 8+\tfrac{p_{2}^{2}}{4}\cdot 8\right)=\mathcal{N}\!\left(0,\ 2p_{1}^{2}+2p_{2}^{2}\right),
Original 8 diagram with all labels
Figure 8: State refinement leaves return distribution unchanged (Example F.1). Bars display the occupancy pπeff​(s+∣s0,z)=(p1,p2) of a skill over the coarse states (a) and the occupancy over the split states (b). Each panel lists the reference mass and per-state reward prior r⁡(s+)∼𝒩⁡(0,1/b⁡(s+)). Splitting states halves the reference mass and doubles the reward variance. Thus, the return obeys the same distribution 𝒩⁡(0, 2​p12+2​p22) in both MDP s.
Equations 13–Cauchy-Schwarz
J⁡(s0)​≥step 1​12​π​(𝔼p⁡(z∣s0)​‖uz‖2)1/2​≥step 2​I⁡(Z;S+∣S0=s0)c​2​π.J(s_{0})\ \overset{\text{step 1}}{\geq}\ \frac{1}{\sqrt{2\pi}}\left(\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left\|u_{z}\right\|^{2}\right)^{1/2}\ \overset{\text{step 2}}{\geq}\ \frac{I(Z;S_{+}\mid S_{0}=s_{0})}{c\sqrt{2\pi}}.\addcontentsline{lla}{section}{\numberline q:linear_{r}ate_{p}ivot} (13)
supz′∈𝒵𝔼pπeff​(s+∣s0,z′)​[r⁡(s+)]≥max⁡(𝔼pπeff​(s+∣s0,z)​[r⁡(s+)],𝔼pπeff​(s+∣s0)​[r⁡(s+)]).\sup_{z^{\prime}\in\mathcal{Z}}\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z^{\prime})}[r(s_{+})]\ \geq\ \max\left(\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)}[r(s_{+})],\ \mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})}[r(s_{+})]\right).
J⁡(s0)≥12​𝔼r​|𝔼pπeff​(s+∣s0,z)​[r⁡(s+)]−𝔼pπeff​(s+∣s0)​[r⁡(s+)]|.J(s_{0})\ \geq\ \tfrac{1}{2}\,\mathbb{E}_{r}\left|\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)}[r(s_{+})]-\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})}[r(s_{+})]\right|.
𝔼pπeff​(s+∣s0,z)​[r⁡(s+)]−𝔼pπeff​(s+∣s0)​[r⁡(s+)]=⟨g,uz⟩.\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0},z)}[r(s_{+})]-\mathbb{E}_{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s_{0})}[r(s_{+})]=\left\langle g,\ u_{z}\right\rangle.
J⁡(s0)\displaystyle J(s_{0})\ ≥12​𝔼r​|⟨g,uz⟩|\displaystyle\geq\ \tfrac{1}{2}\,\mathbb{E}_{r}\left|\left\langle g,\ u_{z}\right\rangle\right|
=12​π​‖uz‖.\displaystyle=\ \frac{1}{\sqrt{2\pi}}\left\|u_{z}\right\|.
I(Z;S+∣S0=s0)=∑s+pπeff(s+∣s0)DKL(pπeff(z∣s0,s+)∥p(z∣s0)).I(Z;S_{+}\mid S_{0}=s_{0})=\sum_{s_{+}}p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\,D_{\mathrm{KL}}\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0},s_{+})\,\big\|\,p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})\bigr).
χ2(pπeff(z∣s0,s+)∥p(z∣s0))\displaystyle\chi^{2}\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0},s_{+})\,\big\|\,p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})\bigr) =𝔼p⁡(z∣s0)​[(pπeff​(s+∣s0,z)pπeff​(s+∣s0)−1)2]\displaystyle=\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left[\left(\frac{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0},z)}{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})}-1\right)^{2}\right]
=𝔼p⁡(z∣s0)​[(pπeff​(s+∣s0,z)−pπeff​(s+∣s0))2]pπeff​(s+∣s0)2.\displaystyle=\frac{\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\bigl[\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0},z)-p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\bigr)^{2}\bigr]}{p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})^{2}}.
I⁡(Z;S+∣S0=s0)\displaystyle I(Z;S_{+}\mid S_{0}=s_{0}) ≤c​∑s+pπeff​(s+∣s0)​χ2(pπeff(z∣s0,s+)∥p(z∣s0))\displaystyle\leq c\sum_{s_{+}}p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\sqrt{\chi^{2}\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0},s_{+})\,\big\|\,p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})\bigr)} (Theorem A.3)
=c​∑s+𝔼p⁡(z∣s0)​[(pπeff​(s+∣s0,z)−pπeff​(s+∣s0))2]\displaystyle=c\sum_{s_{+}}\sqrt{\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\bigl[\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0},z)-p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\bigr)^{2}\bigr]}
=c​∑s+b⁡(s+)​𝔼p⁡(z∣s0)​[(pπeff​(s+∣s0,z)−pπeff​(s+∣s0))2]b⁡(s+)\displaystyle=c\sum_{s_{+}}\sqrt{b(s_{+})}\,\sqrt{\frac{\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\bigl[\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0},z)-p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\bigr)^{2}\bigr]}{b(s_{+})}}
≤c​(∑s+𝔼p⁡(z∣s0)​[(pπeff​(s+∣s0,z)−pπeff​(s+∣s0))2]b⁡(s+))1/2\displaystyle\leq c\left(\sum_{s_{+}}\frac{\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\bigl[\bigl(p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0},z)-p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(s_{+}\mid s_{0})\bigr)^{2}\bigr]}{b(s_{+})}\right)^{1/2} (Cauchy-Schwarz)
=c​(𝔼p⁡(z∣s0)​‖uz‖2)1/2.\displaystyle=c\left(\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left\|u_{z}\right\|^{2}\right)^{1/2}.
Original figure
Figure 9: MI bound as a function of skill commitment (Theorem 4.3). Consider the MDP consisting of 5 states with a central starting state, where any left/right/up/down action sends probability mass to absorbing left/right/up/down states. Actions left/right/up/down have a level of commitment t, which measures the additional mass above uniform assigned to the corresponding state. When commitment is small, probability is almost spread equally over all four states regardless of action so there is little control. In this regime, the adaptation J dominates the lower bound. As the commitment increases, actions/skills commit to states and the bound tightens.
Refer to caption
Figure 10: MI bound as a function of absorbing states (Theorem 4.3). Consider an MDP with one starting state and K absorbing target states. Each of the K actions transitions deterministically to a distinct target. We normalize occupancies over the targets (neglecting the occupancy of the starting state). Thus, K counts the maximum possible number of total skills. The scaled MI (orange) correctly lower bounds the adaptation J (purple). The bound weakens as the number of absorbing target states in the MDP grows.

Appendix G When Information Geometry is Insufficient for Adaptation

Proposition G.1: construct K distinct absorbing targets, one per skill. Every discounted occupancy shares mass 1−γ at the start and places γ at its target. Observing a target identifies the skill, giving MI=γ log K; a continuously indexed set of distinct targets gives unbounded MI.

Proposition G.2: place all targets inside a radius-ε ball containing infinitely many states. Any bounded 1-Lipschitz reward varies by at most 2ε between targets, so normalized discounted skill returns differ by at most 2γε. Increasing the number of targets changes information without increasing this bound.

The shared start-state mass cancels in return differences. The result limits the benefit of choosing between skills; their absolute returns need not be near zero. Exact state distinguishability can therefore coexist with arbitrarily little adaptation advantage, preventing an unrestricted extension of the tabular Gaussian-prior guarantee to smooth continuous rewards.

Equations
ℰpot​(s)=maxπeff⁡Iπeff​(Z;S+∣S0=s)=γ​log⁡K.\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s)=\max_{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}I^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(Z;S_{+}\mid S_{0}=s)=\gamma\log K.
pπeff​(s+∣s,z)=(1−γ)​δs+γ​δsz,p^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z)=(1-\gamma)\,\delta_{s}+\gamma\,\delta_{s_{z}},
Iπeff​(Z;S+∣S0=s)=γ​H​(Z)=γ​log⁡K,I^{\lx@glossaries@gls@link{main}{pi_{m}athrmeff}{{{}}{\hyperlink{glo:pi_mathrmeff}{\pi_{\mathrm{eff}}}}}}(Z;S_{+}\mid S_{0}=s)=\gamma\,H(Z)=\gamma\log K,
p⁡(s+∣s,z)=(1−γ)​δs+γ​δϕ⁡(z)p({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z)=(1-\gamma)\,\delta_{s}+\gamma\,\delta_{\phi(z)}
I⁡(Z;S+∣S0=s)≥supnγ​log⁡n=+∞.I(Z;S_{+}\mid S_{0}=s)\geq\sup_{n}\gamma\log n=+\infty.
I⁡(Z;S+∣S0=s)=γ​log⁡K,I(Z;S_{+}\mid S_{0}=s)=\gamma\log K,
supz,z′|𝔼p⁡(s+∣s,z)​[r⁡(S+)]−𝔼p⁡(s+∣s,z′)​[r⁡(S+)]|≤2​γ​ϵ≤2​ϵ.\sup_{z,z^{\prime}}\left|\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z)}[r(S_{+})]-\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z^{\prime})}[r(S_{+})]\right|\leq 2\gamma\epsilon\leq 2\epsilon.
|𝔼p⁡(s+∣s,z)​[r⁡(S+)]−𝔼p⁡(s+∣s,z′)​[r⁡(S+)]|\displaystyle\left|\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z)}[r(S_{+})]-\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}s_{+}}\mid s,z^{\prime})}[r(S_{+})]\right| =γ​|r⁡(sz)−r⁡(sz′)|\displaystyle=\gamma\,|r(s_{z})-r(s_{z^{\prime}})|
≤γ⁡(d⁡(sz,s¯)+d⁡(s¯,sz′))\displaystyle\leq\gamma\,\bigl(d(s_{z},\bar{s})+d(\bar{s},s_{z^{\prime}})\bigr)
≤2​γ​ϵ\displaystyle\leq 2\gamma\epsilon

Appendix H Anisotropic Reward Priors

Proposition H.1: changing the whitened reward covariance from identity to Σ makes adaptation the Gaussian width of the transformed offsets Σ½u_z. The previous lower bound is multiplied by √λ_min(Σ); positive-definite covariance retains a bound, while weakly rewarded directions weaken it.

Proposition H.2: in the K-target construction, choose a rank-one covariance with fixed trace T whose direction is orthogonal to every occupancy offset. The reward varies, but every skill receives the same return for a given reward draw. Thus expected adaptation is zero while MI=γ log K. The failure concerns misaligned reward directions, not merely insufficient total reward variance.

Equations 14
JΣ​(s0)≜𝔼g∼𝒩⁡(0,Σ)​[supz∈𝒵⟨g,uz⟩].J^{\Sigma}(s_{0})\triangleq\mathbb{E}_{g\sim\mathcal{N}(0,\Sigma)}\left[\sup_{z\in\mathcal{Z}}\bigl\langle g,\ u_{z}\bigr\rangle\right].
JΣ​(s0)≥λmin​(Σ)​I⁡(Z;S+∣S0=s0)c​2​π.J^{\Sigma}(s_{0})\;\geq\;\sqrt{\lambda_{\min}(\Sigma)}\;\frac{I(Z;S_{+}\mid S_{0}=s_{0})}{c\sqrt{2\pi}}.
JΣ​(s0)​≥step 1​12​π​(𝔼p⁡(z∣s0)​‖Σ1/2​uz‖2)1/2​≥step 2​λmin​(Σ)​I⁡(Z;S+∣S0=s0)c​2​π.J^{\Sigma}(s_{0})\ \overset{\text{step 1}}{\geq}\ \frac{1}{\sqrt{2\pi}}\left(\mathbb{E}_{p({\color[rgb]{0.5,0.5,0.5}z}\mid s_{0})}\left\|\Sigma^{1/2}u_{z}\right\|^{2}\right)^{1/2}\ \overset{\text{step 2}}{\geq}\ \sqrt{\lambda_{\min}(\Sigma)}\;\frac{I(Z;S_{+}\mid S_{0}=s_{0})}{c\sqrt{2\pi}}.\addcontentsline{lla}{section}{\numberline q:linear_{r}ate_{p}ivot_{a}nisotropic} (14)
JΣ​(s0)≥12​𝔼g​|⟨g,uz⟩|.J^{\Sigma}(s_{0})\ \geq\ \tfrac{1}{2}\,\mathbb{E}_{g}\left|\left\langle g,\ u_{z}\right\rangle\right|.
JΣ​(s0)≥12​π​‖Σ1/2​uz‖.J^{\Sigma}(s_{0})\ \geq\ \frac{1}{\sqrt{2\pi}}\left\|\Sigma^{1/2}u_{z}\right\|.
‖Σ1/2​uz‖2=⟨uz,Σ​uz⟩≥λmin​(Σ)​‖uz‖2.\|\Sigma^{1/2}u_{z}\|^{2}=\langle u_{z},\Sigma u_{z}\rangle\;\geq\;\lambda_{\min}(\Sigma)\,\|u_{z}\|^{2}.

Appendix I Tabular Experiment Details

All experiments use small tabular systems with known dynamics and illustrate the geometry. Candidate-vertex search approximates, rather than exhaustively enumerates, reachable skill occupancies.

I.1 Tabular 5 × 5 grid experiments

The basic example is a deterministic 5×5 grid with movement and stay actions, no wrapping boundaries, and γ=0.95. Table 1 preserves the solver and Q-learning settings.

Skill-learning via convex optimization.

For each starting state, sample 500 reward directions and solve a linear occupancy optimization subject to Bellman flow. Generic linear objectives expose vertices, producing valid candidate skills; the procedure need not discover every vertex.

With that finite channel fixed, optimize the skill source using Blahut–Arimoto. MI is concave in the source. The resulting capacity is a lower bound on true potential empowerment, improving as the candidate set expands.

Equation
maxπ⁡(a∣s,z;s0),p⁡(z∣s0)⁡I⁡(Z;S+∣S0=s0),\max_{\pi(a\mid s,z;s_{0}),\,p(z\mid s_{0})}I(Z;S_{+}\mid S_{0}=s_{0}),

Learning empowerment-maximizing policies.

Use the estimated capacity field as reward for Q-learning. Exploration decreases from 1 to 0.01 over 200,000 iterations, with step size 0.1. In the open room the learned policy moves toward the center, while skills launched there spread across the grid.

Equations
ℰpot​(s0)=maxπ⁡(a∣s,z;s0),p⁡(z∣s0)⁡I⁡(Z;S+∣S0=s0)\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{0})=\max_{\pi(a\mid s,z;s_{0}),\,p(z\mid s_{0})}I(Z;S_{+}\mid S_{0}=s_{0})
Qπpot(s,a)=𝔼[∑t=0∞γtℰpot(st)|s0=s,a0=a],Q^{\lx@glossaries@gls@link{main}{pi_{m}athrmpot}{{{}}{\hyperlink{glo:pi_mathrmpot}{\pi_{\mathrm{pot}}}}}}(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\lx@glossaries@gls@link{main}{ge_{m}athrmpot}{{{}}{\hyperlink{glo:ge_mathrmpot}{{\mathcal{E}}_{\mathrm{pot}}}}}(s_{t})\middle|s_{0}=s,a_{0}=a\right],
Table 1: Hyperparameters for the 5×5 gridworld experiments (Fig. 2)
Parameters Value
Environment size (n ×\times m) 5 ×\times 5
Environment actions {Left, Right, Up, Down, Stay}
Periodic boundary conditions No
P⁡(random action)P(\text{random action}) 0.0
Gamma for DSOM 0.95
# uniformly sampled vertex directions per s0s_{0} 500
Convex solver Clarabel [48]
Blahut–Arimoto max iterations 50_000
Blahut–Arimoto tolerance (capacity gap) 11e−6-6
Vertex seed 0
Epsilon schedule Linear interpolation from 1→0.011\rightarrow 0.01
Gamma for potential policy 0.95
# Q-learning iterations 2×1052\times 10^{5}
Q-learning step size α\alpha 0.1

I.2 Bottleneck grid experiments

Bottleneck experiments use an 8×8 grid with internal walls and the same occupancy/source optimization. Comparing γ=0.5 with γ=0.95 shows how longer-horizon control shifts high-empowerment regions toward openings. The effect depends on the map and horizon.

I.3 Key-finding experiments

In the 5×5 key task, access to part of the room depends on carrying a key available at (0,2). From (4,2), the potential policy collects it and moves toward (2,2). The state therefore includes access conditions as well as location.

This experiment uses γ=0.99 and 1,000 sampled vertex directions per start. Table 2 retains the full settings; Figure 11 shows both the navigation trajectory and subsequent skills.

Table 2: Hyperparameters for the 5×5 gridworld with walls and a key
Parameters Value
Environment size (n ×\times m) 5 ×\times 5
Environment actions {Left, Right, Up, Down, Stay, Pick up key at (0,2)(0,2), Drop key at (0,2)(0,2)}
Periodic boundary conditions No
P⁡(random action)P(\text{random action}) 0.0
Gamma for skill-learning 0.99
# uniformly sampled vertex directions per s0s_{0} 1000
Convex solver Clarabel [48]
Blahut–Arimoto max iterations 50_000
Blahut–Arimoto tolerance (capacity gap) 10−610^{-6}
Vertex seed 0
Epsilon schedule Linear interpolation from 1→0.011\rightarrow 0.01
Gamma for potential policy 0.99
# Q-learning iterations 2×1052\times 10^{5}
Q-learning step size α\alpha 0.1
Original figure
Figure 11: High empowerment finds central states, where skills are diverse from the central states (Theorem 4.1). Potential policy navigates to central states. Learned skills spread out from the central state and measure the degree of centrality of the state (in information geometry). The dotted line denotes the potential empowerment policy’s trajectory for 11−γ steps. Diamond denotes the handoff state s0, where the agent randomly samples z∼p⁡(z∣s0) and rolls out skill policy π⁡(a∣s,z;s0).

I.4 Adaptation geometry examples

Example X has two deterministic target-reaching skills. Example Y has three skills, each reaching its intended target with probability 23/30 and each other target with probability 7/60. Occupancies are normalized over targets, excluding initial-state mass for the visualization.

With a uniform baseline over targets, X has higher MI but Y has greater Gaussian width. The plotted axes span occupancy offsets, which also admit linear reward projections; colors indicate the best skill for each reward direction. Table 3 supplies all transition probabilities.

Table 3: Skill occupancy distributions of the example MDP s. (a) Example X with mass on the two reachable targets. (b) Example Y with mass on the three reachable targets.
(a) Figure 5(a) MDP
s1s_{1} s2s_{2}
z=1z=1 11 00
z=2z=2 00 11
(b) Figure 5(d) MDP
s1s_{1} s2s_{2} s3s_{3}
z=1z=1 23/3023/30 7/607/60 7/607/60
z=2z=2 7/607/60 23/3023/30 7/607/60
z=3z=3 7/607/60 7/607/60 23/3023/30

I.5 Worked example of centrality in information geometry

The worked example contrasts three distinct deterministic actions from one start with less distinguishable stochastic futures from another. Assigning the same random action distribution to all skills transmits no skill information; assigning different skills to distinct actions spreads their occupancies.

The start with the larger reachable information radius has greater potential empowerment. A potential policy seeks that start, while its effective skills exercise the available alternatives. Figure 12 makes the two roles explicit.

Refer to caption
Figure 12: Worked Example of Empowerment.
Concept demonstration

Open image ↗