Having options and exercising them are different objectives. Potential empowerment measures the maximum control available from a state; effective empowerment measures the information a particular skill policy actually transmits to its future state. The paper relates both to the geometry of reachable state distributions.
High potential empowerment identifies states offering a large information radius. In finite-state settings, achieved skill information lower-bounds average adaptation under a specified Gaussian reward prior. This is not a ranking guarantee: more empowerment can mean worse adaptation, and information can be arbitrarily large while useful reward variation vanishes in continuous spaces or under misaligned reward priors. Experiments are illustrative tabular examples.
1 Introduction
A hiker can either move to a peak that offers many routes or choose a route and descend. Potential-empowerment navigation corresponds to seeking the peak; effective-empowerment skills exercise its options [1][2]. Confusing these objectives obscures what skill-learning algorithms actually optimize.
Figure 1: Two trained empowerment-maximizing policies characterize central states in an MDP: Consider an agent (robot) on a hill that is easier to descend than to ascend.
In Fig. 1(a), the potential policy ascends the empowerment landscape to the top of the hill, using as an intrinsic reward.
In Fig. 1(b), The effective policy π
eff
maximizes , associating each skill with a downhill route.
Empowered States have Broad Access.
The paper formalizes earlier conjectures connecting empowerment, centrality, and broad access [3][5][6]. Centrality is measured through reachable discounted state distributions, rather than physical location alone. Separate results connect information to repeated-rollout hitting counts and, for an extended objective, temporal distances [7].
Information Geometries and Reward Geometries can Diverge in Continuous-State Markov decision process (MDP) s.
The positive adaptation result requires finite states and a particular reward distribution. Continuous rewards may barely distinguish states that remain perfectly distinguishable informationally; anisotropic priors can value directions that the skills do not control. Thus, information centrality does not generally imply reward adaptability.
2 Related Work
Classical empowerment is the capacity of the channel from an open-loop action sequence to a fixed-horizon future state [1][8]. The distinction between capacity and information realized by a specific policy predates this work [2]. Earlier research connected empowerment empirically to central or gateway states, while graph-based centrality and power-seeking analyses largely used discrete settings [5][6][12].
This paper instead studies reactive skills at geometrically sampled times. Information geometry gives a distributional notion of centrality; temporal-distance geometry extends reachability reasoning beyond discrete graphs [7]. Neither is ordinary Euclidean distance between physical states.
Estimating and Maximizing Empowerment in Practice.
MISL methods learn latent-conditioned policies that make future states informative about the latent [18][20][21][22]. Their direct objective resembles effective empowerment; learning diverse skills from one starting state does not itself learn to navigate toward states with maximal potential empowerment.
The paper supplies theoretical analysis and small tabular demonstrations, not a scalable continuous-control method. High-dimensional potential-empowerment maximization remains an open practical problem.
3 Preliminaries
The analysis represents policies by discounted distributions over visited states. This separates which future occupancies are feasible from which skill labels produce them.
The setting is a fully observed MDP with known transition structure for the illustrative experiments. A policy conditions on the current state, a skill latent, and the skill’s starting state. Fixing the latter two gives an ordinary policy. The theoretical skill source can be continuous even when states and actions are finite.
Reachable state-occupancy polytope.
A DSOM weights state visits by (1−γ)γᵗ. Equivalently, it is the distribution of the state observed at a geometric termination time, including time zero. Averaging over skills gives the skill-marginal occupancy.
For a fixed start state, achievable occupancies form a convex reachable set constrained by Bellman flow. In finite-state/action settings this set is a polytope inside the probability simplex. Its extent depends on controllability, stochastic dynamics, and discount horizon—not just the number of physically nearby states.
Equations 1–3
(1)
(2)
(3)
3.2 Empowerment Preliminaries
Potential empowerment optimizes over possible skill policies; effective empowerment evaluates one specified skill policy and source.
Potential and Effective Empowerment.
Definitions 3.1–3.3:potential empowerment is the maximum MI between a skill and its geometrically sampled future state, conditioned on the start. A potential policy uses this state-dependent capacity as intrinsic reward over its own trajectory. Effective empowerment is the MI achieved by a particular skill-conditioned policy [1][2].
These definitions replace classical fixed-horizon action sequences with reactive skills and a discounted horizon. Effective empowerment is an average KL divergence from each skill’s occupancy to the marginal occupancy; potential empowerment maximizes this quantity over policies and, equivalently, sources.
A fixed atomless continuous source loses no expressive generality when policies permit arbitrary measurable relabeling of skills. This does not hold for a fixed finite uniform skill alphabet, nor does it guarantee that a restricted neural policy can learn the needed relabeling. Appendices D–E establish the distinction.
Equations 4
(4)
Notation convention.
The paper writes π for an effective skill policy when unambiguous, π_eff when contrasting it with π_pot, and ℰ(s) for potential empowerment at a start state.
4 The Geometry of Empowerment
The results connect empowerment to an information radius, to repeated-rollout reachability, and to a limited downstream adaptation guarantee. Counterexamples then show where reward geometry departs from information geometry.
4.1 Information Geometry of Potential and Effective Empowerment Policies.
Figures 2–3 illustrate the two optimization problems: learn distinguishable skills from each candidate start, then navigate toward starts offering greater capacity.
Figure 2: Skills measure the centrality of a state. We visualize the potential empowerment field (a) for each state in a tabular GridWorld. The central state has high potential empowerment, because (b) the central state gives an agent access to a maximal set of distinguishable skills. A potential empowerment policy ascends the potential empowerment landscape to the center state in (c), from which it can execute different skills. See Theorem 4.1 for the formal statement. See AppendixI for tabular experiment details.
Geometric intuition.
Each skill corresponds to a point in the space of state distributions. Maximizing effective empowerment spreads these points over the reachable set. The capacity-achieving mixture is a central occupancy distribution; it is not generally a single environment state. Comparing the resulting KL radii ranks starting states by potential empowerment.
High Empowerment States are Central in Information Geometry.
Theorem 4.1:potential empowerment equals the smallest worst-case KL divergence from reachable occupancies to a center distribution. In the finite case, optimal active skills occupy extreme points on the boundary of this information ball, and their mixture is its center.
The general version assumes jointly measurable policies on standard Borel spaces. Finite capacity gives an attainable unique center; the boundary statement is conditional on attaining the optimum and holds almost everywhere under the optimal skill source. It should not be read as a claim about unused skill labels.
In the key-and-room example, the potential policy first obtains a key, then moves into the room. Holding the key changes reachable futures, so the preferred state differs from a location-only notion of the grid’s center.
Equations 5–6
(5)
(6)
(a)State occupancy simplex
(b)Potential policy finds central states.
Figure 3: Potentially empowered states are central states in the polytope. Tabular intuition for Theorem 4.1.
(a) We can use a probability simplex to visualize the discounted state occupancy measures of each skill (circles).
Orange circles indicate the best set of skills starting from state , and teal circles indicate the best skills starting from state .
from starting states and in arbitrary MDP.
The skills starting from define a smaller polytope (orange points) in the simplex than (teal points) as a result of their transition dynamics, so the potential empowerment is higher at than .
(b) We use a GridWorld to illustrate the connection between centrality and tool use.
An agent in the GridWorld begins at the bottom of the hallway.
The agent can set down and pick up a key at the top of the hallway.
The key provides the agent access to shaded regions of the MDP.
(b, i) A potentially empowered agent picks up the key then navigates to the center of the room, correctly identifying the middle state with the key as the most central state.
(b, ii) From the center, the agent can execute the maximal number of distinguishable skills.
See §.I.3 for experimental details.
4.2 Empowerment, Bottlenecks, and Temporal Distance
Lemma 4.2: in a discrete MDP, restart independent rollouts from the same start and stop each at a geometric time. Count how many rollouts are required before the terminal state equals a target. Effective empowerment equals the average reduction in log expected rollout count obtained by fixing a skill rather than sampling from the source. Maximizing over skill policies gives potential empowerment.
This is not primitive-step first-passage time or the event of visiting the target anywhere during a rollout. Appendix C separately introduces contiguous skill executions, with another discount factor; that different objective has an exact temporal-distance interpretation applicable to continuous settings.
Grid examples show empowerment concentrating near doorways as the horizon increases. They illustrate a bottleneck effect whose location depends on dynamics and discount, rather than proving that every geometrical bottleneck universally maximizes empowerment.
Equation
Figure 4: Bottleneck states are high potential empowerment states. Consider GridWorlds with a door, or a “bottleneck”. (a) For short horizon behaviors where , the high empowerment regions are the central door (a, left) and the center of the two sides of the room (a, right). (b) Lifting increases the horizon of possible behaviors, such that the empowerment tracks the location of the door. In §.4.2 we show that empowerment measures the reduction in temporal distance to future states, which quantifies a bottleneck effect. See §.I.2 for experimental details.
4.3 Is Empowerment Optimal for Downstream Adaptation?
Potential empowerment lower bounds skill adaptation under an uninformed reward prior.
Theorem 4.3: in a finite state space, choose a full-support baseline b and independent mean-zero Gaussian rewards with variance 1/b(s) at each state. A uniform baseline makes the prior isotropic. After seeing the reward, adaptation selects the best pretrained skill; it does not retrain the policy or directly optimize a new occupancy.
Let J be the reward-prior average of that best skill’s expected reward under its normalized discounted occupancy. For any skill set and source, J ≥ I/[c√(2π)], where I is achieved effective empowerment and c = supₓ≥₀ ln(1+x)/√x ≈ 0.8. Potential empowerment enters when the skill set achieves capacity.
J is the Gaussian width of baseline-scaled skill-occupancy offsets. The guarantee averages over the specified reward prior; it is neither a per-task guarantee nor proof that greater capacity gives better adaptation. Illustrative examples show the bound becoming tighter with stronger commitment and looser as the number of target states grows.
Equations
The First Limitation of Empowered Adaptation: Gaussian widths versus KL divergence.
First limitation: different geometries can rank starts differently. Two reliably reachable targets can give more MI than three noisier targets, yet the three-target system can have greater Gaussian width and expected adapted reward. Figure 5 supplies a finite-state counterexample even under the isotropic prior.
Theorem 4.4 / Proposition G.2: infinitely many distinct target states can fit in an arbitrarily small metric ball. Skills reaching separate targets can transmit γ log K information for K targets, while their expected rewards differ by at most 2γε for any bounded 1-Lipschitz reward when all targets lie within radius ε. Information can grow without useful reward differentiation.
The formal construction uses distinct absorbing targets, not the Gaussian distributions drawn for intuition in Figure 6. The shrinking plotted spatial width is also different from the Gaussian-width functional used in the adaptation bound.
Figure 5: Discrepancy between information geometry and reward adaptation geometry in discrete settings. (a) By going left/right from , an agent can reach node or with high likelihood, so there are two easily separable skills from .
(d) Targeting nodes , , or can make the agent inadvertently land in other nodes, so there are three somewhat separable skills from .
(b,e) The MDP s in (a) and (d) admit polytopes of feasible state occupancies in the simplex, visualized by the segment and triangle.
(b) Axis trades off occupancy of vs.
.
(e) Axis trades off occupying vs.
; axis trades off occupying vs.
.
Return is linear in the state occupancy, so we can also represent reward functions as vectors in state-occupancy space corresponding to the reward weights.
The color indicates the best adapted skill for that reward coordinate.
The polytope in (e) has a larger Gaussian Width than that in (b), supporting better adaptation.
(c,f) However, increased skill separability in (a) gives larger information radius and MI, where the black contour indicates the maximal MI.
Thus, even though empowerment is higher for the 2 state system (a) versus the 3 state system (d), the 2 state system has worse adapted returns.
See §.I.4 for details.Figure 6: The information ball can collapse in the continuous limit (PropositionG.2). Consider an arbitrary smooth reward function (level curves in gray) and some MDP that admits Gaussian skill state occupancy measures (colored distributions). While the potential empowerment of the starting state increases from subfigure (a) (c), the skills have increasingly identical returns and, thus, provide progressively smaller adaptation benefit. The plotted width is the maximum distance between skill means, not to be confused with Gaussian Width. This width bounds the differences in expected return for the smooth reward and decreases from left to right. Therefore, large channel capacities do not imply adaptation for continuous MDP s and reward functions. See PropositionG.2 for the theoretical result.
The Second Limitation of Empowered Adaptation: Controllable Degrees of Freedom versus Rewarding Degrees of Freedom.
Second limitation: controllable directions may be irrelevant to rewards. If the baseline-whitened Gaussian reward covariance is Σ, the lower bound gains a factor √λ_min(Σ). A positive minimum eigenvalue retains a weaker guarantee; a singular or nearly singular covariance can make it uninformative.
Appendix H constructs finite-state skills with γ log K information but zero adaptation when reward variation is orthogonal to every skill-occupancy difference, even at fixed total covariance trace. Thus, empowerment alone cannot guarantee useful control under arbitrary task preferences.
5 Discussion, Limitations, & Future Work
Potential empowerment measures access to distinguishable future distributions; effective skills realize that access. The geometric characterization clarifies centrality and gives a reward-prior-averaged adaptation lower bound in the tabular setting.
Limitations & Future Work.
The lower bound does not preserve rankings. Continuous smooth rewards and general anisotropic priors can destroy useful alignment with MI. The paper establishes these limits theoretically and tests only small, known-dynamics tabular examples.
Future methods must align information-based control with reward-relevant variation while estimating potential empowerment at scale. Modifying continuous-control or scalable skill-learning methods is a proposed direction, not a demonstrated solution [6][17][21][22][37][38].
Acknowledgments.
The authors acknowledge research feedback, support from Google and NSF graduate research funding, and Font Awesome icons adapted under CC BY 4.0.
The authors disclose generative-AI assistance with figures, the paper website, tabular experiment implementation, and minor readability edits.
Glossary
The paper’s main terms are DSOM, MI, KL divergence, MISL, and the potential/effective policy distinction. Definitions are available through the term links and the Terms tab. “Gaussian width” refers to expected maximal projection onto Gaussian reward directions, rather than an ordinary spatial diameter.
Appendix A Preliminaries for Proofs
The proofs use four standard facts [32][40][41][42][43]: MI is concave in the input source for a fixed channel and convex in the channel for a fixed source; capacity equals a minimax information radius; KL divergence is bounded by log(1+χ²); and an atomless reference random variable can be measurably transformed into a desired distribution on a standard Borel space.
These facts respectively justify source optimization, the occupancy-center characterization, the adaptation inequality, and replacing source learning with skill relabeling. The continuous-source assumption matters for the last step.
Equations
Appendix B Information Geometry of Empowerment
Assumption B.1: states, actions, and skills are standard Borel spaces, with policies jointly measurable in state and skill. This makes the skill-to-occupancy map measurable. Since a skill affects the sampled future only through its occupancy, replacing the latent by that occupancy preserves MI.
Theorem B.2: optimizing the induced channel yields the infimum over center distributions of the supremum KL divergence to reachable occupancies. Both sides may be infinite. With finite capacity the center is unique and attained; if a policy/source attains capacity, its marginal occupancy equals that center.
The proof uses the decomposition of average KL to any center into MI plus the KL of the marginal to that center. At an attained optimum, active skills have maximal radius almost surely; strict convexity excludes non-extreme active occupancies. In finite tabular settings the reachable set is a polytope, so these extreme points are vertices and capacity is at most log|S|.
Equations 7–9
(7)
(Golden formula)
(8)
(9)
B.1 Log reachability interpretation of potential empowerment in discrete deterministic settings
Lemma B.3: for the classical fixed-horizon, open-loop definition in a discrete deterministic MDP, empowerment is log of the number of states reachable in exactly H steps. Determinism makes conditional output entropy zero; choosing one action sequence per reachable endpoint with equal probability achieves the entropy maximum.
This is a special-case reachability formula, not a replacement for the paper’s stochastic discounted-skill objective.
Equations
Appendix C Empowerment and Bottleneck States
The appendix distinguishes independent restarted rollouts from a different process that resamples skills along one continuing trajectory.
C.1 Bottleneck states are high-empowerment states
A restarted rollout succeeds only when its geometrically sampled endpoint equals the target. Its rollout count is geometric, so expected counts are reciprocals of the marginal or skill-conditioned occupancy probabilities. Substituting these reciprocals into the MI log ratio proves Lemma 4.2 for any policy; optimizing gives potential empowerment.
Equations 10
(10)
C.2 Temporal distances over contiguous skill rollouts
For contiguous execution, run a skill until its geometric endpoint, then sample another skill from the current state and continue. γ discounts time within skills; a second factor β discounts the sequence of skill endpoints. The conditional occupancy fixes only the first skill and then follows the source for later skills.
Proposition C.1:MI between the initial skill and this longer-horizon future equals the expected reduction in temporal distance from committing to that first skill. The proof subtracts two negative log occupancy ratios whose shared self-reachability normalization cancels. Optimizing both policy and source defines potential contiguous empowerment.
In discrete spaces, temporal distance can be written as −log E[βᴴ] for appropriate endpoint-process hitting counts, rather than simply E[H]. The distributional formulation extends beyond discrete hitting probabilities under the temporal-distance setup.
Equations 11
(11)
Note on different empowerment expressions.
Contiguous empowerment is mathematically different from the main single-skill objective. Its channel also depends on the source used for later skills, so experiments optimize a parameterized source as well as candidate policies. Figure 7 shows similar bottleneck patterns, without establishing equality of the objectives.
Figure 7: Empowerment over contiguous rollouts concentrates at bottlenecks (PropositionC.1). The figures show the MI of the contiguous process at each start state on the grids from Fig.4, with (a) a center opening and (b) a corner opening. The bottleneck openings have high potential contiguous empowerment.
Appendix D Skill Selection: Fixed Continuous Uniform Priors are General
Lemma D.1: a fixed atomless continuous source can represent any discrete or continuous skill distribution on a standard Borel space through a measurable map. Composing the policy with that map preserves the joint behavior and MI, so optimizing over unrestricted measurable policies removes the need to optimize the source separately.
The skill is sampled once and held fixed during the rollout. This is an expressivity result: it does not assert that finite categorical priors, limited parameterizations, or practical training preserve capacity.
Equations
Appendix E Continuous versus Discrete Skill Spaces
Lemma E.1: a finite uniform skill prior can strictly lose capacity. In the two-skill construction, one action always reaches one target while the other reaches the alternative with probability β. For β=1/2, the optimal probability of the second skill is 2/5, rather than 1/2.
Convexity puts the optimal channel at an extreme action assignment; optimizing the source then shows a strict gain over the best uniform-prior channel. Thus, channel optimization alone does not fix the finite-uniform-source restriction.
VIC and DIAYN implementations use uniform categorical skills [18][20]. The finite counterexample identifies an expressivity restriction; it does not invalidate their empirical results. Atomless continuous priors avoid this restriction in the unrestricted policy class, and several scalable methods already use them [21][22].
Appendix F Empowerment Lower Bounds Tabular Adaptation
Subtract the marginal occupancy from each skill occupancy and divide coordinatewise by √b to obtain offsets u_z. Whiten rewards as g=√b·r. The common marginal return averages to zero, so expected best-skill return becomes the Gaussian width E_g sup_z〈g,u_z〉.
The 1/b reward-variance scaling preserves return distributions when states and their occupancy mass are proportionally subdivided. Figure 8 demonstrates this on two states split into four; unit-variance rewards would instead shrink return variation.
The proof first bounds Gaussian width below using the root-mean-square offset norm. It then bounds MI above by c times that norm, applying the KL–χ² comparison and Cauchy–Schwarz. Combining the inequalities yields Theorem 4.3. This is a finite-state result for the specified independent Gaussian prior.
Figures 9–10 vary control commitment and target count. For those illustrative constructions the bound tightens with stronger commitment but loosens as more distinct targets become available; they do not establish a general tightness theorem.
Equations 12
(12)
Figure 8: State refinement leaves return distribution unchanged (ExampleF.1). Bars display the occupancy of a skill over the coarse states (a) and the occupancy over the split states (b). Each panel lists the reference mass and per-state reward prior . Splitting states halves the reference mass and doubles the reward variance. Thus, the return obeys the same distribution in both MDP s.Equations 13–Cauchy-Schwarz
Figure 9: MI bound as a function of skill commitment (Theorem 4.3). Consider the MDP consisting of 5 states with a central starting state, where any left/right/up/down action sends probability mass to absorbing left/right/up/down states. Actions left/right/up/down have a level of commitment , which measures the additional mass above uniform assigned to the corresponding state. When commitment is small, probability is almost spread equally over all four states regardless of action so there is little control. In this regime, the adaptation dominates the lower bound. As the commitment increases, actions/skills commit to states and the bound tightens.Figure 10: MI bound as a function of absorbing states (Theorem 4.3). Consider an MDP with one starting state and absorbing target states. Each of the actions transitions deterministically to a distinct target. We normalize occupancies over the targets (neglecting the occupancy of the starting state). Thus, counts the maximum possible number of total skills. The scaled MI (orange) correctly lower bounds the adaptation (purple). The bound weakens as the number of absorbing target states in the MDP grows.
Appendix G When Information Geometry is Insufficient for Adaptation
Proposition G.1: construct K distinct absorbing targets, one per skill. Every discounted occupancy shares mass 1−γ at the start and places γ at its target. Observing a target identifies the skill, giving MI=γ log K; a continuously indexed set of distinct targets gives unbounded MI.
Proposition G.2: place all targets inside a radius-ε ball containing infinitely many states. Any bounded 1-Lipschitz reward varies by at most 2ε between targets, so normalized discounted skill returns differ by at most 2γε. Increasing the number of targets changes information without increasing this bound.
The shared start-state mass cancels in return differences. The result limits the benefit of choosing between skills; their absolute returns need not be near zero. Exact state distinguishability can therefore coexist with arbitrarily little adaptation advantage, preventing an unrestricted extension of the tabular Gaussian-prior guarantee to smooth continuous rewards.
Equations
Appendix H Anisotropic Reward Priors
Proposition H.1: changing the whitened reward covariance from identity to Σ makes adaptation the Gaussian width of the transformed offsets Σ½u_z. The previous lower bound is multiplied by √λ_min(Σ); positive-definite covariance retains a bound, while weakly rewarded directions weaken it.
Proposition H.2: in the K-target construction, choose a rank-one covariance with fixed trace T whose direction is orthogonal to every occupancy offset. The reward varies, but every skill receives the same return for a given reward draw. Thus expected adaptation is zero while MI=γ log K. The failure concerns misaligned reward directions, not merely insufficient total reward variance.
Equations 14
(14)
Appendix I Tabular Experiment Details
All experiments use small tabular systems with known dynamics and illustrate the geometry. Candidate-vertex search approximates, rather than exhaustively enumerates, reachable skill occupancies.
I.1 Tabular 5 × 5 grid experiments
The basic example is a deterministic 5×5 grid with movement and stay actions, no wrapping boundaries, and γ=0.95. Table 1 preserves the solver and Q-learning settings.
Skill-learning via convex optimization.
For each starting state, sample 500 reward directions and solve a linear occupancy optimization subject to Bellman flow. Generic linear objectives expose vertices, producing valid candidate skills; the procedure need not discover every vertex.
With that finite channel fixed, optimize the skill source using Blahut–Arimoto. MI is concave in the source. The resulting capacity is a lower bound on true potential empowerment, improving as the candidate set expands.
Equation
Learning empowerment-maximizing policies.
Use the estimated capacity field as reward for Q-learning. Exploration decreases from 1 to 0.01 over 200,000 iterations, with step size 0.1. In the open room the learned policy moves toward the center, while skills launched there spread across the grid.
Equations
Table 1: Hyperparameters for the gridworld experiments (Fig.2)
Bottleneck experiments use an 8×8 grid with internal walls and the same occupancy/source optimization. Comparing γ=0.5 with γ=0.95 shows how longer-horizon control shifts high-empowerment regions toward openings. The effect depends on the map and horizon.
I.3 Key-finding experiments
In the 5×5 key task, access to part of the room depends on carrying a key available at (0,2). From (4,2), the potential policy collects it and moves toward (2,2). The state therefore includes access conditions as well as location.
This experiment uses γ=0.99 and 1,000 sampled vertex directions per start. Table 2 retains the full settings; Figure 11 shows both the navigation trajectory and subsequent skills.
Table 2: Hyperparameters for the gridworld with walls and a key
Parameters
Value
Environment size (n m)
5 5
Environment actions
{Left, Right, Up, Down, Stay, Pick up key at , Drop key at }
Figure 11: High empowerment finds central states, where skills are diverse from the central states (Theorem 4.1). Potential policy navigates to central states. Learned skills spread out from the central state and measure the degree of centrality of the state (in information geometry). The dotted line denotes the potential empowerment policy’s trajectory for steps. Diamond denotes the handoff state , where the agent randomly samples and rolls out skill policy .
I.4 Adaptation geometry examples
Example X has two deterministic target-reaching skills. Example Y has three skills, each reaching its intended target with probability 23/30 and each other target with probability 7/60. Occupancies are normalized over targets, excluding initial-state mass for the visualization.
With a uniform baseline over targets, X has higher MI but Y has greater Gaussian width. The plotted axes span occupancy offsets, which also admit linear reward projections; colors indicate the best skill for each reward direction. Table 3 supplies all transition probabilities.
Table 3: Skill occupancy distributions of the example MDP s. (a) Example X with mass on the two reachable targets. (b) Example Y with mass on the three reachable targets.
I.5 Worked example of centrality in information geometry
The worked example contrasts three distinct deterministic actions from one start with less distinguishable stochastic futures from another. Assigning the same random action distribution to all skills transmits no skill information; assigning different skills to distinct actions spreads their occupancies.
The start with the larger reachable information radius has greater potential empowerment. A potential policy seeks that start, while its effective skills exercise the available alternatives. Figure 12 makes the two roles explicit.
[3]
Christoph Salge, Cornelius Glackin, and Daniel Polani.
Empowerment–an Introduction.
Guided Self-Organization: Inception, pp. 67–114. Springer, 2014.
[13]
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman.
Design Principles of the Hippocampal Cognitive Map.
Neural Information Processing Systems, 27, 2014.
[18]
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra.
Variational Intrinsic Control.
International Conference on Learning Representations, 2017.
[19]
Thomas J Ringstrom.
Reward is not necessary: how to create a modular & compositional self-preserving agent for life-long learning.
arXiv preprint arXiv:2211.10851, 2022.
[24]
Evan Ellis, Vivek Myers, Jens Tuyls, Sergey Levine, Anca Dragan, and Benjamin Eysenbach.
Training LLM Agents to Empower Humans.
International Conference on Machine Learning, 2026.
[26]
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver.
Successor Features for Transfer in Reinforcement Learning.
Neural Information Processing Systems, 30, 2017.
[31]
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv.
All Else Being Equal Be Empowered.
European Conference on Artificial Life, pp. 744–753, 2005b.
[34]
Özgür Şimşek, Alicia P Wolfe, and Andrew G Barto.
Identifying Useful Subgoals in Reinforcement Learning by Local Graph Partitioning.
International Conference on Machine Learning, pp. 816–823, 2005.
[36]
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos.
Transfer in deep reinforcement learning using successor features and generalised policy improvement.
International conference on machine learning, pp. 501–510, 2018.
[42]
Benjamin Bloem-Reddy and Yee Whye Teh.
Probabilistic Symmetries and Invariant Neural Networks.
Journal of Machine Learning Research, 21(90):1–61, 2020.
[44]
Christoph Salge, Cornelius Glackin, and Daniel Polani.
Approximation of Empowerment in the Continuous Domain.
Complex Systems, 16(02n03):1250079, 2013.
[46]
Suguru Arimoto.
An Algorithm for Computing the Capacity of Arbitrary Discrete Memoryless Channels.
IEEE Transactions on Information Theory, 18(1):14–20, 1972.
Creative Commons Attribution: the license credited for adapted Font Awesome icons in the paper.
DIAYN
Diversity Is All You Need: an unsupervised skill-learning method using a uniform categorical skill prior.
DSOM
Discounted state occupancy measure: the normalized distribution of states visited under a policy, weighted by (1−γ)γᵗ; equivalently, the state distribution at a geometric stopping time.
effective empowerment
The mutual information actually achieved by a specified skill-conditioned policy and skill source at a given starting state.
Gaussian width
The expected largest inner product between a set of vectors and a standard Gaussian vector. Here it measures average best-skill return after baseline scaling of occupancy differences.
ICLR
International Conference on Learning Representations.
IEEE
Institute of Electrical and Electronics Engineers.
KL
Kullback–Leibler divergence: an asymmetric information-theoretic discrepancy between probability distributions; it is not a Euclidean distance or generally a metric.
MDP
Markov decision process: a sequential decision model with states, actions, and transition probabilities. This paper assumes full observation.
MI
Mutual information: information shared by two random variables; here it measures how well a future state identifies the executed skill.
MISL
Mutual information skill learning: learning latent-conditioned policies whose future states distinguish their skill labels.
NSF
United States National Science Foundation, acknowledged as a research funder.
potential empowerment
The maximum mutual information achievable between skills and a future state from a given start. A potential policy separately uses this capacity as a state reward.
VIC
Variational Intrinsic Control: a skill-learning method based on maximizing information between options and their outcomes.