Literature
Display
← All papers

HP-JEPA: Hierarchical Partitioning for Multi-Resolution Graph Joint-Embedding Predictive Learning

Ruichen Xu · Jingxiang Qu · Wenhan Gao · Jiaxing Zhang · Linsey Pang · Ravid Shwartz-Ziv · Yann LeCun · Yuefan Deng

Preprint posted · arXiv (preprint) ↗

Graph learning5 min text · CondensedOriginal paper ↗
Contents

At a glance

HP-JEPA represents each graph at several partition resolutions and performs latent prediction independently within each resolution. Frozen resolution-specific graph features are then concatenated or combined with task-specific weights. The tested model improves over fixed-resolution Graph-JEPA on six of eight tasks. Its hierarchy is a bank of coarse-to-fine granularities, not an explicitly nested parent–child graph or a cross-resolution prediction loss.

1. Introduction

A single patch scale can hide small motifs in a large graph or fragment an already small graph too aggressively. The paper therefore retains several resolutions. The key questions are how to define valid tokens for different graph sizes, what latent quantity is predicted, and how downstream tasks combine the resulting representations.

Refer to caption
Figure 1. Motivation and overview of HP-JEPA. Real-world graphs contain complementary structural information at multiple resolutions, including fine-resolution local motifs, intermediate-resolution regional context, and coarse-resolution global topology. A single fixed resolution can be overly fine, fragmenting global context, or overly coarse, obscuring local structure. HP-JEPA constructs graph-region tokens at multiple resolutions and integrates their resolution-specific representations. Task-specific resolution weighting allows downstream models to emphasize the most informative resolutions.

2. Related Work

Related work separates graph contrastive learning, reconstruction and latent prediction. The comparison table summarizes mechanisms, but architectural categories do not substitute for matched benchmark evaluation. HP-JEPA’s direct baseline is a fixed-resolution graph JEPA.

Table 1. Design-level comparison of graph representation learning methods. The check marks (✓) and cross marks (✗) indicate that the property is explicitly satisfied/unsatisfied by design, and a triangle (△\triangle) indicates a partial match.
Method Main training signal Label-Free No Neg. No Raw Recon. Latent Pred. Region Tokens Multi-Resolution Tokens
F-GIN (Xu et al., 2019) Supervised labels ✗ ✓ ✓ ✗ ✗ ✗
InfoGraph (Sun et al., 2020) Local–global mutual information ✓ ✗ ✓ ✗ △\triangle ✗
GraphCL (You et al., 2020) Augmented-view contrast ✓ ✗ ✓ ✗ ✗ ✗
MVGRL (Hassani and Khasahmadi, 2020) Multi-view contrast ✓ ✗ ✓ ✗ ✗ ✗
AD-GCL-FIX/OPT (Suresh et al., 2021) Adversarial graph contrast ✓ ✗ ✓ ✗ ✗ ✗
GraphMAE (Hou et al., 2022) Masked feature reconstruction ✓ ✓ ✗ ✗ ✗ ✗
S2GAE (Tan et al., 2023) Masked structure reconstruction ✓ ✓ ✗ ✗ ✗ ✗
BGRL (Thakoor et al., 2022) Bootstrap latent prediction ✓ ✓ ✓ ✓ ✗ ✗
LaGraph (Xie et al., 2022) Latent graph prediction ✓ ✓ ✓ ✓ △\triangle ✗
Graph-JEPA (Skenderi et al., 2025) Single-resolution patch prediction ✓ ✓ ✓ ✓ ✓ ✗
HP-JEPA Multi-resolution patch prediction ✓ ✓ ✓ ✓ ✓ ✓

Notes. (1) “No Neg.” means the objective does not require explicit negative pairs. (2) “No Raw Recon.” means the method does not reconstruct raw node attributes, edges, or graph structure. (3) “Latent Pred.” means the target is a representation-space target. (4) “Region Tokens” means graph regions, subgraphs, or patches are explicit learning units. (5) “Multi-Resolution Tokens” means that graph-region tokens are explicitly constructed at multiple partition resolutions. (6) For F-GIN, check marks under “No Neg.” and “No Raw Recon.” only indicate that these mechanisms are absent; F-GIN is still supervised. AD-GCL-FIX and AD-GCL-OPT are merged because they share the same method design.

3. Preliminaries

The preliminaries define graph-level representation learning and the context–target prediction problem. The distinction between pretraining without task labels and supervised downstream readout remains important.

3.1. Graph Representation Learning

A graph encoder maps node attributes and connectivity into a fixed-length representation. Downstream classification or regression trains a readout on those features. Multi-resolution tokenization changes what structural information reaches that encoder; it does not remove the need for labeled evaluation tasks.

HP-JEPA independently constructs graph regions at multiple resolutions, performs two-coordinate latent prediction separately within each eligible resolution, mean-pools valid token representations to obtain resolution-specific graph representations, and combines those representations through concatenation or task-specific resolution weighting.
Figure 2. Overview of HP-JEPA. (a) The input graph is partitioned independently at LL configured resolutions. At resolution ℓ\ell, each core region Pi(ℓ)P_{i}^{(\ell)} defines a graph-patch support P~i(ℓ)\widetilde{P}_{i}^{(\ell)}, while padded and inactive token slots are masked and ignored. (b) At each eligible resolution, the sampled context index cℓ​(G)c_{\ell}(G) and target-index set 𝒯ℓ​(G)\mathcal{T}_{\ell}(G) determine the context and target-content inputs processed by the online token encoder fθf_{\theta} and EMA target token encoder fθ¯f_{\bar{\theta}}, respectively. For each t∈𝒯ℓ​(G)t\in\mathcal{T}_{\ell}(G), the predictor qψq_{\psi} estimates y^t(ℓ)\widehat{y}_{t}^{(\ell)} from the context state scℓ(ℓ)s_{c_{\ell}}^{(\ell)} and target structural query pt(ℓ)p_{t}^{(\ell)}, supervised by the two-coordinate target yt,tgt(ℓ)y_{t,\mathrm{tgt}}^{(\ell)} using a coordinate-wise Smooth L1L_{1} loss. (c) During downstream readout, the frozen EMA target token encoder processes each active resolution independently, and its valid token representations rG,i(ℓ)r_{G,i}^{(\ell)} are mean-pooled to obtain hG(ℓ)h_{G}^{(\ell)}. The resolution-specific representations are subsequently combined through concatenation hGconcath_{G}^{\mathrm{concat}} or uniformly smoothed task-specific resolution weighting hGtaskh_{G}^{\mathrm{task}} for graph classification or regression.HP-JEPA independently constructs graph regions at multiple resolutions, performs two-coordinate latent prediction separately within each eligible resolution, mean-pools valid token representations to obtain resolution-specific graph representations, and combines those representations through concatenation or task-specific resolution weighting.

3.2. JEPA for Graphs

A context patch and a structural query predict a transformed statistic of a target patch’s representation. The source’s two-coordinate target retains the feature mean through a Lorentz-coordinate map, not the complete latent vector. The appendix characterizes exactly that retained projection. It should not be described as a lossless embedding of every target-feature direction.

4. Method

The method combines independent multi-resolution partitioning, a separate prediction update at each eligible resolution and frozen readout. These three choices are distinct parts of the pipeline.

4.1. Multi-Resolution Graph Tokenization

Each active resolution partitions the full graph independently. Tokens describe patch content and structural context; padding slots are masked, and overly fine resolutions become inactive for small graphs. There is no required nesting between resolutions. Expanded patch supports can overlap even when their core regions form a partition.

4.2. Resolution-Wise Joint-Embedding Prediction

Prediction uses context content plus a target structural query, while the EMA target branch receives detached target content without that query. The loss matches two coordinates of a feature-mean statistic with SmoothL1. Graphs with more selected targets contribute more within the token-normalized batch loss. Resolutions are processed with sequential optimizer updates; no parent–child or cross-resolution consistency term is added.

4.3. Multi-Resolution Readout and Task-Specific Resolution Weighting

After pretraining, each resolution is encoded and pooled independently. Concatenation preserves the separate feature slots; a learned mixture instead uses weights shared across a downstream task and split. That mixture uses downstream labels and is not graph-specific attention. Inactive slots contribute zero and weights are not renormalized per graph. The appendix’s oracle comparison concerns a bounded fixed-feature readout family, not nonlinear pretraining convergence.

Table 2. Results on graph classification and regression benchmarks. For reference only, we include the results of F-GIN, a pioneering end-to-end supervised GNN. SSL methods are grouped by pretraining paradigm in the following order: contrastive, generative, self-predictive, and JEPA. Baseline results are taken directly from prior work (Hassani and Khasahmadi, 2020; Tan et al., 2023; Suresh et al., 2021; Skenderi et al., 2025) and "-" indicates that the corresponding results are not reported. The best SSL result is underlined, and the second best is wavy underlined. For a more direct comparison, HP-JEPA results that outperform the Graph-JEPA baseline are highlighted with a red cell background.
Model PROTEINS ↑\uparrow MUTAG ↑\uparrow DD ↑\uparrow REDDIT-B ↑\uparrow REDDIT-M5 ↑\uparrow IMDB-B ↑\uparrow IMDB-M ↑\uparrow ZINC-12K ↓\downarrow
F-GIN (Supervised) 72.39±2.7672.39\pm 2.76 90.41±4.6190.41\pm 4.61 74.87±3.5674.87\pm 3.56 86.79±2.0486.79\pm 2.04 53.28±3.1753.28\pm 3.17 71.83±1.9371.83\pm 1.93 48.46±2.3148.46\pm 2.31 0.254±0.0050.254\pm 0.005
InfoGraph 72.57±0.6572.57\pm 0.65 87.71±1.7787.71\pm 1.77 75.23±0.3975.23\pm 0.39 78.79±2.1478.79\pm 2.14 51.11±0.5551.11\pm 0.55 71.11±0.8871.11\pm 0.88 48.66±0.6748.66\pm 0.67 0.890±0.0170.890\pm 0.017
GraphCL 72.86±1.0172.86\pm 1.01 88.29±1.3188.29\pm 1.31 74.70±0.7074.70\pm 0.70 82.63±0.9982.63\pm 0.99 53.05±0.4053.05\pm 0.40 70.80±0.7770.80\pm 0.77 48.49±0.6348.49\pm 0.63 0.627±0.0130.627\pm 0.013
MVGRL – – – 84.5±0.684.5\pm 0.6 – 74.2±0.774.2\pm 0.7 51.2±0.551.2\pm 0.5 –
AD-GCL-FIX 73.59±0.6573.59\pm 0.65 89.25±1.4589.25\pm 1.45 74.49±0.5274.49\pm 0.52 85.52±0.7985.52\pm 0.79 53.00±0.8253.00\pm 0.82 71.57±1.0171.57\pm 1.01 49.04±0.5349.04\pm 0.53 0.578±0.0120.578\pm 0.012
AD-GCL-OPT 73.81±0.4673.81\pm 0.46 89.70±1.0389.70\pm 1.03 75.10±0.3975.10\pm 0.39 85.52±0.7985.52\pm 0.79 54.93±0.4354.93\pm 0.43 72.33±0.5672.33\pm 0.56 49.89±0.6649.89\pm 0.66 0.544±0.0040.544\pm 0.004
GraphMAE 75.30±0.3975.30\pm 0.39 88.19±1.2688.19\pm 1.26 74.27±1.0774.27\pm 1.07 88.01±0.1988.01\pm 0.19 46.06±3.4446.06\pm 3.44 75.52±0.6675.52\pm 0.66 51.63±0.5251.63\pm 0.52 0.935±0.0340.935\pm 0.034
S2GAE 76.37±0.4376.37\pm 0.43 88.26±0.7688.26\pm 0.76 – 87.83±0.2787.83\pm 0.27 – 75.76±0.6275.76\pm 0.62 51.79±0.3651.79\pm 0.36 –
BGRL 70.99±3.8670.99\pm 3.86 74.99±8.8374.99\pm 8.83 71.52±2.9771.52\pm 2.97 50±050\pm 0 20±0.120\pm 0.1 0.5±00.5\pm 0 0.33±00.33\pm 0 1.2±0.0111.2\pm 0.011
LaGraph 75.2±0.475.2\pm 0.4 90.2±1.190.2\pm 1.1 78.1±0.478.1\pm 0.4 90.4±0.890.4\pm 0.8 56.4±0.456.4\pm 0.4 73.7±0.973.7\pm 0.9 – –
Graph-JEPA 75.67±3.7875.67\pm 3.78 91.25±5.7591.25\pm 5.75 78.64±2.3578.64\pm 2.35 91.99±1.5991.99\pm 1.59 56.73±1.9656.73\pm 1.96 73.68±3.2473.68\pm 3.24 50.69±2.9150.69\pm 2.91 0.434±0.0140.434\pm 0.014
HP-JEPA 76.97±0.5076.97\pm 0.50 91.70±5.3191.70\pm 5.31 79.20±0.3879.20\pm 0.38 92.19±1.5692.19\pm 1.56 57.58±1.6357.58\pm 1.63 74.08±0.4074.08\pm 0.40 50.51±1.8150.51\pm 1.81 0.472±0.0100.472\pm 0.010

5. Experiments

Experiments cover graph classification, molecular regression, a representation case study and sensitivity to graph size and resolution banks. Each probes a different part of the multi-resolution motivation.

5.1. Downstream Task Evaluation

The task evaluation separates the frozen representation from the supervised classifier or regressor trained on it. Data splitting and tuning determine the scope of a fair comparison.

5.1.1. Setup

Seven classification benchmarks use repeated stratified cross-validation; ZINC-12K uses its standard regression split. Baselines include both latent-prediction and generative graph methods. The appendix specifies which preprocessing and fitting steps use only the corresponding training data.

5.1.2. Results and Discussion

HP-JEPA improves over Graph-JEPA on six of eight tasks and performs best on five in the reported comparison. The two IMDB tasks still favor generative methods, consistent with their small graphs and limited attributes. The empirical result is therefore a useful extension, not universal dominance of latent prediction over reconstruction.

Refer to caption
Figure 3. 3D t-SNE visualization of graph-level embeddings on the PROTEINS dataset. Each point represents one protein graph. Columns compare Graph-JEPA and HP-JEPA; the top row colors embeddings by graph-size quartile (Q1–Q4), while the bottom row colors the same embeddings by protein class (C0/C1). Compared with fixed-resolution Graph-JEPA, HP-JEPA produces a more structured representation space, distinctly separating proteins. This alignment indicates that HP-JEPA captures graph-size-dependent structural and functional patterns without using class labels during pretraining.

5.2. Case Study: Learning Multi-Resolution Graph Features

The PROTEINS visualization relates embeddings to labels and graph size. It illustrates structural variation preserved at multiple scales. Size–label association in a benchmark is not evidence that graph size alone supplies a universal biological explanation of enzyme function.

5.3. Sensitivity Analysis

Most size quartiles and all tested resolution-bank settings favor HP-JEPA over the fixed-resolution baseline. The results reduce dependence on choosing one granularity in those datasets, while leaving performance variation across banks visible. They do not establish invariance to every graph-size distribution.

Refer to caption
Figure 4. Sensitivity analysis on IMDB-BINARY, IMDB-MULTI, and PROTEINS. Panels (a), (c), and (e) report Graph-JEPA and HP-JEPA accuracy across node-count quartiles Q​1Q1–Q​4Q4. Panels (b), (d), and (f) compare HP-JEPA using resolution banks {2,4,8}\{2,4,8\}, {2,4,8,16}\{2,4,8,16\}, and the full bank, with Graph-JEPA shown as a dashed reference. Results are the mean and standard deviation over ten cross-validation folds from one pretrained seed.

6. Conclusion

The contribution is multi-resolution graph representation learning with independent within-resolution prediction and flexible downstream combination. Its gains are benchmark-bounded, and the target geometry preserves a specific statistic rather than the entire latent state.

Appendix A Dataset and Training Details

The appendix documents graph statistics, input features, partitions, splits and task-specific configurations. These details determine what information is available in pretraining and how downstream evaluation avoids leakage.

Table 3. Descriptive statistics of the datasets used in the main experiments. The statistics are adapted from Graph-JEPA (Skenderi et al., 2025). The symbol “–” indicates that ZINC-12K is a graph-regression benchmark and therefore has no discrete class labels.
Dataset Num. Graphs Num. Classes Avg. Nodes Avg. Edges
PROTEINS 1,113 2 39.06 72.82
MUTAG 188 2 17.93 19.79
DD 1,178 2 284.32 715.66
REDDIT-BINARY 2,000 2 429.63 497.75
REDDIT-MULTI-5K 4,999 5 508.52 594.87
IMDB-BINARY 1,000 2 19.77 96.53
IMDB-MULTI 1,500 3 13.00 65.94
ZINC-12K 12,000 – 23.20 49.80

A.1. Pretraining Configuration

The configuration fixes capacities, target sampling, optimizer/teacher schedules and downstream choices. The source explicitly distinguishes training-only fitting from held-out evaluation data. Multi-resolution selection and readout tuning must follow those split boundaries.

Table 4. Dataset-specific HP-JEPA pretraining configurations. KℓK_{\ell} lists the configured region capacities across resolutions, MℓM_{\ell} lists the corresponding numbers of selected target regions, dRWd_{\mathrm{RW}} is the random-walk positional-encoding dimension, Drop is module dropout, Tok. is token dropout, and Clip is the maximum gradient norm. All configurations use latent dimension 512 and four Transformer blocks.
Dataset KℓK_{\ell} MℓM_{\ell} GNN Layers Heads dRWd_{\mathrm{RW}} LR WD BS Epochs Scheduler Drop. Tok. Clip
PROTEINS 2,4,8,162,4,8,16 1,2,3,41,2,3,4 2 8 15 2.5×10−42.5{\times}10^{-4} 1×10−51{\times}10^{-5} 128 20 cosine 0 0.10 2.0
MUTAG 2,4,8,16,322,4,8,16,32 1,2,3,4,41,2,3,4,4 2 4 15 1.5×10−41.5{\times}10^{-4} 0 64 30 cosine 0 0.05 1.0
DD 2,4,8,16,322,4,8,16,32 1,2,3,4,41,2,3,4,4 3 8 30 2×10−42{\times}10^{-4} 0 32 30 cosine 0 0.05 1.0
REDDIT-BINARY 2,4,8,16,32,64,1282,4,8,16,32,64,128 1,2,3,4,4,4,41,2,3,4,4,4,4 2 8 40 2×10−52{\times}10^{-5} 0 32 40 cosine 0 0 0.5
REDDIT-MULTI-5K 2,4,8,16,32,64,1282,4,8,16,32,64,128 1,2,3,4,4,4,41,2,3,4,4,4,4 2 8 40 2×10−52{\times}10^{-5} 0 32 40 cosine 0 0 0.5
IMDB-BINARY 2,4,82,4,8 1,2,31,2,3 2 8 15 1×10−51{\times}10^{-5} 1×10−51{\times}10^{-5} 16 12 constant 0.05 0 2.0
IMDB-MULTI 2,4,8,16,322,4,8,16,32 1,2,3,4,41,2,3,4,4 2 8 15 1×10−51{\times}10^{-5} 1×10−51{\times}10^{-5} 16 12 cosine 0 0 1.0
ZINC-12K 2,4,8,16,322,4,8,16,32 1,2,3,4,41,2,3,4,4 2 8 20 2×10−52{\times}10^{-5} 0 32 40 cosine 0 0 0.5
Table 5. Dataset-specific downstream readout and linear-probe settings. The readout column reports the mixture of uniform and learned resolution weights, and αprobe\alpha_{\mathrm{probe}} denotes the recorded regularization parameter of the downstream linear probe.
Dataset Resolution Readout αprobe\alpha_{\mathrm{probe}} Evaluation
PROTEINS 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 5×105\times 10-fold accuracy
MUTAG Uniform mean 10−110^{-1} 5×105\times 10-fold accuracy
DD 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 5×105\times 10-fold accuracy
REDDIT-BINARY 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 5×105\times 10-fold accuracy
REDDIT-MULTI-5K 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 5×105\times 10-fold accuracy
IMDB-BINARY 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 5×105\times 10-fold accuracy
IMDB-MULTI Uniform mean 10−310^{-3} 5×105\times 10-fold accuracy
ZINC-12K 0.750.75 uniform + 0.25+\;0.25 learned 10−210^{-2} 10-seed MAE

Appendix B Geometry and Population Characterization of the Latent Target

The geometry result proves that the two-coordinate map preserves distances in the feature-mean projection and is insensitive to components whose mean is zero. It is therefore not injective on the full latent vector. The SmoothL1 population optimum is a conditional Huber location of the target coordinates, not necessarily their conditional mean. The unconstrained predictor is also not guaranteed to output a point on the Lorentz hyperboloid.

Equations 1–4
(1) m⁡(u):=1d​𝟏d⊤​u,Φ⁡(u):=[cosh⁡(m⁡(u))sinh⁡(m⁡(u))],m(u):=\frac{1}{d}\mathbf{1}_{d}^{\top}u,\qquad\Phi(u):=\begin{bmatrix}\cosh(m(u))\\ \sinh(m(u))\end{bmatrix},
(2) JL:=[−1001]J_{\mathrm{L}}:=\begin{bmatrix}-1&0\\ 0&1\end{bmatrix}
(3) ℍ+1:={y∈ℝ2:y⊤JLy=−1,y1>0}.\mathbb{H}_{+}^{1}:=\left\{y\in\mathbb{R}^{2}:y^{\top}J_{\mathrm{L}}y=-1,\;y_{1}>0\right\}.
(4) dℍ​(y,y′):=arcosh⁡(−y⊤​JL​y′).d_{\mathbb{H}}(y,y^{\prime}):=\operatorname{arcosh}\left(-y^{\top}J_{\mathrm{L}}y^{\prime}\right).
Proposition 0 (Geometry and information content of the latent-coordinate map).

For every u,v∈ℝdu,v\in\mathbb{R}^{d}, the coordinate map in Equation (1) satisfies

Equation 5
(5) Φ⁡(u)∈ℍ+1.\Phi(u)\in\mathbb{H}_{+}^{1}.

Moreover,

Equation 6
(6) dℍ​(Φ⁡(u),Φ⁡(v))=|m⁡(u)−m⁡(v)|=1d​|𝟏d⊤​(u−v)|.d_{\mathbb{H}}\left(\Phi(u),\Phi(v)\right)=|m(u)-m(v)|=\frac{1}{d}\left|\mathbf{1}_{d}^{\top}(u-v)\right|.

Consequently,

Equation 7
(7) dℍ​(Φ⁡(u),Φ⁡(v))≤1d​‖u−v‖2.d_{\mathbb{H}}\left(\Phi(u),\Phi(v)\right)\leq\frac{1}{\sqrt{d}}\|u-v\|_{2}.

Finally,

Equation 8
(8) Φ(u)=Φ(v)⟺u−v∈𝟏d⟂,\Phi(u)=\Phi(v)\quad\Longleftrightarrow\quad u-v\in\mathbf{1}_{d}^{\perp},

where

Equation 9
(9) 𝟏d⟂:={a∈ℝd:𝟏d⊤​a=0}.\mathbf{1}_{d}^{\perp}:=\left\{a\in\mathbb{R}^{d}:\mathbf{1}_{d}^{\top}a=0\right\}.

Therefore, Φ\Phi preserves exactly the feature-mean projection of the target-token state and is invariant to all latent components orthogonal to 𝟏d\mathbf{1}_{d}.

Equations 10–21
(10) a:=m⁡(u),b:=m⁡(v).a:=m(u),\qquad b:=m(v).
Φ​(u)⊤​JL​Φ​(u)\displaystyle\Phi(u)^{\top}J_{\mathrm{L}}\Phi(u) =−cosh2⁡(a)+sinh2⁡(a)\displaystyle=-\cosh^{2}(a)+\sinh^{2}(a)
(11) =−1.\displaystyle=-1.
−Φ​(u)⊤​JL​Φ​(v)\displaystyle-\Phi(u)^{\top}J_{\mathrm{L}}\Phi(v) =cosh⁡(a)​cosh⁡(b)−sinh⁡(a)​sinh⁡(b)\displaystyle=\cosh(a)\cosh(b)-\sinh(a)\sinh(b)
(12) =cosh⁡(a−b),\displaystyle=\cosh(a-b),
dℍ​(Φ⁡(u),Φ⁡(v))\displaystyle d_{\mathbb{H}}\left(\Phi(u),\Phi(v)\right) =arcosh⁡(cosh⁡(a−b))\displaystyle=\operatorname{arcosh}\left(\cosh(a-b)\right)
(13) =|a−b|.\displaystyle=|a-b|.
dℍ​(Φ⁡(u),Φ⁡(v))\displaystyle d_{\mathbb{H}}\left(\Phi(u),\Phi(v)\right) =|1d​𝟏d⊤​u−1d​𝟏d⊤​v|\displaystyle=\left|\frac{1}{d}\mathbf{1}_{d}^{\top}u-\frac{1}{d}\mathbf{1}_{d}^{\top}v\right|
(14) =1d​|𝟏d⊤​(u−v)|.\displaystyle=\frac{1}{d}\left|\mathbf{1}_{d}^{\top}(u-v)\right|.
1d​|𝟏d⊤​(u−v)|\displaystyle\frac{1}{d}\left|\mathbf{1}_{d}^{\top}(u-v)\right| ≤1d​‖𝟏d‖2​‖u−v‖2\displaystyle\leq\frac{1}{d}\|\mathbf{1}_{d}\|_{2}\|u-v\|_{2}
(15) =1d​‖u−v‖2,\displaystyle=\frac{1}{\sqrt{d}}\|u-v\|_{2},
(16) |m⁡(u)−m⁡(v)|=0,|m(u)-m(v)|=0,
(17) 𝟏d⊤​(u−v)=0.\mathbf{1}_{d}^{\top}(u-v)=0.
(18) φβ​(r):={r22​β,|r|≤β,|r|−β2,|r|>β.\varphi_{\beta}(r):=\begin{cases}\dfrac{r^{2}}{2\beta},&|r|\leq\beta,\\[6.0pt] |r|-\dfrac{\beta}{2},&|r|>\beta.\end{cases}
(19) 𝖷:=scℓ(ℓ)+pt(ℓ)\mathsf{X}:=s_{c_{\ell}}^{(\ell)}+p_{t}^{(\ell)}
(20) 𝖸:=Φ⁡(ut(ℓ))∈ℝ2\mathsf{Y}:=\Phi\left(u_{t}^{(\ell)}\right)\in\mathbb{R}^{2}
(21) Qj,x​(a):=𝔼⁡[φβ​(a−𝖸j)|𝖷=x].Q_{j,x}(a):=\mathbb{E}\left[\varphi_{\beta}\left(a-\mathsf{Y}_{j}\right)\,\middle|\,\mathsf{X}=x\right].
Proposition 0 (Population target of coordinate-wise Smooth L1).

For ℙ𝖷\mathbb{P}_{\mathsf{X}}-almost every xx, suppose that

Equation 22
(22) 𝔼⁡[|𝖸j||𝖷=x]<∞.\mathbb{E}\left[|\mathsf{Y}_{j}|\,\middle|\,\mathsf{X}=x\right]<\infty.

Then Qj,xQ_{j,x} is convex, coercive, and admits at least one minimizer. A scalar aj,x⋆a_{j,x}^{\star} minimizes Qj,xQ_{j,x} if and only if

Equation 23
(23) 𝔼⁡[clip⁡(aj,x⋆−𝖸j,−β,β)|𝖷=x]=0.\mathbb{E}\left[\operatorname{clip}\left(a_{j,x}^{\star}-\mathsf{Y}_{j},-\beta,\beta\right)\,\middle|\,\mathsf{X}=x\right]=0.

Thus, the population-optimal coordinate-wise Smooth L1 predictor returns a conditional Huber location functional of the target coordinate.

In particular, let

Equation 24
(24) μj​(x):=𝔼⁡[𝖸j|𝖷=x].\mu_{j}(x):=\mathbb{E}\left[\mathsf{Y}_{j}\,\middle|\,\mathsf{X}=x\right].

If

Equation 25
(25) Pr⁡(|𝖸j−μj​(x)|≤β|𝖷=x)=1,\Pr\left(|\mathsf{Y}_{j}-\mu_{j}(x)|\leq\beta\,\middle|\,\mathsf{X}=x\right)=1,

then μj​(x)\mu_{j}(x) is a minimizer of Qj,xQ_{j,x}.

Equations 26–33
(26) φβ′​(r)=1β​clip⁡(r,−β,β).\varphi_{\beta}^{\prime}(r)=\frac{1}{\beta}\operatorname{clip}\left(r,-\beta,\beta\right).
(27) |φβ′​(r)|≤1|\varphi_{\beta}^{\prime}(r)|\leq 1
Qj,x′​(a)\displaystyle Q_{j,x}^{\prime}(a) =𝔼⁡[φβ′​(a−𝖸j)|𝖷=x]\displaystyle=\mathbb{E}\left[\varphi_{\beta}^{\prime}\left(a-\mathsf{Y}_{j}\right)\,\middle|\,\mathsf{X}=x\right]
(28) =1β​𝔼​[clip⁡(a−𝖸j,−β,β)|𝖷=x].\displaystyle=\frac{1}{\beta}\mathbb{E}\left[\operatorname{clip}\left(a-\mathsf{Y}_{j},-\beta,\beta\right)\,\middle|\,\mathsf{X}=x\right].
(29) φβ​(r)≥|r|−β2.\varphi_{\beta}(r)\geq|r|-\frac{\beta}{2}.
Qj,x​(a)\displaystyle Q_{j,x}(a) ≥𝔼⁡[|a−𝖸j||𝖷=x]−β2\displaystyle\geq\mathbb{E}\left[|a-\mathsf{Y}_{j}|\,\middle|\,\mathsf{X}=x\right]-\frac{\beta}{2}
(30) ≥|a|−𝔼⁡[|𝖸j||𝖷=x]−β2.\displaystyle\geq|a|-\mathbb{E}\left[|\mathsf{Y}_{j}|\,\middle|\,\mathsf{X}=x\right]-\frac{\beta}{2}.
(31) Qj,x′​(aj,x⋆)=0.Q_{j,x}^{\prime}(a_{j,x}^{\star})=0.
(32) clip⁡(μj​(x)−𝖸j,−β,β)=μj​(x)−𝖸j.\operatorname{clip}\left(\mu_{j}(x)-\mathsf{Y}_{j},-\beta,\beta\right)=\mu_{j}(x)-\mathsf{Y}_{j}.
𝔼⁡[clip⁡(μj​(x)−𝖸j,−β,β)|𝖷=x]\displaystyle\mathbb{E}\left[\operatorname{clip}\left(\mu_{j}(x)-\mathsf{Y}_{j},-\beta,\beta\right)\,\middle|\,\mathsf{X}=x\right] =μj​(x)−𝔼⁡[𝖸j|𝖷=x]\displaystyle=\mu_{j}(x)-\mathbb{E}\left[\mathsf{Y}_{j}\,\middle|\,\mathsf{X}=x\right]
(33) =0.\displaystyle=0.

Appendix C Generalization of Task-Specific Resolution Weighting

The readout bound assumes a pretrained encoder treated as fixed, independent labeled examples, bounded feature/readout norms and an appropriate bounded Lipschitz loss. Within the closed mixture family, approximate empirical risk minimization competes with the best fixed resolution up to complexity and optimization terms. Finite softmax parameters approach simplex vertices only in the closure; uniform smoothing adds an approximation term. These statements do not guarantee convergence of the actual nonconvex pretraining or tuning algorithm.

Equations 34–37
(34) S={(Gi,Yi)}i=1NS=\left\{(G_{i},Y_{i})\right\}_{i=1}^{N}
(35) ℱℓ:={fℓ,w​(G)=⟨w,hG(ℓ)⟩:‖w‖2≤B}.\mathcal{F}_{\ell}:=\left\{f_{\ell,w}(G)=\left\langle w,h_{G}^{(\ell)}\right\rangle:\|w\|_{2}\leq B\right\}.
(36) ΔL:={α∈ℝ+L:∑ℓ=1Lαℓ=1}\Delta_{L}:=\left\{\alpha\in\mathbb{R}_{+}^{L}:\sum_{\ell=1}^{L}\alpha_{\ell}=1\right\}
(37) ℱmix:={fw,α(G)=⟨w,∑ℓ=1LαℓhG(ℓ)⟩:∥w∥2≤B,α∈ΔL}.\mathcal{F}_{\mathrm{mix}}:=\left\{f_{w,\alpha}(G)=\left\langle w,\sum_{\ell=1}^{L}\alpha_{\ell}h_{G}^{(\ell)}\right\rangle:\|w\|_{2}\leq B,\;\alpha\in\Delta_{L}\right\}.
Assumption 1 (Bounded downstream hypothesis class).

There exist constants Rh,B,κ,M>0R_{h},B,\kappa,M>0 such that the following conditions hold:

Equations 38–41
(38) ‖hG(ℓ)‖2\displaystyle\|h_{G}^{(\ell)}\|_{2} ≤Rh\displaystyle\leq R_{h} for every G and ℓ,\displaystyle\text{for every $G$ and $\ell$},
(39) ‖w‖2\displaystyle\|w\|_{2} ≤B,\displaystyle\leq B,
(40) 0≤ℓds​(y^,y)\displaystyle 0\leq\ell_{\mathrm{ds}}(\widehat{y},y) ≤M,\displaystyle\leq M,
(41) |ℓds​(a,y)−ℓds​(b,y)|\displaystyle\left|\ell_{\mathrm{ds}}(a,y)-\ell_{\mathrm{ds}}(b,y)\right| ≤κ​|a−b|\displaystyle\leq\kappa|a-b| for every a,b,y.\displaystyle\text{for every $a,b,y$}.
Equations 42–44
(42) ℛ⁡(f):=𝔼(G,Y)∼ℙ​[ℓds​(f⁡(G),Y)]\mathcal{R}(f):=\mathbb{E}_{(G,Y)\sim\mathbb{P}}\left[\ell_{\mathrm{ds}}\left(f(G),Y\right)\right]
(43) ℛ^S​(f):=1N​∑i=1Nℓds​(f⁡(Gi),Yi).\widehat{\mathcal{R}}_{S}(f):=\frac{1}{N}\sum_{i=1}^{N}\ell_{\mathrm{ds}}\left(f(G_{i}),Y_{i}\right).
(44) ℛ^S​(f^mix)≤inff∈ℱmixℛ^S​(f)+εopt.\widehat{\mathcal{R}}_{S}\left(\widehat{f}_{\mathrm{mix}}\right)\leq\inf_{f\in\mathcal{F}_{\mathrm{mix}}}\widehat{\mathcal{R}}_{S}(f)+\varepsilon_{\mathrm{opt}}.
Theorem 1 (Resolution-adaptive oracle bound).

Under Assumption 1, there exist universal constants C1,C2>0C_{1},C_{2}>0 such that, for every δ∈(0,1)\delta\in(0,1), with probability at least 1−δ1-\delta,

Equation 45
ℛ⁡(f^mix)≤\displaystyle\mathcal{R}\left(\widehat{f}_{\mathrm{mix}}\right)\leq\; min⁡inff∈ℱℓℓ∈{1,…,L}⁡ℛ⁡(f)\displaystyle\min_{\ell\in\{1,\ldots,L\}}\inf_{f\in\mathcal{F}_{\ell}}\mathcal{R}(f)
(45) +C1​κ​B​Rh​1+log⁡LN+C2​M​log⁡(2/δ)N+εopt.\displaystyle+C_{1}\kappa BR_{h}\sqrt{\frac{1+\log L}{N}}+C_{2}M\sqrt{\frac{\log(2/\delta)}{N}}+\varepsilon_{\mathrm{opt}}.

Thus, the closed resolution-weighting family competes with the best fixed-resolution linear predictor selected in hindsight, while its additional dependence on the number of configured resolutions is logarithmic in LL.

Equations 46–66
(46) ℜ^S​(ℱ):=𝔼σ​[supf∈ℱ1N​∑i=1Nσi​f​(Gi)],\widehat{\mathfrak{R}}_{S}(\mathcal{F}):=\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}f(G_{i})\right],
ℜ^S​(ℱℓ)\displaystyle\widehat{\mathfrak{R}}_{S}(\mathcal{F}_{\ell}) =𝔼σ​[sup‖w‖2≤B1N​∑i=1Nσi​⟨w,hGi(ℓ)⟩]\displaystyle=\mathbb{E}_{\sigma}\left[\sup_{\|w\|_{2}\leq B}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\left\langle w,h_{G_{i}}^{(\ell)}\right\rangle\right]
(47) =BN​𝔼σ​‖∑i=1Nσi​hGi(ℓ)‖2.\displaystyle=\frac{B}{N}\mathbb{E}_{\sigma}\left\|\sum_{i=1}^{N}\sigma_{i}h_{G_{i}}^{(\ell)}\right\|_{2}.
(48) 𝔼σ​‖∑i=1Nσi​hGi(ℓ)‖2\displaystyle\mathbb{E}_{\sigma}\left\|\sum_{i=1}^{N}\sigma_{i}h_{G_{i}}^{(\ell)}\right\|_{2} ≤(𝔼σ​‖∑i=1Nσi​hGi(ℓ)‖22)1/2.\displaystyle\leq\left(\mathbb{E}_{\sigma}\left\|\sum_{i=1}^{N}\sigma_{i}h_{G_{i}}^{(\ell)}\right\|_{2}^{2}\right)^{1/2}.
𝔼σ​‖∑i=1Nσi​hGi(ℓ)‖22\displaystyle\mathbb{E}_{\sigma}\left\|\sum_{i=1}^{N}\sigma_{i}h_{G_{i}}^{(\ell)}\right\|_{2}^{2} =∑i=1N‖hGi(ℓ)‖22\displaystyle=\sum_{i=1}^{N}\left\|h_{G_{i}}^{(\ell)}\right\|_{2}^{2}
(49) ≤N​Rh2.\displaystyle\leq NR_{h}^{2}.
(50) ℜ^S​(ℱℓ)≤B​RhN.\widehat{\mathfrak{R}}_{S}(\mathcal{F}_{\ell})\leq\frac{BR_{h}}{\sqrt{N}}.
(51) Zℓ​(σ):=supf∈ℱℓ1N​∑i=1Nσi​f​(Gi).Z_{\ell}(\sigma):=\sup_{f\in\mathcal{F}_{\ell}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}f(G_{i}).
(52) |f⁡(Gi)|≤‖w‖2​‖hGi(ℓ)‖2≤B​Rh.|f(G_{i})|\leq\|w\|_{2}\|h_{G_{i}}^{(\ell)}\|_{2}\leq BR_{h}.
(53) 𝔼σ​[exp⁡(λ⁡[Zℓ​(σ)−𝔼σ​Zℓ​(σ)])]≤exp⁡(λ2​B2​Rh22​N)\mathbb{E}_{\sigma}\left[\exp\left(\lambda\left[Z_{\ell}(\sigma)-\mathbb{E}_{\sigma}Z_{\ell}(\sigma)\right]\right)\right]\leq\exp\left(\frac{\lambda^{2}B^{2}R_{h}^{2}}{2N}\right)
(54) 𝔼σ​[maxℓ∈{1,…,L}⁡Zℓ​(σ)]≤\displaystyle\mathbb{E}_{\sigma}\left[\max_{\ell\in\{1,\ldots,L\}}Z_{\ell}(\sigma)\right]\leq\; maxℓ⁡𝔼σ​[Zℓ​(σ)]+log⁡Lλ+λ​B2​Rh22​N.\displaystyle\max_{\ell}\mathbb{E}_{\sigma}\left[Z_{\ell}(\sigma)\right]+\frac{\log L}{\lambda}+\frac{\lambda B^{2}R_{h}^{2}}{2N}.
ℜ^S​(⋃ℓ=1Lℱℓ)\displaystyle\widehat{\mathfrak{R}}_{S}\left(\bigcup_{\ell=1}^{L}\mathcal{F}_{\ell}\right) ≤maxℓ⁡ℜ^S​(ℱℓ)+B​Rh​2​log⁡LN\displaystyle\leq\max_{\ell}\widehat{\mathfrak{R}}_{S}(\mathcal{F}_{\ell})+BR_{h}\sqrt{\frac{2\log L}{N}}
(55) ≤B​Rh​(1N+2​log⁡LN).\displaystyle\leq BR_{h}\left(\frac{1}{\sqrt{N}}+\sqrt{\frac{2\log L}{N}}\right).
fw,α​(G)\displaystyle f_{w,\alpha}(G) =⟨w,∑ℓ=1Lαℓ​hG(ℓ)⟩\displaystyle=\left\langle w,\sum_{\ell=1}^{L}\alpha_{\ell}h_{G}^{(\ell)}\right\rangle
(56) =∑ℓ=1Lαℓ​⟨w,hG(ℓ)⟩.\displaystyle=\sum_{\ell=1}^{L}\alpha_{\ell}\left\langle w,h_{G}^{(\ell)}\right\rangle.
(57) ℱmix⊆conv⁡(⋃ℓ=1Lℱℓ).\mathcal{F}_{\mathrm{mix}}\subseteq\operatorname{conv}\left(\bigcup_{\ell=1}^{L}\mathcal{F}_{\ell}\right).
ℜ^S​(ℱmix)\displaystyle\widehat{\mathfrak{R}}_{S}\left(\mathcal{F}_{\mathrm{mix}}\right) ≤ℜ^S​(conv⁡(⋃ℓ=1Lℱℓ))\displaystyle\leq\widehat{\mathfrak{R}}_{S}\left(\operatorname{conv}\left(\bigcup_{\ell=1}^{L}\mathcal{F}_{\ell}\right)\right)
(58) =ℜ^S​(⋃ℓ=1Lℱℓ).\displaystyle=\widehat{\mathfrak{R}}_{S}\left(\bigcup_{\ell=1}^{L}\mathcal{F}_{\ell}\right).
(59) ℜN​(ℱmix)≤B​Rh​(1N+2​log⁡LN),\mathfrak{R}_{N}\left(\mathcal{F}_{\mathrm{mix}}\right)\leq BR_{h}\left(\frac{1}{\sqrt{N}}+\sqrt{\frac{2\log L}{N}}\right),
supf∈ℱmix|ℛ⁡(f)−ℛ^S​(f)|≤\displaystyle\sup_{f\in\mathcal{F}_{\mathrm{mix}}}\left|\mathcal{R}(f)-\widehat{\mathcal{R}}_{S}(f)\right|\leq\; C0​κ​B​Rh​(1N+2​log⁡LN)\displaystyle C_{0}\kappa BR_{h}\left(\frac{1}{\sqrt{N}}+\sqrt{\frac{2\log L}{N}}\right)
(60) +C0​M​log⁡(2/δ)N.\displaystyle+C_{0}M\sqrt{\frac{\log(2/\delta)}{N}}.
ℛ⁡(f^mix)\displaystyle\mathcal{R}\left(\widehat{f}_{\mathrm{mix}}\right) ≤ℛ^S​(f^mix)+ΓN​(δ)\displaystyle\leq\widehat{\mathcal{R}}_{S}\left(\widehat{f}_{\mathrm{mix}}\right)+\Gamma_{N}(\delta)
≤inff∈ℱmixℛ^S​(f)+εopt+ΓN​(δ)\displaystyle\leq\inf_{f\in\mathcal{F}_{\mathrm{mix}}}\widehat{\mathcal{R}}_{S}(f)+\varepsilon_{\mathrm{opt}}+\Gamma_{N}(\delta)
(61) ≤inff∈ℱmixℛ⁡(f)+εopt+2​ΓN​(δ).\displaystyle\leq\inf_{f\in\mathcal{F}_{\mathrm{mix}}}\mathcal{R}(f)+\varepsilon_{\mathrm{opt}}+2\Gamma_{N}(\delta).
(62) fw,eℓ​(G)=⟨w,hG(ℓ)⟩.f_{w,e_{\ell}}(G)=\left\langle w,h_{G}^{(\ell)}\right\rangle.
(63) ⋃ℓ=1Lℱℓ⊆ℱmix,\bigcup_{\ell=1}^{L}\mathcal{F}_{\ell}\subseteq\mathcal{F}_{\mathrm{mix}},
(64) inff∈ℱmixℛ⁡(f)≤min⁡inff∈ℱℓℓ∈{1,…,L}⁡ℛ⁡(f).\inf_{f\in\mathcal{F}_{\mathrm{mix}}}\mathcal{R}(f)\leq\min_{\ell\in\{1,\ldots,L\}}\inf_{f\in\mathcal{F}_{\ell}}\mathcal{R}(f).
(65) uL:=1L​𝟏Lu_{L}:=\frac{1}{L}\mathbf{1}_{L}
(66) α(λ):=(1−λunif)​α+λunif​uL,λunif∈[0,1].\alpha^{(\lambda)}:=(1-\lambda_{\mathrm{unif}})\alpha+\lambda_{\mathrm{unif}}u_{L},\qquad\lambda_{\mathrm{unif}}\in[0,1].
Lemma 0 (Stability under uniform resolution smoothing).

For α∈ΔL\alpha\in\Delta_{L}, define

Equation 67
(67) hGα:=∑ℓ=1Lαℓ​hG(ℓ)h_{G}^{\alpha}:=\sum_{\ell=1}^{L}\alpha_{\ell}h_{G}^{(\ell)}

and

Equation 68
(68) hGα(λ):=∑ℓ=1Lαℓ(λ)​hG(ℓ).h_{G}^{\alpha^{(\lambda)}}:=\sum_{\ell=1}^{L}\alpha_{\ell}^{(\lambda)}h_{G}^{(\ell)}.

Under Assumption 1,

Equation 69
(69) ‖hGα(λ)−hGα‖2≤2​λunif​Rh.\left\|h_{G}^{\alpha^{(\lambda)}}-h_{G}^{\alpha}\right\|_{2}\leq 2\lambda_{\mathrm{unif}}R_{h}.

Consequently, for every ‖w‖2≤B\|w\|_{2}\leq B,

Equation 70
(70) |⟨w,hGα(λ)⟩−⟨w,hGα⟩|≤2​λunif​B​Rh.\left|\left\langle w,h_{G}^{\alpha^{(\lambda)}}\right\rangle-\left\langle w,h_{G}^{\alpha}\right\rangle\right|\leq 2\lambda_{\mathrm{unif}}BR_{h}.

The corresponding population risks satisfy

Equation 71
(71) ℛ⁡(fw,α(λ))≤ℛ⁡(fw,α)+2​κ​λunif​B​Rh.\mathcal{R}\left(f_{w,\alpha^{(\lambda)}}\right)\leq\mathcal{R}\left(f_{w,\alpha}\right)+2\kappa\lambda_{\mathrm{unif}}BR_{h}.
Equations 72–76
(72) hGα(λ)−hGα\displaystyle h_{G}^{\alpha^{(\lambda)}}-h_{G}^{\alpha} =λunif​(1L​∑ℓ=1LhG(ℓ)−∑ℓ=1Lαℓ​hG(ℓ)).\displaystyle=\lambda_{\mathrm{unif}}\left(\frac{1}{L}\sum_{\ell=1}^{L}h_{G}^{(\ell)}-\sum_{\ell=1}^{L}\alpha_{\ell}h_{G}^{(\ell)}\right).
‖hGα(λ)−hGα‖2≤\displaystyle\left\|h_{G}^{\alpha^{(\lambda)}}-h_{G}^{\alpha}\right\|_{2}\leq\; λunif​‖1L​∑ℓ=1LhG(ℓ)‖2+λunif​‖∑ℓ=1Lαℓ​hG(ℓ)‖2\displaystyle\lambda_{\mathrm{unif}}\left\|\frac{1}{L}\sum_{\ell=1}^{L}h_{G}^{(\ell)}\right\|_{2}+\lambda_{\mathrm{unif}}\left\|\sum_{\ell=1}^{L}\alpha_{\ell}h_{G}^{(\ell)}\right\|_{2}
≤\displaystyle\leq\; λunif​1L​∑ℓ=1L‖hG(ℓ)‖2+λunif​∑ℓ=1Lαℓ​‖hG(ℓ)‖2\displaystyle\lambda_{\mathrm{unif}}\frac{1}{L}\sum_{\ell=1}^{L}\left\|h_{G}^{(\ell)}\right\|_{2}+\lambda_{\mathrm{unif}}\sum_{\ell=1}^{L}\alpha_{\ell}\left\|h_{G}^{(\ell)}\right\|_{2}
(73) ≤\displaystyle\leq\; 2​λunif​Rh.\displaystyle 2\lambda_{\mathrm{unif}}R_{h}.
|⟨w,hGα(λ)−hGα⟩|\displaystyle\left|\left\langle w,h_{G}^{\alpha^{(\lambda)}}-h_{G}^{\alpha}\right\rangle\right| ≤‖w‖2​‖hGα(λ)−hGα‖2\displaystyle\leq\|w\|_{2}\left\|h_{G}^{\alpha^{(\lambda)}}-h_{G}^{\alpha}\right\|_{2}
(74) ≤2​λunif​B​Rh,\displaystyle\leq 2\lambda_{\mathrm{unif}}BR_{h},
|ℓds​(fw,α(λ)​(G),Y)−ℓds​(fw,α​(G),Y)|\displaystyle\left|\ell_{\mathrm{ds}}\left(f_{w,\alpha^{(\lambda)}}(G),Y\right)-\ell_{\mathrm{ds}}\left(f_{w,\alpha}(G),Y\right)\right|
≤κ​|fw,α(λ)​(G)−fw,α​(G)|\displaystyle\hskip 99.58464pt\leq\kappa\left|f_{w,\alpha^{(\lambda)}}(G)-f_{w,\alpha}(G)\right|
(75) ≤2​κ​λunif​B​Rh.\displaystyle\hskip 99.58464pt\leq 2\kappa\lambda_{\mathrm{unif}}BR_{h}.
(76) ℱmix(λ):={fw,α(λ):∥w∥2≤B,α∈ΔL}.\mathcal{F}_{\mathrm{mix}}^{(\lambda)}:=\left\{f_{w,\alpha^{(\lambda)}}:\|w\|_{2}\leq B,\;\alpha\in\Delta_{L}\right\}.
Corollary 3 (Oracle bound under uniform smoothing).

Under Assumption 1, there exist universal constants C1,C2>0C_{1},C_{2}>0 such that, for every δ∈(0,1)\delta\in(0,1), with probability at least 1−δ1-\delta,

Equation 77
ℛ⁡(f^mix(λ))≤\displaystyle\mathcal{R}\left(\widehat{f}_{\mathrm{mix}}^{(\lambda)}\right)\leq\; min⁡inff∈ℱℓℓ∈{1,…,L}⁡ℛ⁡(f)+2​κ​λunif​B​Rh\displaystyle\min_{\ell\in\{1,\ldots,L\}}\inf_{f\in\mathcal{F}_{\ell}}\mathcal{R}(f)+2\kappa\lambda_{\mathrm{unif}}BR_{h}
(77) +C1​κ​B​Rh​1+log⁡LN+C2​M​log⁡(2/δ)N+εopt.\displaystyle+C_{1}\kappa BR_{h}\sqrt{\frac{1+\log L}{N}}+C_{2}M\sqrt{\frac{\log(2/\delta)}{N}}+\varepsilon_{\mathrm{opt}}.

Therefore, uniform smoothing introduces an additional approximation term that is at most linear in λunif\lambda_{\mathrm{unif}}.

Equations 78–81
(78) ℱmix(λ)⊆ℱmix.\mathcal{F}_{\mathrm{mix}}^{(\lambda)}\subseteq\mathcal{F}_{\mathrm{mix}}.
(79) hGeℓ=hG(ℓ).h_{G}^{e_{\ell}}=h_{G}^{(\ell)}.
(80) ℛ⁡(fw,eℓ(λ))≤ℛ⁡(fℓ,w)+2​κ​λunif​B​Rh.\mathcal{R}\left(f_{w,e_{\ell}^{(\lambda)}}\right)\leq\mathcal{R}\left(f_{\ell,w}\right)+2\kappa\lambda_{\mathrm{unif}}BR_{h}.
(81) inff∈ℱmix(λ)ℛ⁡(f)≤\displaystyle\inf_{f\in\mathcal{F}_{\mathrm{mix}}^{(\lambda)}}\mathcal{R}(f)\leq\; min⁡inff∈ℱℓℓ∈{1,…,L}⁡ℛ⁡(f)+2​κ​λunif​B​Rh.\displaystyle\min_{\ell\in\{1,\ldots,L\}}\inf_{f\in\mathcal{F}_{\ell}}\mathcal{R}(f)+2\kappa\lambda_{\mathrm{unif}}BR_{h}.
Concept demonstration

Open image ↗