Overview
This edition covers arXiv v2, including the full kernel, communication and model-configuration details. Figure 1 compares ordinary residual connections, Hyper-Connections and mHC. The later ICML 2026 publication lists Mingyu Xu where this source version lists Jiang Chang; this edition preserves the v2 byline.
At a glance
Intuition. Multiple residual streams are parallel paths carrying intermediate activations. Their mixer acts like a small signal-routing circuit. Repeating a mixer with gain two would amplify a signal exponentially across layers, so the paper restricts how streams can be mixed.
Reading the math. A two-stream example is [[0.8, 0.2], [0.2, 0.8]]: each output is a weighted average, and both row and column totals are one. These constraints define the doubly stochastic mixing matrices; Sinkhorn iterations approximately enforce them. They prevent the mixer from increasing the Euclidean norm across streams, but can still shrink differences. The restriction concerns this mixing operation, not a guarantee that every part of the network preserves signal size.
Hyper-Connections widen a network’s residual stream and learn how its parallel streams mix, but unconstrained repeated mixing can destabilize training and increase memory traffic. mHC approximately constrains residual mixing to nonnegative matrices with unit row and column sums, using Sinkhorn–Knopp normalization. Fused kernels, selective recomputation and communication overlap make the added streams more practical. Experiments on mixture-of-experts language models show better training stability and downstream results, with a reported 6.7% training-time overhead for the optimized four-stream implementation.
1 Introduction
Identity residual paths let information pass directly through depth. Hyper-Connections instead maintain n streams: a pre-map reads them into one layer input, a post-map writes the layer output back, and a residual map mixes the streams. Products of unconstrained residual maps can amplify or suppress signals. mHC targets this mixing with an approximately doubly stochastic constraint, while separately reducing the resulting memory and communication costs. Conserving a stream mean is weaker than reproducing an exact identity map on every feature.
Equations 1–4
| (1) |
| (2) |
| (3) |
| (4) |
2 Related Works
The paper separates changes inside computational blocks from changes to the topology connecting those blocks. mHC addresses the latter.
2.1 Micro Design
Micro-design includes convolution, attention and feed-forward layers, along with efficiency variants such as grouped-query or latent attention and sparse mixture-of-experts feed-forward computation. These change how each block transforms its inputs.
2.2 Macro Design
Macro-design controls routing and merging across layers. DenseNet, FractalNet and aggregation methods expand paths; recent methods widen or dynamically combine residual states. mHC builds specifically on HC, retaining learned inter-stream communication while constraining the residual map and optimizing its implementation.
3 Preliminary
Represent the widened hidden state as an n×C matrix. Input-dependent projections plus static biases generate the pre-, post- and residual maps; small learned gates modulate the dynamic components. The expensive layer function still receives width C, so residual-state capacity can grow without proportionally enlarging that function’s FLOPs. In Table 1’s ablation, enabling residual mixing gives most of the observed loss improvement (−.022), with pre/post maps reaching −.025/−.027. Disabled maps use specified fixed weights to keep dimensions compatible.
Equation 5
| (5) |
| Absolute Loss Gap | |||
| ✓ | |||
| ✓ | ✓ | ||
| ✓ | ✓ | ✓ |
3.1 Numerical Instability
The product of learned residual matrices controls the direct multi-layer signal path. In the 27B HC run, a loss surge near step 12,000 coincides with unstable gradient norms. The reported forward/backward gain diagnostics use row/column sums of the composed map; peaks approach 3,000. These diagnose the residual branch’s propagation, rather than the complete input-dependent network Jacobian.
3.2 System Overhead
Small added FLOP counts can conceal substantial memory traffic. Table 2 counts forward residual-maintenance reads/writes, excluding the layer function itself. Widening also increases saved activations and pipeline communication by roughly the stream expansion factor. Fused kernels, recomputation and scheduling are therefore central to the practical method.
| Method | Operation | Read (Elements) | Write (Elements) |
|
Residual Connection |
Residual Merge | ||
| Total I/O | |||
|
Hyper- Connections |
Calculate , , | ||
| Residual Merge | |||
| Total I/O |
4 Method
mHC couples constrained mixing with a different coefficient parameterization and an implementation designed for the widened residual state.
4.1 Manifold-Constrained Hyper-Connections
Constrain the residual matrix to the Birkhoff polytope: nonnegative entries and row/column sums equal to one. Such a map forms convex combinations of streams, preserves their mean, is non-expansive in spectral norm, and remains doubly stochastic under composition. It can still contract differences between streams, so the source’s “norm preservation” wording should not be read as preserving every vector’s norm or excluding all vanishing-gradient mechanisms. At n=1 it reduces to the ordinary identity residual map. Pre/post coefficients are separately constrained to be nonnegative.
Equation 6
| (6) |
4.2 Parameterization and Manifold Projection
Flatten the entire n×C state before RMS normalization and dynamic projections, allowing each coefficient to depend on all streams. Apply sigmoid to pre-coefficients, twice sigmoid to post-coefficients, and exponentiation followed by alternating column/row normalization to the residual matrix. Sinkhorn–Knopp converges to double stochasticity in the limit; the experiments use twenty iterations, producing an approximation rather than exact unit sums.
Equations 7–9
| (7) |
| (8) |
| (9) |
4.3 Efficient Infrastructure Design
The optimized n=4 implementation reports 6.7% additional training time. This is an infrastructure-dependent measurement, not a cost-free architectural change.
4.3.1 Kernel Fusion
Move RMS division after projection and absorb normalization weights into the projection. Fuse the projection and norm scans; fuse small coefficient operations; run Sinkhorn iterations in one kernel with a custom recomputing backward pass. Additional kernels apply the pre-map and combine post-map, residual mixing and merge. For the latter kernel, reads fall from (3n+1)C to (n+1)C and writes from 3nC to nC. Equations 10–19 retain the precise tensor shapes, precision choices and transformations. Most kernels use TileLang.
Equations 10–19
| (10) | ||||||
| (11) | ||||||
| Scalars | (12) | |||||
| (13) | ||||||
| (14) | ||||||
| (15) | ||||||
| (16) | ||||||
| (17) | ||||||
| (18) | ||||||
| (19) |
4.3.2 Recomputing
Discard mHC intermediates after the forward pass and recompute them in blocks during backward, without rerunning the expensive layer function. Save the block’s input plus each layer-function output. Balancing resident block inputs against transient reconstructed states gives the approximate block length √(nL/(n+2)) in Equation 20. Blocks cannot cross pipeline stages; the implementation aligns their boundaries with stages. Table 3 preserves which activations are stored or transient.
| Activations | |||||
| Size (Elements) | |||||
| Stored Method | Every layers | Every layer | Transient inside layers | ||
Equation 20
| (20) |
4.3.3 Overlapping Communication in DualPipe
Extend DualPipe to overlap the additional communication and block recomputation. Run the feed-forward post/residual kernel on a high-priority compute stream and avoid long-running persistent attention kernels that would delay it. Cached stage-input activations let recomputation proceed independently of incoming pipeline communication. Figure 4 is a scheduling schematic; block widths do not represent measured durations.
5 Experiments
Experiments compare ordinary residual connections, unconstrained HC and mHC using the same four-stream expansion for the latter two.
5.1 Experimental Setup
DeepSeek-V3-inspired mixture-of-experts models cover 3B, 9B and 27B total parameters, with parameter-proportional training data. A separate 3B run examines token scaling on approximately one trillion tokens. The principal stability and downstream comparison uses the 27B model; Appendix A gives all active-parameter counts and schedules.
5.2 Main Results
The 27B mHC run finishes .021 loss below the baseline and avoids HC’s reported loss/gradient disturbance. It beats the baseline on all eight displayed benchmarks and HC on seven; MATH is 26.0 for mHC versus 26.4 for HC. Relative to HC, BBH rises 2.1 points and DROP 2.3 points. Table 4 preserves task-specific metrics and zero/few-shot settings, so these gains are percentage-point score differences rather than a uniform relative improvement.
| Benchmark | BBH | DROP | GSM8K | HellaSwag | MATH | MMLU | PIQA | TriviaQA |
| (Metric) | (EM) | (F1) | (EM) | (Acc.) | (EM) | (Acc.) | (Acc.) | (EM) |
| # Shots | 3-shot | 3-shot | 8-shot | 10-shot | 4-shot | 5-shot | 0-shot | 5-shot |
| 27B Baseline | 43.8 | 47.0 | 46.7 | 73.7 | 22.0 | 59.0 | 78.5 | 54.3 |
| 27B w/ HC | 48.9 | 51.6 | 53.2 | 74.3 | 26.4 | 63.0 | 79.9 | 56.3 |
| 27B w/ mHC | 51.0 | 53.9 | 53.8 | 74.7 | 26.0 | 63.4 | 80.5 | 57.6 |
5.3 Scaling Experiments
The loss advantage over the baseline persists across the three reported compute/model scales, with some attenuation, and through the separate 3B token-scaling trajectory. Figure 6 provides the absolute gaps and relative ratios. These observations support scaling within the tested configurations; the additional in-house large-scale result is not accompanied by a comparable detailed benchmark table.
5.4 Stability Analysis
With twenty Sinkhorn iterations, single-layer backward gain deviates slightly from one and the composed-map gain reaches approximately 1.6. This is far below HC’s roughly 3,000 peak, but confirms that the practical constraint is approximate. Figures 7–8 retain all gain curves and representative matrices, averaged over tokens in a selected sequence. They measure the specified residual-map diagnostics rather than proving stability for every input or the full nonlinear network.
6 Conclusion and Outlook
Constraining inter-stream mixing and engineering its execution yields a more stable practical form of HC in the reported language-model runs. Mean conservation and non-expansive mixing explain part of the motivation; empirical training behavior supplies the broader evidence. Other geometric constraints and the balance between mixing flexibility and stability remain future work.
Appendix A Appendix
The appendix supplies the complete model, optimizer and training configurations used in the experiments.
A.1 Detailed Model Specifications and Hyper-parameters.
The 3B/9B/27B models have 12/18/30 layers and 612M/1.66B/4.14B active parameters. Proportional-data runs use 39.3B/105B/262B tokens; the separate 3B run uses 1.05T. All use length 4,096, expansion four, initial dynamic gates .01 and twenty Sinkhorn iterations. Table 5 retains expert counts, latent-attention and rotary settings, normalization, batch sizes, steps, warmup and learning-rate schedules. Total parameter count and per-token active parameter count are distinct quantities.
| Attribute | 3B | 9B | 27B | 3B |
| 1T Tokens | ||||
| Vocab Params | 331M | 496M | 662M | 331M |
| Active Params | 612M | 1.66B | 4.14B | 612M |
| Total Params | 2.97B | 9.18B | 27.0B | 2.97B |
| Layers | 12 | 18 | 30 | 12 |
| Leading Dense Layers | 1 | 1 | ||
| Routed Experts | 64 | 64 | 72 | 64 |
| Active Experts | 6 | 6 | ||
| Shared Experts | 2 | 2 | ||
| Dimension | 1280 | 1920 | 2560 | 1280 |
| FFN Dimension | 896 | 1280 | 1536 | 896 |
| Load Balancing Method | Loss-Free (Wang et al., 2024) | Loss-Free | ||
| Attention Heads | 16 | 24 | 32 | 16 |
| Attention Dimension | 128 | 128 | ||
| Attention Variant | MLA (Liu et al., 2024a) | MLA | ||
| KV Rank | 512 | 512 | ||
| Position Embedding | RoPE (Su et al., 2024) | RoPE | ||
| RoPE Dimension | 64 | 64 | ||
| RoPE | 10000 | 10000 | ||
| Layer Norm Type | RMSNorm (Zhang and Sennrich, 2019) | RMSNorm | ||
| Layer Norm | 1e-20 | 1e-20 | ||
| mHC/HC Expansion Rate | 4 | 4 | ||
| mHC/HC Gating Factor Init | 0.01 | 0.01 | ||
| mHC Sinkhorn-Knopp | 20 | 20 | ||
| Sequence Length | 4096 | 4096 | ||
| Vocab Size | 129280 | 129280 | ||
| Batch Size | 320 | 512 | 1280 | 2560 |
| Training Steps | 30000 | 50000 | 50000 | 100000 |
| Training Tokens | 39.3B | 105B | 262B | 1.05T |
| Warmup Steps | 2000 | 2000 | ||
| Optimizer | AdamW (Loshchilov and Hutter, 2017) | AdamW | ||
| AdamW Betas | (0.9, 0.95) | (0.9, 0.95) | ||
| AdamW | 1e-20 | 1e-20 | ||
| Base Learning Rate | 8.6e-4 | 5.9e-4 | 4.0e-4 | 9.0e-4 |
| Lr Scheduler | Step | Step | ||
| Lr Decay Step Ratio | [0.8 , 0.9 ] | [0.8 , 0.9 ] | ||
| Lr Decay Rate | [0.316, 0.1] | [0.316, 0.1] | ||
| Weight Decay | 0.1 | 0.1 | ||