Literature
Display
← All papers

Exploring Diffusion Transformer Designs via Grafting

Keshigeyan Chandrasegaran · Michael Poli · Daniel Y. Fu · Dongjun Kim · Lea M. Hadzic · Manling Li · Agrim Gupta · Stefano Massaroli · Azalia Mirhoseini · Juan Carlos Niebles · Stefano Ermon · Li Fei-Fei

· NeurIPS 2025 ↗

Computer vision12 min text · CondensedOriginal paper ↗
Contents

At a glance

Grafting studies new diffusion-transformer architectures by editing pretrained models. It first trains replacement operators to match intermediate activations, then fine-tunes the assembled model to repair accumulated errors. Interleaved attention replacements, altered feed-forward widths and parallelized transformer blocks preserve much of the original image quality with limited additional training. The method measures compatibility with an existing pretrained model; whether the resulting architecture would train equally well from scratch remains unresolved.

1 Introduction

Architecture experiments are expensive when each candidate must be pretrained. Grafting treats a pretrained diffusion transformer [1] as a reusable starting point. Its two problems are initializing a replacement operator and preventing errors from compounding when many replacements are combined. It explores which operator to replace, its alternative, the selected layers and the replacement fraction.

The main evidence covers ImageNet class-conditional generation with DiT-XL/2, high-resolution text-to-image generation with PixArt-Σ [11], and a depth-to-width restructuring case. Several hybrid variants retain low FID after a small adaptation budget. These are incremental costs on top of existing pretraining, not equivalent costs for training a new model from no checkpoint.

Refer to caption
Figure 1: Grafting overview. (a,b) Model architecture design via grafting. Studying new model architecture designs requires costly pretraining. Grafting materializes new architectures by editing pretrained models under small compute budgets (Sec. 3). (c) Class-conditional image generation. Samples generated by hybrid architectures obtained via grafting (Sec. 4). (d) High-resolution text-to-image generation. 2048×\times2048 samples generated using a grafted model (Sec. 5). (e) Depth →\rightarrow width case study. Samples generated using a model restructured via grafting (depth: 28 →\rightarrow 14) (Sec. 6).

2 Prerequisites

Latent diffusion adds time-dependent Gaussian noise to an encoded image and trains a denoiser conditioned on a class or text prompt. DiT processes patchified noisy latents using attention and feed-forward blocks. The original denoising objective remains available.

ImageNet evaluation generates 50,000 samples with 250-step DDPM sampling and classifier-free guidance 1.5, reporting FID, spatial FID, Inception Score, precision and recall. Text-to-image quality uses GenEval [16]. Low FID and high GenEval measure different aspects of generated distributions and prompt alignment; neither alone establishes visual fidelity on every image.

Equation 1
ℒD​M​(ϕ)=𝔼q⁡(t)​q​(𝐳,𝐜)​𝒩​(ϵ,0,I)​[‖ϵ−ϵϕ​(𝐳t,t,𝐜)‖22],\displaystyle\begin{split}\mathcal{L}_{DM}(\bm{\phi})=\mathbb{E}_{q(t)q(\mathbf{z},\mathbf{c})\mathcal{N}(\bm{\epsilon};0,I)}[\|\bm{\epsilon}-\bm{\epsilon}_{\bm{\phi}}(\mathbf{z}_{t},t,\mathbf{c})\|_{2}^{2}],\end{split} (1)

3 Grafting Diffusion Transformers

The two stages separately address operator initialization and whole-model recovery. Self-grafting is the control that preserves the architecture while replacing operator weights.

3.1 Two - Stage Grafting Approach

Activation distillation trains each replacement to reproduce its pretrained operator’s outputs under noisy inputs and conditioning. The regression loss compares teacher and student intermediate tensors, using around 8,000 training examples. This borrows the teacher’s behavior rather than deriving a new architecture objective from scratch.

After insertion, small operator errors can accumulate through the network. End-to-end fine-tuning with the ordinary denoising objective repairs this mismatch, commonly using 10% of ImageNet. Success therefore depends on both local imitation and global adaptation; a low isolated activation loss does not guarantee a usable assembled generator.

Equation 2
ℒ⁡(𝜽)=𝔼q⁡(t)​q​(𝐳,𝐜)​qt​(𝐳t|𝐳)​[ℒreg​(g𝜽l​(𝐳t,t,𝐜),fϕl​(𝐳t,t,𝐜))]\displaystyle\mathcal{L}(\bm{\theta})=\mathbb{E}_{q(t)q(\mathbf{z},\mathbf{c})q_{t}(\mathbf{z}_{t}|\mathbf{z})}\big[\mathcal{L}_{\text{reg}}(g_{\bm{\theta}}^{l}(\mathbf{z}_{t},t,\mathbf{c}),f_{\bm{\phi}}^{l}(\mathbf{z}_{t},t,\mathbf{c}))\big] (2)
[Uncaptioned image]

(iii) MHA initialization

IS ↑\uparrow FID ↓\downarrow sFID ↓\downarrow Prec. ↑\uparrow Rec. ↑\uparrow
Baseline 278.20 2.27 4.60 0.83 0.57
Random Init. 40.86 76.27 10.33 0.33 0.52
L2 269.31 2.58 5.75 0.82 0.58
Huber (δ\delta=1.0) 271.30 2.55 5.44 0.82 0.57
L1 273.03 2.51 5.48 0.83 0.58

(iv) MLP initialization

IS ↑\uparrow FID ↓\downarrow sFID ↓\downarrow Prec. ↑\uparrow Rec. ↑\uparrow
Baseline 278.20 2.27 4.60 0.83 0.57
Random Init. 2.18 297.34 161.76 0.01 0.00
L2 265.34 2.33 4.38 0.81 0.59
Huber (δ\delta=1.0) 262.93 2.38 4.49 0.81 0.59
L1 235.54 2.83 4.69 0.77 0.61
Table 1: Activation statistics and self-grafting (Stage 1) results (DiT-XL/2). (i) Activation ranges (max–min) across all 28 MHA and MLP operators, computed using 1,000 samples. Deeper layers exhibit higher variance in activation values. (ii) Activation distributions (log-scale histograms) for five selected layers (1, 8, 17, 27, 28), used in our initialization study. MLP layers show higher variance in activations than MHA, especially in deeper layers. (iii, iv) Stage 1 results for these layers using L2, Huber, and L1 regression. L1 yields the best FID for MHA (2.51), while L2 performs best for MLP (2.33), which contains 2×\times more parameters than MHA (10.6M vs. 5.3M). This study shows that high-quality initialization can be achieved by choosing operator-specific regression objectives.

3.2 Self - grafting Baseline

Self-grafting swaps attention or MLP operators for randomly initialized copies with identical shapes, receptive fields and parameter counts. It isolates the recovery procedure from changes in architecture. Its near-baseline results are evidence that the initialization/fine-tuning pipeline works, not evidence that attention can be removed without recovery.

3.3 Activation Behavior Analysis and Self-grafting Results

Activation ranges vary substantially across layers, especially in deeper MLPs. In five selected layers, L1 distillation yields attention FID 2.51 versus 2.58 for L2 and 2.55 for Huber. MLPs favor L2: 2.33 versus 2.38 for Huber and 2.83 for L1. The authors attribute this difference partly to MLPs having twice as many parameters, but no capacity-controlled experiment isolates that explanation.

Full attention self-grafting is poor immediately after initialization: Table 2 gives FID 16.78 with 8K distillation examples and no final adaptation. With 128K fine-tuning examples it recovers to 2.49, compared with the original 2.27; full MLP self-grafting reaches 2.54. The paragraph says 2.53 for attention, but Table 2 and subsequent plots use 2.49. Both source values are preserved, and the table is used for the numerical comparison here.

Stage 1 Stage 2 IS ↑\uparrow FID ↓\downarrow sFID ↓\downarrow Prec. ↑\uparrow Rec. ↑\uparrow
Baseline 278.20 2.27 4.60 0.83 0.57
MHA (Full Self-grafting)
Random Init. 1.66 289.23 154.00 0.00 0.00
0.63% – 117.68 16.78 13.69 0.60 0.61
0.63% 0.63% 148.56 11.26 11.10 0.66 0.60
0.63% 5.0% 270.39 2.70 5.46 0.81 0.57
0.63% 10.0% 287.81 2.49 4.71 0.83 0.56
MLP (Full Self-grafting)
Random Init. 1.27 314.72 204.99 0.00 0.00
0.63% 10.0% 277.72 2.54 4.52 0.83 0.57
Table 2: Full self-grafting (Stage 2) results (DiT-XL/2). We report results after replacing all 28 MHA and MLP operators using different amounts of training data. As we increase the training data from 0.63% (8k) to 10.0% (128k), FID improves consistently. Using only 10% of the training data, near-baseline performance is achieved: FID 2.49 for MHA and 2.54 for MLP.

3.4 Locality Analysis of Self-attention

The band-k metric measures attention mass within k positions on either side in the flattened token sequence. It averages 250 generated samples, diffusion timesteps and attention patterns; at k = 32, 15 of 28 layers put more than half their mass locally. Early layers are less local. This is sequence-index locality, not necessarily a uniform two-dimensional image neighborhood.

Local mass guides replacement but is not a proof that distant interactions are dispensable. Directly replacing all attention with sliding-window attention gives FID 53.9, and full replacement remains poor in the grafting experiments. Interleaving preserves some global attention and generally works better than indiscriminate substitution.

Original PDF figure S3.F2
Figure 2: Locality of self-attention in DiT-XL/2. We plot LkL_{k} values for all 28 MHA operators, averaged over timesteps and samples. At k=32k{=}32, 15 out of 28 layers exhibit values exceeding 0.5, indicating that several MHA operators model local interactions.
Equations 3
(Bk)i,j={1,if ​|i−j|≤k0,otherwise(B_{k})_{i,j}=\begin{cases}1,&\text{if }|i-j|\leq k\\ 0,&\text{otherwise}\end{cases}
Lk=1N​∑i,jAi,j​(Bk)i,jL_{k}=\frac{1}{N}\sum_{i,j}A_{i,j}(B_{k})_{i,j} (3)

4 Experiments I: Hybrid Architectures via Grafting

The ImageNet testbed compares attention alternatives and feed-forward alternatives under full or interleaved replacement. All configurations and numerical outcomes are retained in restored semantic tables.

4.1 Testbed and Experiment setup

Attention alternatives include sliding-window attention [26], Mamba-2 [28] and local gated convolutions. Hyena-X removes the long implicit convolution from a Hyena-style operator [52]; Hyena-Y instead removes the three featurizer convolutions and keeps a short inner filter. Both use dense projections plus local mixing. Hyena-X also supplies an MLP alternative that mixes channels.

Stage 1 independently fits replacement operators on 8K examples for 200 epochs, batch 64, with pre-extracted features. The main text specifies learning rate 10⁻⁴; Appendix Table B.1 says 10⁻³, an unresolved recipe discrepancy. Stage 2 usually uses 128K examples, 50K updates, batch 256 and learning rate 10⁻⁴ after 1K warmup steps. Reported standard runs take under ten hours on eight H100 GPUs, with larger ablations up to 24 hours. Feature extraction and all previous model pretraining should be considered when interpreting incremental compute savings.

Original PDF figure S4.F3
Figure 3: Our proposed Hyena-X and Hyena-Y, efficient local gated convolution operators used as drop-in replacements for MHA.
Operator Type: Which operator types are we replacing?
MHA, MLP
Efficient Alternative: What do we replace it with?
MHA Convolutions: Hyena-SE [24], Hyena-X/ Hyena-Y (Ours) KK=4, causal
Local Attention: Sliding Window Attention (SWA) [26, 27] ww=4, bidirectional
Linear Attention: Mamba-2 [28] dsd_{s}=64, EE=2
MLP Variable expansion ratio rr=3,6
Hyena-X (Ours) rr=2, KK=4, causal, mix channels
Layer Selection: In which layers is the operator replaced?
Full Replace the operator in all layers
Interleaved Replace the operator in a repeating pattern (e.g., every 2 or 3 out of 4)
Replacement Ratio: What percentage of operators are replaced?
50%, 75%, 100%
Table 3: Grafting testbed with configurations. This table defines the core design axes used in our study: operator type, efficient alternatives, layer selection strategy, and replacement ratio. For each alternative, we report configurations, including kernel size KK, window size ww, state size dsd_{s}, expand factor EE, and MLP expansion ratio rr. The baseline DiT-XL/2 operator uses H=16H{=}16 attention heads and MLP expansion ratio r=4r{=}4. Operator variants marked with (Ours) are proposed in this work.

4.2 Results and Insights

At 50% attention replacement, standard 10%-data results remain within about 0.5 FID of the 2.27 baseline: Hyena-SE 2.73, Hyena-X 2.74, Hyena-Y 2.72, sliding-window attention 2.67 and Mamba-2 2.65. Increasing replacement to 75% generally hurts; sliding-window attention is still 3.09. Full replacement fails badly. Expanding fine-tuning data to 20% improves the 50% variants, including Hyena-X to 2.61 and Mamba-2 to 2.55. The headline range combines selected settings rather than describing every standard configuration.

For Hyena-X, interleaving gives 2.74, selecting the most local layers 3.02, the least local 3.18 and the deepest layers 4.00. The locality score has some predictive value, yet interleaving beats it. Feed-forward width changes tolerate even full replacement: expansion ratio three reaches 2.66, while wider ratio-six variants retain strong quality but increase parameters and FLOPs. The convolutional MLP alternative degrades more at high replacement fractions.

Table 4 separates operator mixing from projections/featurizers. A roughly 50% reduction in the attention-mixing component is not a 50% reduction in whole-model work. At 50% Mamba-2 replacement, mixing FLOPs fall 37.59% while projection/featurizer FLOPs rise 77.89% and operator parameters rise 28.02%. Wider MLPs also deliberately spend more computation. These distinctions prevent treating every grafted variant as a smaller or faster model.

Original PDF figure S4.F4
Figure 4: Ablation studies. (a) Data scale: Increasing fine-tuning data from 10% to 20% improves FID. (b) Layer selection strategies: Interleaved replacement outperforms other heuristics.
Ratio IS ↑\uparrow FID ↓\downarrow sFID ↓\downarrow Prec. ↑\uparrow Rec. ↑\uparrow Δ\DeltaFLOPsop. ↓\downarrow Δ\DeltaFLOPsft. ↓\downarrow Δ\DeltaParam ↓\downarrow
Baseline – 278.20 2.27 4.60 0.83 0.57 — — —
MHA Grafting
Random Init. 100% 1.66 289.23 154.00 0.00 0.00 — — —
Self-grafting 100% 287.81 2.49 4.71 0.83 0.56 — — —
(KK=4)
50% 274.73 2.73 5.05 0.82 0.56 -49.52% +0.13% +0.22%
75% 231.15 3.62 6.04 0.81 0.54 -74.27% +0.20% +0.33%
100% ✗ ✗ ✗ ✗ ✗ -99.03% +0.26% +0.43%
(KK=4)
50% 273.30 2.74 5.03 0.83 0.56 -49.90% +0.13% +0.16%
75% 229.11 3.69 6.10 0.81 0.53 -74.85% +0.20% +0.24%
100% ✗ ✗ ✗ ✗ ✗ -99.81% +0.26% +0.33%
(KK=4)
50% 273.37 2.72 5.02 0.83 0.55 -49.52% 0.00% +0.05%
75% 228.99 3.66 5.95 0.81 0.53 -74.27% 0.00% +0.08%
100% ✗ ✗ ✗ ✗ ✗ -99.03% 0.00% +0.11%
(ww=4)
50% 280.62 2.67 4.90 0.83 0.56 -48.24% 0.00% 0.00%
75% 249.99 3.09 5.54 0.82 0.55 -72.36% 0.00% 0.00%
100% ✗ ✗ ✗ ✗ ✗ -96.48% 0.00% 0.00%
(dsd_{s}=64, EE=2)
50% 285.08 2.65 4.84 0.83 0.55 -37.59% +77.89% +28.02%
75% 257.66 3.02 5.48 0.82 0.53 -56.38% +116.83% +42.04%
100% ✗ ✗ ✗ ✗ ✗ -75.17% +155.77% +56.05%
MLP Grafting
Random Init. 100% 1.27 314.72 204.99 0.00 0.00 — — —
Self-grafting 100% 277.72 2.54 4.52 0.83 0.57 — — —
Exp. ratio ↓\downarrow
(rr=3)
50% 272.14 2.53 4.51 0.83 0.57 -12.50% 0.00% -12.50%
75% 279.72 2.61 4.61 0.83 0.56 -18.75% 0.00% -18.75%
100% 252.11 2.66 4.57 0.81 0.57 -25.00% 0.00% -25.00%
Exp. ratio ↑\uparrow
(rr=6)
50% 278.00 2.38 4.50 0.83 0.58 +25.00% 0.00% +25.00%
75% 277.94 2.37 4.48 0.82 0.58 +37.50% 0.00% +37.50%
100% 276.86 2.42 4.50 0.82 0.58 +50.00% 0.00% +50.00%
(rr=2,KK=4)
50% 265.60 2.64 4.66 0.83 0.56 +0.01% 0.00% +0.02%
75% 226.13 3.26 4.79 0.81 0.55 +0.02% 0.00% +0.03%
100% ✗ ✗ ✗ ✗ ✗ +0.02% 0.00% +0.03%
Table 4: Generation quality and efficiency metrics for MHA and MLP grafting. We report quality (IS, FID, sFID, Precision, Recall) and efficiency (Δ\DeltaFLOPs and Δ\DeltaParam) results. Baseline refers to DiT-XL/2. For each alternative, setups that maintain FID within 0.5 of the baseline and offer the largest FLOPs reduction (or smallest FLOPs increase) are highlighted. ✗ denotes setups with poor generation (FID > 50). Δ​FLOPs\Delta\text{FLOPs} and Δ​Param\Delta\text{Param} denote the percentage change in operator FLOPs and parameters, respectively. For MHA, total cost is split into Δ​FLOPsop.\Delta\text{FLOPs}_{\text{op.}} (softmax attention, gating) and Δ​FLOPsft.\Delta\text{FLOPs}_{\text{ft.}} (QKV/output projections, featurizers). We do not use this decomposition for MLP variants. Mamba-2 incurs higher Δ​FLOPsft.\Delta\text{FLOPs}_{\text{ft.}} due to additional projections. FLOP expressions are provided in Sec. F.1. Key result: Many interleaved designs achieve good quality generation (FID within 0.5 of baseline). All experiments use 10% training data and <<1% pretraining compute.
Takeaway 1: Grafting is effective for constructing efficient hybrid architectures with good generative quality under small compute budgets. Interleaved designs are particularly effective.

5 Experiments II: Grafting Text-to-Image Diffusion Transformers

PixArt-Σ generates 2048×2048 images with 16,384 tokens. Attention consumes over 62% of its measured generation latency, motivating Hyena-X replacement. A small synthetic dataset contains 12K uncurated image-text pairs, with prompts sampled from the model’s public 30K evaluation prompt set. Stage 1 uses 8K pairs and L1 distillation; Stage 2 uses rank-64 LoRA [29] on the full synthetic set. Generation uses a 20-step DPM solver [30].

The 50% variant replaces layers 8, 10, 12, 14, 16, 18 and 20–27, a mixture of interleaved and contiguous late layers. Its GenEval score falls from 49.75 to 47.78: 1.97 percentage points, approximately 3.96% relative. The abstract’s “less than 2%” should be read as a point difference, not a relative percentage. Counts and positions improve slightly while several color and object-related scores decline.

Single-forward-pass latency on one H100 at batch two falls from 235.46 to 164.58 ms, a 1.43× speed ratio or about 30% less latency. This table does not measure end-to-end prompt-to-image wall time including every sampling and decoding step. The 29% variant gives 48.04 GenEval and 194.95 ms. Failure examples reveal incorrect counts, malformed structure and inherited synthetic-data artifacts; LoRA capacity is a proposed explanation, not an isolated causal result.

Takeaway 2: We graft high-resolution text-to-image DiTs, constructing hybrid architectures with meaningful speedups and minimal quality drop.
Model Ratio Obj(1) Obj(2) Count Colors Pos Color Attr. Overall ↑\uparrow Latency (ms) ↓\downarrow
Baseline - 81.45 61.62 46.25 77.13 10.75 21.50 49.75 235.46
Hyena-X 29% 80.31 59.34 49.69 68.62 11.50 18.75 48.04 194.95 (1.21×\times)
Hyena-X 50% 80.00 57.07 48.13 70.74 11.25 19.50 47.78 164.58 (1.43×\times)
Table 5: GenEval results and latency for PixArt-Σ\Sigma and the grafted variants. The 50% grafted model achieves a 1.43× speedup while retaining strong text-image alignment (GenEval overall score: 47.78 vs. 49.75). Attribute-specific scores remain comparable across models. Latency is measured for a single forward pass on an Nvidia H100 (batch size=2).

6 Case Study: Converting Model Depth to Width via Grafting

The depth-to-width case sends the same input through two formerly sequential transformer blocks, then combines outputs with a linear projection. The 28-block model becomes 14 parallel pairs, retaining both branches and adding projection parameters. It is shallower in sequential computation, not a halving of the model’s parameters: the total rises from 675M to 712M.

Unlike random self-grafting, each branch begins from its corresponding pretrained block. L1 activation matching uses 8K examples; recovery uses 320K examples, 25% of ImageNet, for up to 230K steps. FID is 3.12 at 100K updates and 2.77 at 230K, compared with the original 2.27. The longer case has a different budget from the standard 50K-step hybrid experiments.

At 14 sequential stages, 2.77 beats TinyDiT with masked distillation at 2.86 and other reported shallow baselines, but the grafted model has 712M parameters versus 340M for the pruning baselines. Its spatial FID 4.87 is worse than TinyDiT’s 4.75. Reported 2× speed uses a fused implementation and one H100 forward pass at batch two; quality uses the exact illustrated implementation. This is a quality-versus-depth study, not a parameter- or pretraining-matched dominance claim.

Original PDF figure S6.F5
Figure 5: Convert model depth →\rightarrow width via grafting: (a) Two sequential transformer layers. (b) Rewiring in parallel via grafting (includes skip connections).
Takeaway 3: Grafting enables architectural restructuring at the transformer block level, allowing model depth to be traded for width.
Method Depth A.R Iters IS ↑\uparrow FID ↓\downarrow sFID ↓\downarrow Prec. ↑\uparrow Recall ↑\uparrow Speedup ↑\uparrow Params ↓\downarrow
DiT-L/2 [1] 24 42.7 1,000K 196.26 3.73 4.62 0.82 0.54 — 458M
U-ViT-L [33] 21 48.8 300K 221.29 3.44 6.58 0.83 0.52 — 287M
DiT-B/2 [1] 12 64.0 1000K 119.63 10.12 5.39 0.73 0.55 — 130M
BK-SDM [31] 14 82.3 100K 141.18 7.43 6.09 0.75 0.55 2×\times 340M
TinyDiT-D14 [32] 14 82.3 500K 198.85 3.92 5.69 0.78 0.58 2×\times 340M
TinyDiT-D14 w/ MKD [32] 14 82.3 500K 234.50 2.86 4.75 0.82 0.55 2×\times 340M
DiT-XL/2 [1] 28 41.4 7,000K 278.20 2.27 4.60 0.83 0.57 1×\times 675M
Grafting (Ours) 14 164.6 100K 231.91 3.12 4.71 0.82 0.55 2×\times^{\lx@paragraphsign} 712M
Grafting (Ours) 14 164.6 230K 251.77 2.77 4.87 0.82 0.56 2×\times^{\lx@paragraphsign} 712M
Table 6: Generative quality vs. model depth. We report generative quality metrics (IS, FID, sFID, Precision, and Recall). A.R. (Aspect Ratio) is defined as model width divided by depth (e.g., 1152/14 = 82.3). Parameters (Params) are reported in millions. For pruning and grafting setups, we report speedup with respect to DiT-XL/2 (depth=28). Off-the-shelf DiT-L/2, U-ViT-L, and DiT-B/2 scores, along with pruning baselines (BK-SDM, TinyDiT-D14, and TinyDiT-D14 w/ MKD), are sourced from  [32]. MKD refers to Masked Knowledge Distillation, a recovery method used in TinyDiT [32]. Speedup is measured for a single forward pass on an Nvidia H100 (batch size=2). More implementation details are provided in Sec. C. Key Result. Our grafted models achieve better generative quality at depth=14, surpassing baselines in FID, IS, Precision, and Recall.

7 Related Work

Related designs replace attention with state-space or linear operators, add experts, or search architectures from scratch. Earlier editing methods also transform pretrained language or diffusion models. Grafting’s broader testbed studies both attention and MLP replacement plus block-level restructuring; it does not establish that pretrained-model editing is itself a new idea.

8 Conclusion and Discussion

Grafting makes architectural experiments practical by reusing pretrained computation and recovering quality with modest additional training in the tested cases. Results are limited to attention/MLP edits in latent diffusion transformers; normalization, activations, VAEs, video and autoregressive transfer remain future work. Synthetic supervision can carry teacher errors and biases. Architecture rankings after grafting need not match rankings after training from scratch, and a pretrained checkpoint remains necessary.

Acknowledgments

Liquid AI sponsored compute. The authors acknowledge colleagues’ feedback on the manuscript.

Supplementary Material

The supplement reports limited repeat-run statistics, full training tables, modulation ablations, all generated examples and failure cases, operator definitions and component-level FLOP accounting. Original equations and every table are preserved.

Appendix A Standard Deviation of Experiments

Only two representative variants are repeated with seeds 0, 200 and 300. Hyena-Y attention has mean FID 2.73 ± 0.01, and higher-width MLP has 2.41 ± 0.01. These small reported standard deviations support repeatability of those particular runs; they do not quantify uncertainty for every grafting variant, GenEval result or speedup.

Setup IS FID sFID Precision Recall
MHA/ Hyena-Y 273.19 ±\pm 0.46 2.73 ±\pm 0.01 5.06 ±\pm 0.04 0.83 ±\pm 0.00 0.55 ±\pm 0.00
MLP/ higher width (r=6r=6) 277.91 ±\pm 0.95 2.41 ±\pm 0.01 4.48 ±\pm 0.02 0.82 ±\pm 0.00 0.58 ±\pm 0.00
Table A.1: Mean and standard deviation of IS, FID, sFID, Precision and Recall calculated for three runs with different random seeds (0,200,300).

Appendix B Hybrid Architecture Experiments: Additional details

The appendix supplies the common hybrid recipe, an alternative regression target and validation curves. These details qualify the initialization mechanism and expose a learning-rate discrepancy with the main text.

B.1 Experiment details and additional samples

Table B.1 gives Stage 1 AdamW with β = (0.9, 0.999), no weight decay, gradient clipping at ten, 200 epochs and batch 64. It lists learning rate 10⁻³, whereas the main experiment description says 10⁻⁴. Stage 2 gives AdamW, weight decay 5×10⁻⁵, batch 256, 50K updates and 1K-step warmup to constant 10⁻⁴. The seven-row gallery includes variants from different data/settings; its best sample-associated FIDs should not be silently substituted for all Table 4 rows.

Stage 1: Activation Distillation
Initial Learning Rate 1×10−31\times 10^{-3}
Weight Decay 0
Epochs 200
Batch Size 64
Clip Norm 10.0
Optimizer AdamW (b​e​t​a​s=(0.9,0.999)betas=(0.9,0.999))
Loss Function L1 (MHA), L2 (MLP)
Stage 2: Lightweight Finetuning
Initial Learning Rate 1×10−41\times 10^{-4}
Weight Decay 5×10−55\times 10^{-5}
Iterations 50,000 (100 epochs)
Batch Size 256
Optimizer AdamW (b​e​t​a​s=(0.9,0.999)betas=(0.9,0.999))
Scheduler Linear Warmup over 1K steps, then constant lr
Training Data 10% of ImageNet-1K (128k samples)
Table B.1: Experiment details for MHA/MLP grafting experiments using DiT-XL/2 (ImageNet-1K).

B.2 Modulated Regression Targets for MHA

A modulation-aware target multiplies attention output by its learned gate. With L2 regression, this worsens FID from 2.58 to 3.00; L1 gives 2.51 in either case. This supports using ordinary, unmodulated targets with L1 for attention in this setup. The paper does not systematically vary all scales or prove that gating alone explains sensitivity.

B.3 Validation Loss Curves for Self-grafting Experiments

Validation curves use held-out 8K examples and an L2 evaluation loss regardless of the training regression objective. L1 tends to generalize better for deeper attention layers; L2 works better across the displayed MLP layers. These are isolated-operator curves and complement, rather than replace, end-to-end generation evaluation.

Modulation-aware IS FID sFID Precision Recall
Baseline 278.20 2.27 4.60 0.83 0.57
L2 ✓ 246.17 3.00 7.11 0.79 0.58
✗ 269.31 2.58 5.75 0.82 0.58
L1 ✓ 272.86 2.51 5.29 0.82 0.57
✗ 273.03 2.51 5.48 0.83 0.58
Table B.2: Comparison of modulation-aware and standard regression targets for Stage 1. The modulation-aware setup includes the learned scalar (gate_msa) as a multiplicative factor in the regression target. L2 loss is sensitive to the amplified target scale and performs worse, while L1 loss remains robust and performs similarly in both cases. We adopt the standard formulation by default.
Refer to caption
Figure B.1: Additional samples generated by grafted DiT-XL/2 variants. Each row corresponds to a different hybrid. We report FID scores (lower is better, ImageNet-1K 256×256) for each hybrid. MHA variants (top 4 rows): Hyena-X (2.61), Hyena-Y (2.61), SWA (2.62), Mamba-2 (2.55). MLP variants (bottom 3 rows): Lower width (2.53), Higher width (2.38), Hyena-X MLP (2.64). These results highlight the flexibility of grafting in constructing high-quality hybrid architectures by replacing MHA or MLP operators.
Original PDF figure A2.F2
Figure B.2: Validation loss curves for MHA (top) and MLP (bottom) operator distillation showing the training dynamics for three regression objectives. As one can observe, L1 shows better generalization for MHA and L2 shows better generalization for MLP.

Appendix C Depth to Width Grafting Experiments: Additional details

Depth-to-width Stage 1 uses L1, AdamW with β = (0.9, 0.999), batch 64 and learning rate 10⁻⁴ for 200 epochs. Stage 2 uses β = (0.9, 0.95), zero weight decay, batch 256 and 230K updates, with 1K warmup then learning-rate halving every 75K steps. Its 320K examples are a quarter of ImageNet. The table’s reported aspect ratio 164.6 reflects the combined width of two branches divided by 14 stages; the caption’s 1152/14 = 82.3 example instead uses one branch width.

The authors explicitly expect speed gains to shrink at larger batch sizes because of the 712M parameter count. They also distinguish the exact rewiring used for quality from the simple fused version used for timing. No full image-generation latency curve across batch sizes is supplied.

Refer to caption
Figure C.1: Depth-to-width grafting samples. Samples from a DiT-XL/2 model in which every pair of transformer block is converted into a parallel block, effectively reducing depth by 2×2\times (FID=2.77).
Stage 1: Activation Distillation
Initial Learning Rate 1×10−41\times 10^{-4}
Regression Objective L1
Epochs 200
Batch Size 64
Optimizer AdamW (b​e​t​a​s=(0.9,0.999)betas=(0.9,0.999))
Stage 2: Lightweight Finetuning
Learning Rate 1×10−41\times 10^{-4}
Weight Decay 0
Iterations 230k
Batch Size 256
Optimizer AdamW (b​e​t​a​s=(0.9,0.95)betas=(0.9,0.95))
Scheduler Warmup over 1K steps, half every 75k steps
Training Data 25% of ImageNet-1K (320k)
Table C.1: Experiment details for depth-to-width grafting experiments using DiT-XL/2 (ImageNet-1K).

Appendix D Text-to-Image Generation Experiments: Additional details

The text-to-image supplement retains activation histograms, hyperparameters, successful outputs, prompt-alignment failures and low-quality synthetic supervision.

D.1 MHA activation plots

Input/output activation plots for layers 15, 17, 19, 21 and 23 show large ranges in PixArt-Σ attention. This motivates robust L1 regression but is not an independent comparison of every initialization loss on this model.

D.2 Experiment details

PixArt Stage 1 uses learning rate 10⁻⁴, weight decay 10⁻⁵, 100 epochs, batch 16 and AdamW; clipping is 0.1 for layers 20–27 and 0.01 elsewhere, with learning rate halved at epoch 50. Stage 2 uses 18K updates, effective batch 64 through accumulation, learning rate 10⁻⁵, no weight decay, 500-step warmup and rank-64 LoRA.

D.3 Generated samples and failure cases

The galleries show both strengths and limits of synthetic-data adaptation. Some teacher images satisfy a prompt that the grafted model then violates, including a four-donut request and a guitar-playing character. Other prompts already receive poor teacher outputs, so both models fail. Additional examples expose implausible geometry and physics in the unfiltered training images. These figures support curation as a practical concern; they do not isolate LoRA capacity from dataset quality as the cause of every artifact.

Refer to caption
Figure D.1: Similar to DiT-XL/2, we observe high activation variance in PixArt-Σ\Sigma MHA operators. We show input (left) and output (right) activation values corresponding to five representative layers (15, 17, 19, 21, 23) in PixArt-Σ\Sigma.
Refer to caption
Figure D.2: Additional 2048×\times2048 samples generated using our grafted PixArt-Σ\Sigma model.
Refer to caption
Figure D.3: Text-to-image generation failure cases. Each pair shows outputs from PixArt-Σ\Sigma (left) and the grafted model (right) for the same prompt. In the top row, the prompt specifies four donuts with unique toppings and a full-body anime-style character playing classical guitar. The grafted outputs deviate from these prompts—showing incorrect object counts (e.g., five donuts) and degraded structure (e.g., distorted hands), reflecting text-image misalignment and visual artifacts introduced during grlafting. In the bottom row, the supervision itself is poor: prompts such as a ‘dog chasing its tail in UHD stippling style’ and ‘a black-and-white street photo’ are not faithfully captured by either model. These examples highlight challenges arising both from LORA finetuning and low-quality synthetic data.
Refer to caption
Figure D.4: Examples of low-quality samples generated by PixArt-Σ\Sigma used for grafting. These images contain unrealistic features, inconsistent physics, and visual artifacts. Their presence in the grafting dataset can degrade generation quality of grafted models, highlighting the importance of data curation when using synthetic data.
Stage 1: Activation Distillation
Initial Learning Rate 1×10−41\times 10^{-4}
Weight Decay 1×10−51\times 10^{-5}
Epochs 100
Clip Norm Value 0.1 (Layers 20-27), 0.01 (Other layers)
Batch Size 16
Optimizer AdamW
Scheduler Half lr at epochs = 50
Stage 2: Lightweight Finetuning
Initial Learning Rate 1×10−51\times 10^{-5}
Weight Decay 0
Iterations 18k
Batch Size 64 (with gradient accumulation)
Optimizer AdamW
Scheduler linear warmup (500 steps), then constant lr
LoRA rank 64
Table D.1: Experiment details for PixArt-Σ\Sigma grafting experiments.

Appendix E Hyena-X and Hyena-Y operators: Additional details

The generic Hyena equations combine dense feature projections, short convolutions and multiplicative gates with an inner convolution. Hyena-X sets the inner operator G to identity while retaining featurizer convolutions. Hyena-Y removes those three convolutions and replaces the inner long convolution with a short explicit filter. These are local operators whose overall effectiveness depends on the remaining global-attention layers and fine-tuning.

Equation
qsc\displaystyle q_{s}^{c} =∑s′Ts​s′c​∑c′xs′c′​Wc′​c\displaystyle=\sum_{s^{\prime}}T^{c}_{ss^{\prime}}\sum_{c^{\prime}}x_{s^{\prime}}^{c^{\prime}}W^{c^{\prime}c}
ksc\displaystyle k_{s}^{c} =∑s′Hs​s′c​∑c′xs′c′​Uc′​c\displaystyle=\sum_{s^{\prime}}H^{c}_{ss^{\prime}}\sum_{c^{\prime}}x_{s^{\prime}}^{c^{\prime}}U^{c^{\prime}c}
vsc\displaystyle v_{s}^{c} =∑s′Ks​s′c​∑c′xs′c′​Pc′​c\displaystyle=\sum_{s^{\prime}}K^{c}_{ss^{\prime}}\sum_{c^{\prime}}x_{s^{\prime}}^{c^{\prime}}P^{c^{\prime}c}
ysc\displaystyle y_{s}^{c} =∑c′∑s′(qsc′​Gs​s′c′​ks′c′​vs′c′)​Mc′​c\displaystyle=\sum_{c^{\prime}}\sum_{s^{\prime}}(q_{s}^{c^{\prime}}G^{c^{\prime}}_{ss^{\prime}}k_{s^{\prime}}^{c^{\prime}}v_{s^{\prime}}^{c^{\prime}})M^{c^{\prime}c}

Appendix F FLOP calculation

The FLOP appendix counts dense projections, mixing, filtering, gates and output projections separately. It defines sequence length L, hidden width D, head count H, convolution width K, attention half-window w, MLP expansion r and Mamba expansion/state size. All original component formulas are retained below. FLOPs are analytical operation counts, not measured latency, and should not be equated with the speed tables.

F.1 MHA

Multi-head attention: quadratic attention mixing plus dense input/output projections.

  • Input projections (Q, K, V): 6​L​D26LD^{2}
  • Softmax attention: 4​L2​D+2​H​L24L^{2}D+2HL^{2}
  • Output projection: 2​L​D22LD^{2}

F.2 SWA

Sliding-window attention: the mixing window contains 2w + 1 token positions.

  • Input projections (Q, K, V): 6​L​D26LD^{2}
  • Sliding window attention (Bidirectional): 4​L​(2​w+1)​D+2​H​L​(2​w+1)4L(2w+1)D+2HL(2w+1)
  • Output projection: 2​L​D22LD^{2}

F.3 Hyena-SE

Hyena-SE: dense projections plus three featurizer convolutions, an inner filter and gates.

  • Input projections: 6​L​D26LD^{2}
  • Featurizer: 3​L​D​K×23LDK\times 2
  • Inner filter convolution: L​D​K×2LDK\times 2
  • gates: L​D×2LD\times 2
  • Output projection: 2​L​D22LD^{2}

F.4 Hyena-X

Hyena-X: retains featurizer convolutions and gates, without the inner filter.

  • Input projections: 6​L​D26LD^{2}
  • Featurizer: 3​L​D​K×23LDK\times 2
  • gates: L​D×2LD\times 2
  • Output projection: 2​L​D22LD^{2}

F.5 Hyena-Y

Hyena-Y: retains a short inner filter and gates, without featurizer convolutions.

  • Input projections: 6​L​D26LD^{2}
  • Inner filter convolution: L​D​K×2LDK\times 2
  • gates: L​D×2LD\times 2
  • Output projection: 2​L​D22LD^{2}

F.6 Hyena-X (MLP)

Hyena-X as an MLP: projected channel mixing with expansion ratio r.

  • Dense input projections: 6​L​D2​r6LD^{2}r
  • Featurizer: 3​L​D​K×23LDK\times 2
  • Gates: L​D×2LD\times 2
  • Dense output projections: 2​L​D2​r2LD^{2}r

F.7 Mamba-2

Mamba-2: expanded projections, short convolution, featurization, state scan and output projection.

  • Projections: 8​L​D2​E8LD^{2}E
  • Short convolution: 6​L​D​E6LDE
  • Featurization: 2​L​D​E​(1+2​dstate)+2​L​D​E2LDE(1+2d_{\text{state}})+2LDE
  • Associative scan: 2​L​D​E​dstate2LDEd_{\text{state}}
  • Output layer: 2​L​D2​E2LD^{2}E
Symbol Description
LL Sequence length
DD Hidden dimension
HH Number of attention heads
KK Kernel size for convolutions
ww Window size for sliding window attention
rr MLP expansion ratio
EE Expansion factor in Mamba-2
dstated_{\text{state}} State size in Mamba-2
Table F.1: Notation for FLOP calculation.
Concept demonstration

Open image ↗