Literature
Display
← All papers

Text-to-Image Generative Modeling Without Language Pretraining

Keshigeyan Chandrasegaran · Yue Zhao · Armin W. Thomas · Ehsan Adeli · Jiajun Wu · Juan Carlos Niebles · Li Fei-Fei · Michael Poli

Preprint posted · arXiv (preprint) ↗

Computer vision9 min text · CondensedOriginal paper ↗
Contents

Overview

Halcyon learns text-conditioned image generation directly from paired captions and images. Controlled experiments test whether a separately pretrained language model improves that task under the studied budget.

Complete original figure 1 including all source labels and values.
Figure 1: Text-to-image generation without internet-scale language pretraining. Left: Generation quality on GPIC (Chandrasegaran et al., 2026) (gFD, log scale) vs. total pretraining FLOPs required to build each T2I system, including language model, image tokenizer, and image generator pretraining. Each ⋆\star marks total pretraining FLOPs and gFD, while the donut breaks down language and vision compute. Halcyon (Ours), trained entirely from scratch, achieves lower gFD with substantially lower total pretraining compute compared to: (i) LlamaGen (Sun et al., 2024), late-fusion autoregressive models with 0.78–0.90B image generators and pretrained FLAN-T5 encoders (0.11–1.22B); (ii) JiT-T2I (Li and He, 2026; Ma et al., 2026), late-fusion diffusion models with a 1.10B image generator and pretrained Qwen3-1.7B; and (iii) Qwen3-0.6B-CPT, an autoregressive model continually pretrained on image–text pairs, initialized from a pretrained Qwen3-0.6B language model (Yang et al., 2025a). Right: 256×256256\times 256 samples generated by Halcyon-4B with prompts shown for two samples. Best viewed in color.

At a glance

Early-fusion autoregressive models trained on one hundred million GPIC image/text pairs reach gFD 48.0 at 0.52B parameters, 35.3 at 1.6B, and 28.4 at 4B. The larger variants outperform the tested late-fusion baselines. External text-only model pretraining provides no measured benefit in the controlled small-model experiments. Replacing unconditional image-only examples with image-caption sequences adds captioning at similar generation quality. The headline compute ratios count full upstream language-model pretraining; they are not incremental finetuning or inference speedups.

1 Introduction

Most recent image generators use a separately pretrained text encoder or initialize/train an early-fusion model with text-only data. This study asks how much that upstream language-model training contributes to image generation. Its definition of “language pretraining” excludes image-paired captions. The generator learns from GPIC pairs, but the text tokenizer also uses OpenWebText, and image-tokenizer training includes perceptual supervision. Thus the finding is about learned language-model pretraining, not removing every external text resource or learned perceptual component.

Complete original figure 2 including all source labels and values.
Figure 2: Language vs. vision pretraining tokens. Language tokens count text-only pretraining. Emu3 and Tuna-2 values are estimated.

2 Related Work

Earlier systems such as DALL-E, CogView, and GLIDE already learned conditioning from paired data, so the contribution is a controlled reassessment rather than the first encoder-free image generator. The paper compares language/vision token budgets and representative autoregressive/diffusion baselines. Those families differ in representation and optimization; the experiments are designed to study language pretraining, not establish a universal ranking of autoregression versus diffusion.

3 Approach

A decoder-only transformer predicts a joint sequence of caption and discrete image tokens. The main models, text tokenizer, and image-tokenizer weights are trained from scratch, with the tokenizer’s separate training data and losses specified below. Early fusion allows text representations to be learned jointly with the image task, although the experiments do not isolate joint training as the sole cause of improvement.

3.1 Tokenization

A Cosmos-style finite-scalar-quantized image tokenizer maps each 256×256 image to a 32×32 raster sequence of 1,024 tokens from 64,000 codes. The byte-level BPE tokenizer learns 16,384 text entries from GPIC captions plus OpenWebText. A tied embedding/output vocabulary totals 80,640 entries including 256 specials. The original token markers are [SOV] and [EOV] for image boundaries, [EOS] for sequence end, and [PAD] for padding.

3.2 Sequence Format and Training Objective

Text-only sequences are packed; image-containing sequences contain one pair and are padded to fixed length. Text-to-image predicts image tokens after a caption, while image-to-text predicts the image first and then its caption. Image-only and image-to-text share the unconditional image prefix, allowing both to train the classifier-free-guidance branch. The original guidance and weighted next-token equations are preserved together: image loss weight is 0.5, padding is excluded, and special tokens count with text. Guidance combines conditional and unconditional image-token logits only.

Complete original figure 3 including all source labels and values.
Figure 3: Sequence formats. Text-only pretraining tokens, caption tokens, and image tokens occupy one joint vocabulary alongside special tokens. The top two formats are unimodal, while the bottom two represent paired image-caption examples. Rows I\mathrm{I} and I2T\mathrm{I2T} share the same [SOV] 𝐱\mathbf{x} [EOV] prefix, which is also obtained when dropping the caption from a T2I sequence used for classifier-free guidance. Image-caption sequences are padded to a fixed length, with [PAD] positions excluded from the training loss. Box widths are schematic and not proportional to token counts.
Equation

ℓcfg=ℓu+γ⁡(ℓc−ℓu)\bm{\ell}_{\mathrm{cfg}}=\bm{\ell}_{\mathrm{u}}+\gamma(\bm{\ell}_{\mathrm{c}}-\bm{\ell}_{\mathrm{u}})

ℒ=(ℒtext+λimg​ℒimg)/M\mathcal{L}=(\mathcal{L}_{\mathrm{text}}+\lambda_{\mathrm{img}}\mathcal{L}_{\mathrm{img}})/M

ℒtext=−∑i∈𝒯logpθ(si∣s<i)\mathcal{L}_{\mathrm{text}}=-\sum_{i\in\mathcal{T}}\log p_{\theta}(s_{i}\mid s_{<i})

ℒimg=−∑i∈ℐlogpθ(si∣s<i)\mathcal{L}_{\mathrm{img}}=-\sum_{i\in\mathcal{I}}\log p_{\theta}(s_{i}\mid s_{<i})

M=|𝒯|+|ℐ|M=|\mathcal{T}|+|\mathcal{I}|

λimg=0.5\lambda_{\mathrm{img}}=0.5

3.3 Pretraining Data Composition and Scheduling

Five recipes separate paired-data learning from additional supervision: pure T2I; text-only then T2I; I2T then T2I; interleaved I2T/T2I; and replacing caption-dropped examples with I2T while disabling dropout. The last substitution preserves conditional/unconditional image-token counts rather than allocating a new fraction away from conditional generation. Staged language experiments usually redistribute a fixed budget, with a separate extra-compute control preserving a full T2I epoch.

Complete original figure 4 including all source labels and values.
Figure 4: Pretraining data composition and scheduling described in Section 3.3. The four token types are:  image tokens conditioned on text (ℐ|𝒯\mathcal{I}\mid\mathcal{T});  image tokens without text conditioning (ℐ|∅\mathcal{I}\mid\varnothing);  text tokens conditioned on images (𝒯|ℐ\mathcal{T}\mid\mathcal{I}); and  text-only tokens (𝒯|∅\mathcal{T}\mid\varnothing). Vertical boundaries denote training phases, while horizontal bands show the token composition within each phase. Bands represent interleaved examples. Band height and width are schematic.

4 Experiments

Experiments measure tokenizer fidelity, generated-image distribution/composition, language-pretraining interventions, captioning, and compute. Evaluation and training variation are reported separately in the appendices.

4.1 Experimental Setup

The main generator trains on GPIC at 256×256, with sizes 0.1B–4B and 1,280-token sequences. Caption dropout is normally 0.1. The image tokenizer trains 1.5 epochs, adding adversarial loss after one epoch; text-tokenizer training uses 21.3 GB GPIC captions and 11.9 GB OpenWebText, which is not supplied as generative-model language pretraining. The tokenizer improves reconstruction gFD from Cosmos’s 37.93 to 18.86, with real-versus-real reference 7.59, but small text remains difficult.

LlamaGen controls vary pretrained T5 size and image tokenization. JiT controls add perceptual loss and change patch size, following PixelGen. A separate matched-architecture experiment compares pretrained versus randomly initialized Qwen3. Main Halcyon and baselines have different optimizer schedules detailed in their tables, so not every cross-family difference is a single-variable intervention.

GPIC uses Test-50K distribution metrics; guidance is chosen on Validation-50K, then reused for GenEval and DPG-Bench. GenEval prompts are rephrased following PixelGen; DPG uses original prompts. Captioning is assessed by a Qwen3-VL reranker rather than a human study. Separate token likelihoods distinguish conditional image/text tasks from their unconditional counterparts.

Table 1: Reconstruction results on GPIC Test-50K (256×256256\times 256). Our tokenizer, trained from scratch on GPIC, outperforms Cosmos-DI-8x8 on all metrics.
Tokenizer rFD ↓\downarrow PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow
LlamaGen-VQ-16x16 129.87 20.18 0.560 0.146
Cosmos-DI-8x8 37.93 25.38 0.776 0.153
Halcyon 18.86 25.61 0.792 0.062
Real-vs-Real 7.59 – – –
Table 2: Text tokenizer comparison on GPIC test captions. Higher bytes/token and lower tokens/caption indicate better compression.
Tokenizer Vocab. Bytes/token ↑\uparrow Tok./caption ↓\downarrow
FLAN-T5 32,100 4.152 51.1
Qwen3 151,669 4.763 44.6
Halcyon 16,384 4.774 44.5
Table 3: Experiment configurations for late-fusion methods. LlamaGen and JiT-T2I configurations, varying the language model, image sequence length, and perceptual supervision. LlamaGen and Halcyon are autoregressive; JiT is diffusion-based. Pretraining tokens report the total language and vision tokens used during training; Halcyon uses no pretrained language model, and all its language tokens come from GPIC captions. We report encoder parameters for FLAN-T5.

Complete original table 3 including all source labels and values.

4.2 Results: Comparison with Late-fusion T2I Methods

Changing LlamaGen’s tokenizer and token count improves gFD 147.0→63.3. Adding JiT perceptual loss improves 67.9→48.9, and smaller patches further improve it to 41.0. Reducing T5 encoder size from 1.22B to 0.11B changes LlamaGen gFD only 63.3→66.7. These controls show that image representation and supervision substantially affect the comparison.

Halcyon-0.52B reaches gFD 48.02, GenEval 44.91, and DPG 76.84; the 1.6B version reaches 35.28/53.48/77.90 and the 4B version 28.43/56.39/77.90. The smallest model is much weaker at gFD 144.16. Headline 18.2-fold and 275-fold compute reductions charge each comparator its entire upstream language-model pretraining. They describe rebuilding the whole system, not the additional cost of reusing an already available text encoder.

Table 4: Comparison of Halcyon with late-fusion T2I methods on GPIC. Without language pretraining, Halcyon achieves stronger T2I generation than the late-fusion baselines: Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD), while Halcyon-1.60B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD). Vision parameter counts refer to the generative models, excluding image tokenizers (72M for LlamaGen and 136M for Halcyon). The CFG scale for each model is reported in Appendix Table 16.

Complete original table 4 including all source labels and values.

Table 5: Comparison of Halcyon with late-fusion T2I methods on GenEval and DPG-Bench. We also report GPIC gFD for comparison. Per-category breakdowns of GenEval and DPG-Bench are provided in Sec. C.1.

Complete original table 5 including all source labels and values.

Complete original takeaway including all source labels and values.
Refer to caption
Figure 5: Comparison of generated images. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon models from 0.1B to 4B parameters. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K. Additional samples are provided in Appendix D.

4.3 Results: Effect of Language Pretraining on T2I generation

Allocating 10%, 20%, or 50% of the 0.52B training budget to text-only learning worsens gFD from 48.02 to 51.56, 52.76, or 65.13. Keeping a full paired-data epoch after extra language training gives 49.22 at 1.5× budget. With matched Qwen architecture/vocabulary, scratch gives 46.61 versus pretrained 47.81; lowering the pretrained model’s learning rate worsens it to 63.62. These tested recipes show no measured gain, without proving that all larger-scale language pretraining or other datasets are useless.

Table 6: Effect of language pretraining on Halcyon-0.52B. Under a fixed pretraining budget, we measure the impact of allocating compute to external text-only pretraining on T2I generation. Each language-pretraining stage exceeds the Chinchilla reference of 20 text tokens per parameter. The † configuration preserves a full T2I epoch after additional language pretraining (1.5×1.5\times total pretraining compute), but does not improve gFD. Token counts exclude padding.
Text tokens (B) Image tokens (B) NLL ↓\downarrow
Configuration α\alpha 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} gFD ↓\downarrow
Halcyon-0.52B – 4.05 0 10.24 92.16 1.433 18.414 8.428 8.366 48.02
Staged: Language →\rightarrow T2I 0.10 16.44 0 9.22 82.94 1.447 19.098 8.442 8.382 51.56
0.20 28.83 0 8.19 73.73 1.451 17.197 8.445 8.388 52.76
0.50 65.80 0 5.12 46.08 1.454 22.850 8.505 8.451 65.13
Staged: Language →\rightarrow T2I† 0.33 68.03 0 10.24 92.16 1.445 18.809 8.428 8.371 49.22
Table 7: Continual pretraining from a language model. Qwen3-0.6B-CPT is initialized from Qwen3-0.6B-Base (pretrained on 36T text tokens), while Qwen3-0.6B is randomly initialized. Both share the same Transformer architecture as Halcyon-0.52B but use a larger vocabulary (216k vs. 80,640 for Halcyon). We additionally evaluate continual pretraining with a lower peak learning rate (∗: 1×10−41\times 10^{-4} vs. 7.5×10−47.5\times 10^{-4}) to test whether aggressive weight updates diminish the benefits of pretrained language weights. Since vocabularies differ, we report bits per byte (bpb) for both text and images. Neither configuration improves T2I generation over random initialization. All models are trained for one epoch on GPIC and evaluated with CFG 6.0. LP denotes language pretraining tokens. Token counts exclude padding.
Text tokens (B) Image tokens (B) Text bpb ↓\downarrow Image bpb (×10−2\times 10^{-2}) ↓\downarrow
Model LP 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} gFD ↓\downarrow
Halcyon-0.52B 0 4.05 0 10.24 92.16 0.433 5.565 6.333 6.286 48.02
Qwen3-0.6B 0 4.05 0 10.24 92.16 0.434 5.436 6.324 6.278 46.61
Qwen3-0.6B-CPT 36T 4.05 0 10.24 92.16 0.430 4.974 6.326 6.283 47.81
Qwen3-0.6B-CPT ∗ 36T 4.05 0 10.24 92.16 0.424 6.743 6.409 6.367 63.62

4.4 Results: Effect of I2T Supervision on T2I generation

Spending a separate 10% budget on I2T gives similar image quality, while 50% reduces conditional image training and worsens it—especially interleaving, gFD 75.97 versus 54.22 staged. Interleaving nevertheless learns captioning better; staged I2T followed by T2I loses that capability by the reported text likelihood.

Replacing the baseline’s 10% caption-dropped examples with I2T preserves image-token counts and improves caption likelihood while keeping generation broadly comparable. At 0.52B, conditional text NLL falls 18.414→1.102; at 4B the caption-alignment score reaches 0.559, still below ground-truth captions at 0.891. Generation changes are mixed across metrics/sizes, so this is a useful additional capability rather than a universal image-quality improvement.

Table 8: Effect of I2T supervision and scheduling on T2I generation for Halcyon-0.52B. Under a fixed pretraining budget, staged and mixed training use identical token counts at each I2T allocation, differing only in scheduling. At 10%, both schedules preserve comparable T2I generation, while at 50%, staged training achieves substantially lower gFD. However, only mixing substantially reduces captioning NLL, namely NLL⁡(𝒯∣ℐ)\mathrm{NLL}(\mathcal{T}\mid\mathcal{I}). NLL and gFD are evaluated on GPIC Test-50K; token counts exclude padding.
Text tokens (B) Image tokens (B) NLL ↓\downarrow
Configuration β\beta (or η\eta) 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} gFD ↓\downarrow
Baseline – 4.05 0 10.24 92.16 1.433 18.414 8.428 8.366 48.02
Staged: I2T →\rightarrow T2I β=0.1\beta=0.1 3.65 0.45 19.46 82.94 1.440 18.415 8.426 8.366 48.35
β=0.5\beta=0.5 2.03 2.25 56.32 46.08 1.435 20.437 8.421 8.370 54.22
Mixing: T2I + I2T η=0.1\eta=0.1 3.65 0.45 19.46 82.94 1.443 1.125 8.414 8.362 48.18
η=0.5\eta=0.5 2.03 2.25 56.32 46.08 1.475 1.009 8.453 8.417 75.97
Table 9: Replacing caption dropout with I2T supervision across model sizes. Replacing caption-dropped T2I examples with I2T preserves the pretraining budget and image-token counts, enabling image captioning while maintaining comparable T2I generation. We report mean Qwen3-VL-Reranker-8B alignment scores over 50K generated image–caption pairs from GPIC Test-50K, using the image as the query and caption as the document. GPIC Test-50k image-caption pairs obtain a mean alignment score of 0.891.
NLL ↓\downarrow T2I generation Captioning
Size η\eta pdropp_{\mathrm{drop}} 𝒯|∅\mathcal{T}\mid\varnothing 𝒯|ℐ\mathcal{T}\mid\mathcal{I} ℐ|∅\mathcal{I}\mid\varnothing ℐ|𝒯\mathcal{I}\mid\mathcal{T} gFD ↓\downarrow GenEval ↑\uparrow DPG-Bench ↑\uparrow Alignment ↑\uparrow
0.10B – 0.1 1.601 19.066 8.774 8.727 144.16 27.31 66.36 –
0.1 0 1.598 1.410 8.771 8.724 144.14 27.66 68.89 0.090
0.25B – 0.1 1.497 21.472 8.566 8.509 72.55 35.72 72.66 –
0.1 0 1.491 1.218 8.559 8.503 70.44 36.54 73.07 0.207
0.52B – 0.1 1.433 18.414 8.428 8.366 48.02 44.91 76.84 –
0.1 0 1.435 1.102 8.419 8.358 46.48 44.60 76.14 0.315
1.6B – 0.1 1.370 20.426 8.303 8.235 35.28 53.48 77.90 –
0.1 0 1.372 0.977 8.292 8.228 35.91 53.51 78.21 0.467
4B – 0.1 1.338 16.738 8.193 8.122 28.43 56.39 77.90 –
0.1 0 1.325 0.890 8.196 8.126 29.30 55.33 78.61 0.559
Complete original figure 6 including all source labels and values.
Figure 6: Image captioning examples from Halcyon-4B. For each example, the ground-truth caption is shown above the input image, and the generated caption is shown below. The model learns image captioning by replacing caption-dropped T2I examples with I2T supervision, using only 10% of the pretraining budget for captioning.

5 Discussion and Conclusion

Larger models lower image/text likelihood and generation distance. The validation-optimal guidance scale declines from 7.5 to 4.5, but guidance remains beneficial. Medium/long caption groups yield lower gFD than short captions; this comparison does not by itself isolate a causal benefit of appending more words to otherwise identical prompts.

The scope is one dataset and 256×256 images. Controlled composition tests use 0.52B and pretrained-initialization tests Qwen3-0.6B; larger budgets, other datasets, diffusion-style early fusion, and video remain open. Text rendering and some compositional categories remain weak. Full pretraining accounting makes upstream costs visible, but does not establish deployment throughput or the marginal cost of adopting an existing checkpoint.

Complete original figure 7 including all source labels and values.
Figure 7: Scaling behavior of Halcyon across model sizes. Validation negative log-likelihood (NLL) for image tokens (left) and text tokens (right) throughout pretraining on GPIC. As model size increases from 0.1B to 4B, both image and text NLL consistently decrease, with larger models achieving lower NLL throughout training. All models are pretrained from scratch under the same training schedule.
Original figure
Figure 8: CFG sweep on GPIC validation. The optimal scale drops from 7.5 (0.1B) to 4.5 (4B) as gFD improves from 142.7 to 28.4; these scales are used on Test-50K.
Table 10: Effect of caption length on T2I generation. We report gFD (↓\downarrow) for Halcyon across model sizes using 50K examples with short, medium, and long captions from GPIC Test-1M. Test-50K results are included for comparison. Across all model sizes, medium and long captions achieve lower gFD than short captions, with comparable performance between medium and long captions from 0.52B onward. Generation quality consistently improves with model size across all caption lengths.
Model Test-50K Short Medium Long
0.1B 144.16 149.03 139.27 147.85
0.25B 72.55 77.18 70.14 72.53
0.52B 48.02 51.99 46.12 46.83
1.6B 35.28 38.59 34.21 34.33
4B 28.43 31.48 27.91 27.77

AI Use Statement

The authors report generative-AI assistance with implementation, code, literature search, manuscript drafting/editing, and readability. They state that hypotheses, methodology, and result interpretation were not delegated, and that generated code and suggested references were checked. The original disclosure is retained.

Ethics Statement

The authors acknowledge realistic-image misuse risks and possible GPIC biases, with a cheaper training recipe lowering barriers for beneficial and harmful uses alike. No new human-subject study or dataset collection is reported; the paper does not present a comprehensive bias or misuse evaluation.

Reproducibility Statement

The paper supplies architecture, tokenization, objective, data schedules, baseline changes, evaluation, compute accounting, and detailed configurations. It states that checkpoints and GPIC are publicly available. Reproduction still depends on following the distinct model-specific schedules and guidance selections.

Acknowledgments

Radical Numerics supplied compute; the authors thank colleagues for feedback, technical discussion, and infrastructure support. The original names remain in the source acknowledgment.

Appendix

The appendix retains all model/tokenizer configurations, sampling settings, compute breakdowns, per-category results, seed studies, and qualitative sample grids.

Appendix A Training Details

Halcyon uses a Qwen3-style decoder with tied embeddings, 97,632 updates, batch 1,024, maximum 254 caption tokens, image loss weight 0.5, and warmup/stable/decay learning rates. The tokenizer has 136 million parameters and trains 242,172 updates on eight H100s. Baseline tables retain their own batch sizes, 390,600 updates, samplers, and perceptual-loss settings. Compute Table 15 assigns 1.45×10²⁰ tokenizer FLOPs to modified LlamaGen despite using the Halcyon tokenizer, while Halcyon’s tokenizer entry is 1.31×10²⁰; the reported accounting is retained without silently reconciling that discrepancy.

Table 11: Halcyon configurations.
0.1B 0.25B 0.52B 1.6B 4B
architecture
backbone Qwen3 (Yang et al., 2025a), decoder-only
layers 14 21 28 28 36
hidden dim 512 768 1024 2048 2560
FFN dim 1536 2304 3072 6144 9728
query / KV heads 8 / 4 12 / 6 16 / 8 16 / 8 32 / 8
head dim 128
non-embedding params 55.1M 185.8M 440.4M 1409.3M 3634M
embedding params 41.3M 61.9M 82.6M 165.2M 206.4M
total params 96.4M 247.7M 523.0M 1574.5M 3840M
vocabulary 80,640 (16,384 text + 64,000 image + 256 special)
tied embeddings yes
image size 256
image tokens 1,024
max caption tokens 254
sequence length 1,280
training
training steps 97,632 (1 epoch on GPIC)
global batch size 1,024
warmup steps 1,000
peak learning rate 1e-3 7.5e-4 7.5e-4 7.5e-4 7.5e-4
lr schedule linear warmup, constant to step 49,595, cooldown to peak/10\text{peak}/10
optimizer AdamW, β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
weight decay 0.1
gradient clipping 1.0
init std 0.02
image loss weight λimg\lambda_{\mathrm{img}} 0.5
caption dropout pdropp_{\mathrm{drop}} 0.1
precision bf16
sampling
cfg 7.5 6.0 6.0 5.0 4.5
Table 12: JiT-T2I configurations.
JiT-T2I-XXL/16 JiT-T2I-XXL/16++ JiT-T2I-XXL/8++
language model (frozen)
text encoder Qwen3-1.7B
feature dim 2048
max caption tokens 300
image generator
backbone JiT-T2I (pixel space, no image tokenizer)
blocks (image / text) 16 / 4
hidden dim 1536
attention heads 24
patch bottleneck dim 256
params 1.10B
patch size 16 16 8
image tokens 256 256 1,024
training
objective flow matching, linear schedule
timestep sampling logit-normal (μ=−0.8\mu{=}{-}0.8, σ=0.8\sigma{=}0.8), tϵ=0.05t_{\epsilon}{=}0.05
LPIPS weight – 0.1 0.1
LPIPS timestep gate – t≥0.3t\geq 0.3 t≥0.3t\geq 0.3
training steps 390,600
global batch size 256
peak learning rate 1e-4
lr schedule 4,000 warmup, constant, 4,000 cooldown
optimizer AdamW, β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
weight decay 0
gradient clipping 1.0
EMA decay 0.9999
caption dropout 0.1
precision bf16
sampling
sampler Euler (ODE), 50 steps, EMA weights
CFG scale 6.00 5.00 4.00
Table 13: LlamaGen configurations.
LlamaGen-XL LlamaGen-XL/8 LlamaGen-XL-T5-Base/8
language model (frozen)
text encoder FLAN-T5-XL FLAN-T5-XL FLAN-T5-Base
encoder params 1.22B 1.22B 0.11B
feature dim 2048 2048 768
max caption tokens 300
image tokenizer
tokenizer VQ-16 (pretrained) Ours (GPIC-trained)
spatial downsampling 16×\times 8×\times
codebook size 16,384 64,000
image tokens 256 1,024
image generator
backbone GPT-XL (Llama-style, decoder-only)
layers / hidden dim / heads 36 / 1280 / 20
FFN dim 3584 (SwiGLU)
params 0.78B 0.90B 0.90B
sequence length 556 1,324 1,324
training
training steps 390,600 390,600 390,600
global batch size 256
peak learning rate 1e-4
lr schedule 3,906 warmup, then constant
optimizer AdamW, β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
weight decay 0.05
gradient clipping 1.0
dropout / token dropout 0.1 / 0.1
caption dropout 0.1
precision bf16
sampling
CFG scale 6.5 6.5 5.5
temperature / top-kk / top-pp 1.0 / 1000 / off 1.0 / off / off
Table 14: Tokenizer training configurations.
Image tokenizer
architecture Cosmos (Agarwal et al., 2025) encoder–decoder, FSQ (Mentzer et al., 2024)
decoder width 192 (128 in Cosmos)
spatial downsampling 8×8\times
codebook size 64,000 (FSQ levels [8,8,8,5,5,5][8,8,8,5,5,5])
Params 136M
tokens per image 1,024 (32×3232\times 32, raster order)
losses (stage 1) reconstruction (1.0) + LPIPS (0.1)
losses (stage 2) + PatchGAN adversarial (5e-3)
training data GPIC train images, 1.5 epochs (1.0 stage 1 / 0.5 stage 2)
iterations 242,172
batch size 640
learning rate 1e-4, constant
optimizer Adam, β1,β2=0.9,0.999\beta_{1},\beta_{2}=0.9,0.999
weight decay 0.001
gradient clipping 1.0
EMA yes
hardware 1×1\times 8×\timesH100 node
Text tokenizer
algorithm byte-level BPE (Sennrich et al., 2016)
vocabulary size 16,384
training data GPIC train captions (21.3 GB) + OpenWebText (11.9 GB)
caption length mean 44.5, p50 36, p90 64, p99 182 tokens
Table 15: Total pretraining compute of T2I systems. We report total pretraining FLOPs, decomposed into language, image-generator, and image-tokenizer compute, for all configurations shown in Fig. 1. Vision compute includes both image-generator and image-tokenizer pretraining. For late-fusion methods, language FLOPs include pretraining of the language model. For Halcyon, language FLOPs correspond to the text-token portion of joint multimodal pretraining, without external language pretraining. All image generators are pretrained on GPIC for one epoch at 256×256256\times 256 resolution. Generation Fréchet distance (gFD) is evaluated on GPIC Test-50K; lower is better.

Complete original table 15 including all source labels and values.

Appendix B Sampling Details

Halcyon autoregressively samples 1,024 image tokens per prompt at temperature one, without top-k/top-p filtering, applying guidance only to image logits. Model-specific guidance is selected on validation and held fixed for the three benchmarks. Official baseline sampling differs: original LlamaGen uses top-k 1,000, while modified versions disable it; JiT uses fifty Euler steps and EMA weights. Guidance entails conditional and unconditional forward passes per image token.

Table 16: Classifier-free guidance (CFG) scales for T2I evaluation. We report the CFG scale selected for each model based on the lowest gFD on GPIC Validation-50K. Sweeps are shown in Figures 8 and 9.

Complete original table 16 including all source labels and values.

Original figure
Original figure
Figure 9: Classifier-free guidance (CFG) sweeps for late-fusion T2I baselines. We evaluate LlamaGen variants (left) and JiT-T2I variants (right) across CFG scales on GPIC Validation-50K. Stars indicate the scale achieving the lowest gFD for each configuration, with the corresponding gFD and CFG scale reported in the figure. The selected CFG scales are used for all subsequent evaluations on GPIC Test-50K, GenEval, and DPG-Bench.
Table 17: Inference cost of Halcyon. Wall-clock time and throughput for generating 50K images (GPIC Test-50K set) using 8xH100 GPUs with vLLM, averaged over three runs. We use temperature 1.0 without top-kk/top-pp filtering and each model’s CFG scale (two forward passes per image token). Image decoding takes 198.0 ±\pm 5.7 seconds (averaged over 3 runs) for all model sizes and is excluded from the reported results.
Params Wall time (s) ↓\downarrow Throughput (img/s) ↑\uparrow
0.1B 395.3 126.5
0.25B 507.4 98.5
0.52B 662.8 75.4
1.6B 712.2 70.2
4B 935.5 53.4
Original figure
Figure 10: Effect of language pretraining on validation NLL. Validation NLL for image tokens conditioned on text (ℐ|𝒯\mathcal{I}\mid\mathcal{T}, left) and text tokens without conditioning (𝒯|∅\mathcal{T}\mid\varnothing, right) during training of Qwen3-0.6B with random vs. language-pretrained initialization. Random initialization consistently achieves lower image NLL, while language-pretrained initialization achieves lower text NLL.

Appendix C Additional Results

Additional results separate compositional categories, training-seed variation, sampling-seed variation, and node-level inference throughput.

C.1 Per-Category Results on GenEval and DPG-Bench

I2T replacement keeps GenEval aggregate differences within the reported standard deviations, with mixed category-level effects. DPG overall improves at four sizes and decreases at 0.52B. At 4B the standard recipe still reaches only 29.06 in counting and 30.25 in color-attribute binding, despite stronger single-object scores. Aggregate quality should not be read as reliable compositional reasoning in every category.

Table 18: GenEval results by category. Overall is reported as mean ±\pm std. “+ I2T 10%” denotes the I2T supervision recipe (pdrop=0.0p_{\text{drop}}{=}0.0, η=0.1\eta{=}0.1).

Complete original table 18 including all source labels and values.

Table 19: DPG-Bench results by category. “+ I2T 10%” denotes the I2T supervision recipe (pdrop=0.0p_{\text{drop}}{=}0.0, η=0.1\eta{=}0.1).

Complete original table 19 including all source labels and values.

C.2 Standard Deviation (Training Seeds)

Only the 0.10B model with 10% I2T is retrained with three seeds. Its gFD values 144.14, 154.63, and 152.50 yield mean 150.42 and sample standard deviation 5.54. This demonstrates meaningful training variation for that configuration. It cannot supply confidence intervals for the larger models or make every small cross-model difference statistically conclusive.

Table 20: Standard deviation of training runs. Halcyon-0.10B with 10% I2T supervision, pretrained using three random seeds and evaluated with CFG 7.57.5. The last row reports mean ±\pm standard deviation (sample std, n=3n{=}3).

Complete original table 20 including all source labels and values.

C.3 Standard Deviation (Sampling Seeds)

Holding each checkpoint and guidance fixed, three sampling seeds yield gFD standard deviation at most 0.14, much smaller than the tested small-model training variation. Repeated image sampling therefore measures a different uncertainty source from retraining the models.

Table 21: Standard deviation of sampling runs. Each Halcyon model is evaluated using a fixed checkpoint and three random sampling seeds (50K images per seed). We report mean ±\pm standard deviation (n=3n{=}3).

Complete original table 21 including all source labels and values.

C.4 Inference Cost

Using vLLM across one eight-H100 node, generating fifty thousand images takes 395.3–935.5 seconds from 0.1B to 4B, corresponding to 126.5–53.4 images/second for the whole node. The table excludes image-token decoding, which adds 198.0±5.7 seconds. These are batched node-throughput measurements, not single-image latency or a direct speed comparison with diffusion baselines.

Appendix D Samples

All tag/short/medium/long prompt grids are retained at original resolution and with their complete captions. They illustrate better detail and prompt adherence as model size grows, while legible text remains difficult across models. Selected qualitative examples complement rather than replace distribution and category-level evaluation.

Refer to caption
Figure 11: Comparison of generated images with “tag” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.
Refer to caption
Figure 12: Comparison of generated images with “short” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.
Refer to caption
Figure 13: Comparison of generated images with “medium” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.
Refer to caption
Figure 14: Comparison of generated images with “long” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.
Concept demonstration

Open image ↗