Overview
Halcyon learns text-conditioned image generation directly from paired captions and images. Controlled experiments test whether a separately pretrained language model improves that task under the studied budget.
At a glance
Early-fusion autoregressive models trained on one hundred million GPIC image/text pairs reach gFD 48.0 at 0.52B parameters, 35.3 at 1.6B, and 28.4 at 4B. The larger variants outperform the tested late-fusion baselines. External text-only model pretraining provides no measured benefit in the controlled small-model experiments. Replacing unconditional image-only examples with image-caption sequences adds captioning at similar generation quality. The headline compute ratios count full upstream language-model pretraining; they are not incremental finetuning or inference speedups.
1 Introduction
Most recent image generators use a separately pretrained text encoder or initialize/train an early-fusion model with text-only data. This study asks how much that upstream language-model training contributes to image generation. Its definition of “language pretraining” excludes image-paired captions. The generator learns from GPIC pairs, but the text tokenizer also uses OpenWebText, and image-tokenizer training includes perceptual supervision. Thus the finding is about learned language-model pretraining, not removing every external text resource or learned perceptual component.
2 Related Work
Earlier systems such as DALL-E, CogView, and GLIDE already learned conditioning from paired data, so the contribution is a controlled reassessment rather than the first encoder-free image generator. The paper compares language/vision token budgets and representative autoregressive/diffusion baselines. Those families differ in representation and optimization; the experiments are designed to study language pretraining, not establish a universal ranking of autoregression versus diffusion.
3 Approach
A decoder-only transformer predicts a joint sequence of caption and discrete image tokens. The main models, text tokenizer, and image-tokenizer weights are trained from scratch, with the tokenizer’s separate training data and losses specified below. Early fusion allows text representations to be learned jointly with the image task, although the experiments do not isolate joint training as the sole cause of improvement.
3.1 Tokenization
A Cosmos-style finite-scalar-quantized image tokenizer maps each 256×256 image to a 32×32 raster sequence of 1,024 tokens from 64,000 codes. The byte-level BPE tokenizer learns 16,384 text entries from GPIC captions plus OpenWebText. A tied embedding/output vocabulary totals 80,640 entries including 256 specials. The original token markers are [SOV] and [EOV] for image boundaries, [EOS] for sequence end, and [PAD] for padding.
3.2 Sequence Format and Training Objective
Text-only sequences are packed; image-containing sequences contain one pair and are padded to fixed length. Text-to-image predicts image tokens after a caption, while image-to-text predicts the image first and then its caption. Image-only and image-to-text share the unconditional image prefix, allowing both to train the classifier-free-guidance branch. The original guidance and weighted next-token equations are preserved together: image loss weight is 0.5, padding is excluded, and special tokens count with text. Guidance combines conditional and unconditional image-token logits only.
Equation
3.3 Pretraining Data Composition and Scheduling
Five recipes separate paired-data learning from additional supervision: pure T2I; text-only then T2I; I2T then T2I; interleaved I2T/T2I; and replacing caption-dropped examples with I2T while disabling dropout. The last substitution preserves conditional/unconditional image-token counts rather than allocating a new fraction away from conditional generation. Staged language experiments usually redistribute a fixed budget, with a separate extra-compute control preserving a full T2I epoch.
4 Experiments
Experiments measure tokenizer fidelity, generated-image distribution/composition, language-pretraining interventions, captioning, and compute. Evaluation and training variation are reported separately in the appendices.
4.1 Experimental Setup
The main generator trains on GPIC at 256×256, with sizes 0.1B–4B and 1,280-token sequences. Caption dropout is normally 0.1. The image tokenizer trains 1.5 epochs, adding adversarial loss after one epoch; text-tokenizer training uses 21.3 GB GPIC captions and 11.9 GB OpenWebText, which is not supplied as generative-model language pretraining. The tokenizer improves reconstruction gFD from Cosmos’s 37.93 to 18.86, with real-versus-real reference 7.59, but small text remains difficult.
LlamaGen controls vary pretrained T5 size and image tokenization. JiT controls add perceptual loss and change patch size, following PixelGen. A separate matched-architecture experiment compares pretrained versus randomly initialized Qwen3. Main Halcyon and baselines have different optimizer schedules detailed in their tables, so not every cross-family difference is a single-variable intervention.
GPIC uses Test-50K distribution metrics; guidance is chosen on Validation-50K, then reused for GenEval and DPG-Bench. GenEval prompts are rephrased following PixelGen; DPG uses original prompts. Captioning is assessed by a Qwen3-VL reranker rather than a human study. Separate token likelihoods distinguish conditional image/text tasks from their unconditional counterparts.
| Tokenizer | Vocab. | Bytes/token | Tok./caption |
|---|---|---|---|
| FLAN-T5 | 32,100 | 4.152 | 51.1 |
| Qwen3 | 151,669 | 4.763 | 44.6 |
| Halcyon | 16,384 | 4.774 | 44.5 |
4.2 Results: Comparison with Late-fusion T2I Methods
Changing LlamaGen’s tokenizer and token count improves gFD 147.0→63.3. Adding JiT perceptual loss improves 67.9→48.9, and smaller patches further improve it to 41.0. Reducing T5 encoder size from 1.22B to 0.11B changes LlamaGen gFD only 63.3→66.7. These controls show that image representation and supervision substantially affect the comparison.
Halcyon-0.52B reaches gFD 48.02, GenEval 44.91, and DPG 76.84; the 1.6B version reaches 35.28/53.48/77.90 and the 4B version 28.43/56.39/77.90. The smallest model is much weaker at gFD 144.16. Headline 18.2-fold and 275-fold compute reductions charge each comparator its entire upstream language-model pretraining. They describe rebuilding the whole system, not the additional cost of reusing an already available text encoder.
4.3 Results: Effect of Language Pretraining on T2I generation
Allocating 10%, 20%, or 50% of the 0.52B training budget to text-only learning worsens gFD from 48.02 to 51.56, 52.76, or 65.13. Keeping a full paired-data epoch after extra language training gives 49.22 at 1.5× budget. With matched Qwen architecture/vocabulary, scratch gives 46.61 versus pretrained 47.81; lowering the pretrained model’s learning rate worsens it to 63.62. These tested recipes show no measured gain, without proving that all larger-scale language pretraining or other datasets are useless.
| Text tokens (B) | Image tokens (B) | NLL | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Configuration | gFD | |||||||||
| Halcyon-0.52B | – | 4.05 | 0 | 10.24 | 92.16 | 1.433 | 18.414 | 8.428 | 8.366 | 48.02 |
| Staged: Language T2I | 0.10 | 16.44 | 0 | 9.22 | 82.94 | 1.447 | 19.098 | 8.442 | 8.382 | 51.56 |
| 0.20 | 28.83 | 0 | 8.19 | 73.73 | 1.451 | 17.197 | 8.445 | 8.388 | 52.76 | |
| 0.50 | 65.80 | 0 | 5.12 | 46.08 | 1.454 | 22.850 | 8.505 | 8.451 | 65.13 | |
| Staged: Language T2I† | 0.33 | 68.03 | 0 | 10.24 | 92.16 | 1.445 | 18.809 | 8.428 | 8.371 | 49.22 |
| Text tokens (B) | Image tokens (B) | Text bpb | Image bpb () | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | LP | gFD | ||||||||
| Halcyon-0.52B | 0 | 4.05 | 0 | 10.24 | 92.16 | 0.433 | 5.565 | 6.333 | 6.286 | 48.02 |
| Qwen3-0.6B | 0 | 4.05 | 0 | 10.24 | 92.16 | 0.434 | 5.436 | 6.324 | 6.278 | 46.61 |
| Qwen3-0.6B-CPT | 36T | 4.05 | 0 | 10.24 | 92.16 | 0.430 | 4.974 | 6.326 | 6.283 | 47.81 |
| Qwen3-0.6B-CPT ∗ | 36T | 4.05 | 0 | 10.24 | 92.16 | 0.424 | 6.743 | 6.409 | 6.367 | 63.62 |
4.4 Results: Effect of I2T Supervision on T2I generation
Spending a separate 10% budget on I2T gives similar image quality, while 50% reduces conditional image training and worsens it—especially interleaving, gFD 75.97 versus 54.22 staged. Interleaving nevertheless learns captioning better; staged I2T followed by T2I loses that capability by the reported text likelihood.
Replacing the baseline’s 10% caption-dropped examples with I2T preserves image-token counts and improves caption likelihood while keeping generation broadly comparable. At 0.52B, conditional text NLL falls 18.414→1.102; at 4B the caption-alignment score reaches 0.559, still below ground-truth captions at 0.891. Generation changes are mixed across metrics/sizes, so this is a useful additional capability rather than a universal image-quality improvement.
| Text tokens (B) | Image tokens (B) | NLL | ||||||||
| Configuration | (or ) | gFD | ||||||||
| Baseline | – | 4.05 | 0 | 10.24 | 92.16 | 1.433 | 18.414 | 8.428 | 8.366 | 48.02 |
| Staged: I2T T2I | 3.65 | 0.45 | 19.46 | 82.94 | 1.440 | 18.415 | 8.426 | 8.366 | 48.35 | |
| 2.03 | 2.25 | 56.32 | 46.08 | 1.435 | 20.437 | 8.421 | 8.370 | 54.22 | ||
| Mixing: T2I + I2T | 3.65 | 0.45 | 19.46 | 82.94 | 1.443 | 1.125 | 8.414 | 8.362 | 48.18 | |
| 2.03 | 2.25 | 56.32 | 46.08 | 1.475 | 1.009 | 8.453 | 8.417 | 75.97 | ||
| NLL | T2I generation | Captioning | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Size | gFD | GenEval | DPG-Bench | Alignment | ||||||
| 0.10B | – | 0.1 | 1.601 | 19.066 | 8.774 | 8.727 | 144.16 | 27.31 | 66.36 | – |
| 0.1 | 0 | 1.598 | 1.410 | 8.771 | 8.724 | 144.14 | 27.66 | 68.89 | 0.090 | |
| 0.25B | – | 0.1 | 1.497 | 21.472 | 8.566 | 8.509 | 72.55 | 35.72 | 72.66 | – |
| 0.1 | 0 | 1.491 | 1.218 | 8.559 | 8.503 | 70.44 | 36.54 | 73.07 | 0.207 | |
| 0.52B | – | 0.1 | 1.433 | 18.414 | 8.428 | 8.366 | 48.02 | 44.91 | 76.84 | – |
| 0.1 | 0 | 1.435 | 1.102 | 8.419 | 8.358 | 46.48 | 44.60 | 76.14 | 0.315 | |
| 1.6B | – | 0.1 | 1.370 | 20.426 | 8.303 | 8.235 | 35.28 | 53.48 | 77.90 | – |
| 0.1 | 0 | 1.372 | 0.977 | 8.292 | 8.228 | 35.91 | 53.51 | 78.21 | 0.467 | |
| 4B | – | 0.1 | 1.338 | 16.738 | 8.193 | 8.122 | 28.43 | 56.39 | 77.90 | – |
| 0.1 | 0 | 1.325 | 0.890 | 8.196 | 8.126 | 29.30 | 55.33 | 78.61 | 0.559 | |
5 Discussion and Conclusion
Larger models lower image/text likelihood and generation distance. The validation-optimal guidance scale declines from 7.5 to 4.5, but guidance remains beneficial. Medium/long caption groups yield lower gFD than short captions; this comparison does not by itself isolate a causal benefit of appending more words to otherwise identical prompts.
The scope is one dataset and 256×256 images. Controlled composition tests use 0.52B and pretrained-initialization tests Qwen3-0.6B; larger budgets, other datasets, diffusion-style early fusion, and video remain open. Text rendering and some compositional categories remain weak. Full pretraining accounting makes upstream costs visible, but does not establish deployment throughput or the marginal cost of adopting an existing checkpoint.
| Model | Test-50K | Short | Medium | Long |
|---|---|---|---|---|
| 0.1B | 144.16 | 149.03 | 139.27 | 147.85 |
| 0.25B | 72.55 | 77.18 | 70.14 | 72.53 |
| 0.52B | 48.02 | 51.99 | 46.12 | 46.83 |
| 1.6B | 35.28 | 38.59 | 34.21 | 34.33 |
| 4B | 28.43 | 31.48 | 27.91 | 27.77 |
AI Use Statement
The authors report generative-AI assistance with implementation, code, literature search, manuscript drafting/editing, and readability. They state that hypotheses, methodology, and result interpretation were not delegated, and that generated code and suggested references were checked. The original disclosure is retained.
Ethics Statement
The authors acknowledge realistic-image misuse risks and possible GPIC biases, with a cheaper training recipe lowering barriers for beneficial and harmful uses alike. No new human-subject study or dataset collection is reported; the paper does not present a comprehensive bias or misuse evaluation.
Reproducibility Statement
The paper supplies architecture, tokenization, objective, data schedules, baseline changes, evaluation, compute accounting, and detailed configurations. It states that checkpoints and GPIC are publicly available. Reproduction still depends on following the distinct model-specific schedules and guidance selections.
Acknowledgments
Radical Numerics supplied compute; the authors thank colleagues for feedback, technical discussion, and infrastructure support. The original names remain in the source acknowledgment.
Appendix
The appendix retains all model/tokenizer configurations, sampling settings, compute breakdowns, per-category results, seed studies, and qualitative sample grids.
Appendix A Training Details
Halcyon uses a Qwen3-style decoder with tied embeddings, 97,632 updates, batch 1,024, maximum 254 caption tokens, image loss weight 0.5, and warmup/stable/decay learning rates. The tokenizer has 136 million parameters and trains 242,172 updates on eight H100s. Baseline tables retain their own batch sizes, 390,600 updates, samplers, and perceptual-loss settings. Compute Table 15 assigns 1.45×10²⁰ tokenizer FLOPs to modified LlamaGen despite using the Halcyon tokenizer, while Halcyon’s tokenizer entry is 1.31×10²⁰; the reported accounting is retained without silently reconciling that discrepancy.
| 0.1B | 0.25B | 0.52B | 1.6B | 4B | |
| architecture | |||||
| backbone | Qwen3 (Yang et al., 2025a), decoder-only | ||||
| layers | 14 | 21 | 28 | 28 | 36 |
| hidden dim | 512 | 768 | 1024 | 2048 | 2560 |
| FFN dim | 1536 | 2304 | 3072 | 6144 | 9728 |
| query / KV heads | 8 / 4 | 12 / 6 | 16 / 8 | 16 / 8 | 32 / 8 |
| head dim | 128 | ||||
| non-embedding params | 55.1M | 185.8M | 440.4M | 1409.3M | 3634M |
| embedding params | 41.3M | 61.9M | 82.6M | 165.2M | 206.4M |
| total params | 96.4M | 247.7M | 523.0M | 1574.5M | 3840M |
| vocabulary | 80,640 (16,384 text + 64,000 image + 256 special) | ||||
| tied embeddings | yes | ||||
| image size | 256 | ||||
| image tokens | 1,024 | ||||
| max caption tokens | 254 | ||||
| sequence length | 1,280 | ||||
| training | |||||
| training steps | 97,632 (1 epoch on GPIC) | ||||
| global batch size | 1,024 | ||||
| warmup steps | 1,000 | ||||
| peak learning rate | 1e-3 | 7.5e-4 | 7.5e-4 | 7.5e-4 | 7.5e-4 |
| lr schedule | linear warmup, constant to step 49,595, cooldown to | ||||
| optimizer | AdamW, | ||||
| weight decay | 0.1 | ||||
| gradient clipping | 1.0 | ||||
| init std | 0.02 | ||||
| image loss weight | 0.5 | ||||
| caption dropout | 0.1 | ||||
| precision | bf16 | ||||
| sampling | |||||
| cfg | 7.5 | 6.0 | 6.0 | 5.0 | 4.5 |
| JiT-T2I-XXL/16 | JiT-T2I-XXL/16++ | JiT-T2I-XXL/8++ | |
| language model (frozen) | |||
| text encoder | Qwen3-1.7B | ||
| feature dim | 2048 | ||
| max caption tokens | 300 | ||
| image generator | |||
| backbone | JiT-T2I (pixel space, no image tokenizer) | ||
| blocks (image / text) | 16 / 4 | ||
| hidden dim | 1536 | ||
| attention heads | 24 | ||
| patch bottleneck dim | 256 | ||
| params | 1.10B | ||
| patch size | 16 | 16 | 8 |
| image tokens | 256 | 256 | 1,024 |
| training | |||
| objective | flow matching, linear schedule | ||
| timestep sampling | logit-normal (, ), | ||
| LPIPS weight | – | 0.1 | 0.1 |
| LPIPS timestep gate | – | ||
| training steps | 390,600 | ||
| global batch size | 256 | ||
| peak learning rate | 1e-4 | ||
| lr schedule | 4,000 warmup, constant, 4,000 cooldown | ||
| optimizer | AdamW, | ||
| weight decay | 0 | ||
| gradient clipping | 1.0 | ||
| EMA decay | 0.9999 | ||
| caption dropout | 0.1 | ||
| precision | bf16 | ||
| sampling | |||
| sampler | Euler (ODE), 50 steps, EMA weights | ||
| CFG scale | 6.00 | 5.00 | 4.00 |
| LlamaGen-XL | LlamaGen-XL/8 | LlamaGen-XL-T5-Base/8 | |
| language model (frozen) | |||
| text encoder | FLAN-T5-XL | FLAN-T5-XL | FLAN-T5-Base |
| encoder params | 1.22B | 1.22B | 0.11B |
| feature dim | 2048 | 2048 | 768 |
| max caption tokens | 300 | ||
| image tokenizer | |||
| tokenizer | VQ-16 (pretrained) | Ours (GPIC-trained) | |
| spatial downsampling | 16 | 8 | |
| codebook size | 16,384 | 64,000 | |
| image tokens | 256 | 1,024 | |
| image generator | |||
| backbone | GPT-XL (Llama-style, decoder-only) | ||
| layers / hidden dim / heads | 36 / 1280 / 20 | ||
| FFN dim | 3584 (SwiGLU) | ||
| params | 0.78B | 0.90B | 0.90B |
| sequence length | 556 | 1,324 | 1,324 |
| training | |||
| training steps | 390,600 | 390,600 | 390,600 |
| global batch size | 256 | ||
| peak learning rate | 1e-4 | ||
| lr schedule | 3,906 warmup, then constant | ||
| optimizer | AdamW, | ||
| weight decay | 0.05 | ||
| gradient clipping | 1.0 | ||
| dropout / token dropout | 0.1 / 0.1 | ||
| caption dropout | 0.1 | ||
| precision | bf16 | ||
| sampling | |||
| CFG scale | 6.5 | 6.5 | 5.5 |
| temperature / top- / top- | 1.0 / 1000 / off | 1.0 / off / off | |
| Image tokenizer | |
|---|---|
| architecture | Cosmos (Agarwal et al., 2025) encoder–decoder, FSQ (Mentzer et al., 2024) |
| decoder width | 192 (128 in Cosmos) |
| spatial downsampling | |
| codebook size | 64,000 (FSQ levels ) |
| Params | 136M |
| tokens per image | 1,024 (, raster order) |
| losses (stage 1) | reconstruction (1.0) + LPIPS (0.1) |
| losses (stage 2) | + PatchGAN adversarial (5e-3) |
| training data | GPIC train images, 1.5 epochs (1.0 stage 1 / 0.5 stage 2) |
| iterations | 242,172 |
| batch size | 640 |
| learning rate | 1e-4, constant |
| optimizer | Adam, |
| weight decay | 0.001 |
| gradient clipping | 1.0 |
| EMA | yes |
| hardware | 8H100 node |
| Text tokenizer | |
| algorithm | byte-level BPE (Sennrich et al., 2016) |
| vocabulary size | 16,384 |
| training data | GPIC train captions (21.3 GB) + OpenWebText (11.9 GB) |
| caption length | mean 44.5, p50 36, p90 64, p99 182 tokens |
Appendix B Sampling Details
Halcyon autoregressively samples 1,024 image tokens per prompt at temperature one, without top-k/top-p filtering, applying guidance only to image logits. Model-specific guidance is selected on validation and held fixed for the three benchmarks. Official baseline sampling differs: original LlamaGen uses top-k 1,000, while modified versions disable it; JiT uses fifty Euler steps and EMA weights. Guidance entails conditional and unconditional forward passes per image token.
| Params | Wall time (s) | Throughput (img/s) |
|---|---|---|
| 0.1B | 395.3 | 126.5 |
| 0.25B | 507.4 | 98.5 |
| 0.52B | 662.8 | 75.4 |
| 1.6B | 712.2 | 70.2 |
| 4B | 935.5 | 53.4 |
Appendix C Additional Results
Additional results separate compositional categories, training-seed variation, sampling-seed variation, and node-level inference throughput.
C.1 Per-Category Results on GenEval and DPG-Bench
I2T replacement keeps GenEval aggregate differences within the reported standard deviations, with mixed category-level effects. DPG overall improves at four sizes and decreases at 0.52B. At 4B the standard recipe still reaches only 29.06 in counting and 30.25 in color-attribute binding, despite stronger single-object scores. Aggregate quality should not be read as reliable compositional reasoning in every category.
C.2 Standard Deviation (Training Seeds)
Only the 0.10B model with 10% I2T is retrained with three seeds. Its gFD values 144.14, 154.63, and 152.50 yield mean 150.42 and sample standard deviation 5.54. This demonstrates meaningful training variation for that configuration. It cannot supply confidence intervals for the larger models or make every small cross-model difference statistically conclusive.
C.3 Standard Deviation (Sampling Seeds)
Holding each checkpoint and guidance fixed, three sampling seeds yield gFD standard deviation at most 0.14, much smaller than the tested small-model training variation. Repeated image sampling therefore measures a different uncertainty source from retraining the models.
C.4 Inference Cost
Using vLLM across one eight-H100 node, generating fifty thousand images takes 395.3–935.5 seconds from 0.1B to 4B, corresponding to 126.5–53.4 images/second for the whole node. The table excludes image-token decoding, which adds 198.0±5.7 seconds. These are batched node-throughput measurements, not single-image latency or a direct speed comparison with diffusion baselines.
Appendix D Samples
All tag/short/medium/long prompt grids are retained at original resolution and with their complete captions. They illustrate better detail and prompt adherence as model size grows, while legible text remains difficult across models. Selected qualitative examples complement rather than replace distribution and category-level evaluation.