Studying Image Tokenizers as Visual Languages
in Unified Multimodal Models

Motivation

A discrete image tokenizer is not merely a preprocessing module: it defines the visual language that a language model must learn, align with text, and use for both visual generation and understanding. Yet existing studies mostly examine it outside the unified modeling context — through reconstruction-based metrics such as rFID, ImageNet classification accuracy, or generation-only / understanding-only downstream pipelines.

Those evaluations are valuable, but they leave open how a tokenizer affects joint modeling behavior. In unified pure-AR models, image and text tokens are modeled jointly under a shared model and training objective. Joint training with text may affect image-token modeling, while the choice of image tokenizer may in turn affect cross-modal alignment and text modeling. Consequently, design choices such as compression ratio and auxiliary losses may be optimized without fully accounting for their effects on joint image–text modeling.

We therefore study tokenizers inside unified AR training itself, using task-specific validation losses during continual pretraining as the measurement instrument. Because that instrument is not yet well characterized in this setting, we first establish how the losses behave before using them to study tokenizers.

Interpreting loss in unified multimodal training

Losses scale differently per task

Losses over text, image, T2I and I2T prediction all scale with data and model size, but they exhibit distinct behaviors and produce different tokenizer rankings — making an averaged loss too coarse as an analysis signal.

Within one tokenizer, loss tracks generation quality

Validation loss measures next-token modeling quality; the benchmarks emphasize semantic faithfulness and perceptual quality. Do they agree? With tokenizer type controlled, yes: T2I loss aligns strongly with GenAI-Bench VQAScore and MJHQ-30K gFID, across data scales and hyperparameter settings. Surprisingly, I2T loss also tracks generation quality, with weaker correlation, even though it predicts text rather than image tokens. This implies that I2T loss reflects the general multimodal training progress when the recipe is fixed.

Across tokenizers, the token space changes the picture

Across tokenizers the T2I loss–performance relationship shifts with the visual token space: T2I loss clusters by tokenizer type and no longer directly aligns with generation quality. I2T loss, computed over shared text tokens, is still compared on a shared basis and stays aligned with benchmark performance.

Part of that shift comes from vocabulary size (B): normalizing T2I loss by log₂B recovers a more consistent loss–performance relation across the IBQ variants. The gap that remains is related to reconstruction fidelity — with the benchmark score controlled, tokenizers of the same vocabulary size follow the reverse order of their rFID ranking.

The signal survives supervised finetuning

When the tokenizer is fixed (GigaTok), both the T2I and I2T losses measured before SFT stay strongly correlated with post-SFT generation performance.

Across tokenizers, pre-SFT I2T loss still correlates with post-SFT generation and general VQA (VQAv2, GQA), though only moderately. That may partly reflect differences in both task and training stage: the loss measures caption prediction during pretraining, while the benchmarks measure question answering after SFT, so caption fit is informative about VQA performance without determining it. The relationship is less consistent for specialized benchmarks such as TextVQA, which additionally require capabilities such as OCR.

Revisiting tokenizer design axes

With this lens in place, we revisit three design decisions through controlled ablations.

Discriminator

Replacing GigaTok's PatchGAN discriminator with a DINO-based one improves rFID from 0.81 to 0.51 — but none of the validation losses improve significantly, GenAI is unchanged at 0.720, and VQAv2 drops from 52.25 to 51.31.

Semantic supervision

Semantic supervision worsens reconstruction fidelity yet improves multimodal learnability. It strengthens object-level associations between image tokens and words, without consistently reducing the local n-gram entropy of the image tokens — evidence for “better visual words” rather than a simpler local visual grammar.

Vocabulary size

Larger vocabularies are known to improve reconstruction fidelity but may also change how hard image tokens are to model, so we compare IBQ at B ∈ {1024, 8192, 16384} and find the effect non-monotonic: IBQ-8192 gives the lowest normalized T2I and image loss, IBQ-16384 the best I2T loss, and the largest vocabulary still performs best downstream.

Limitations and future directions

1. Seven tokenizers, one shape — all fixed-length and single-codebook. Varying the sequence length K cannot hold both the image-token count and the image count constant, so sub-codebooks, larger input resolutions and other K are left to future work.
2. No from-scratch scaling law across more model scales, and no compute-efficient frontiers; comparing tokenizers on that frontier is the natural next step.
3. One architecture. Our findings come from dense, pure-AR Qwen3 models, and holding that fixed is itself a control. We therefore do not claim the loss relationships or tokenizer rankings transfer to another language-model family, to Mixture-of-Experts variants, or to unified models that pair AR with a diffusion module (serial or hybrid). The analysis framework itself does transfer, and validating it in those settings is an important direction.
4. A limited benchmark suite — general VQA and OCR, but no region-level grounding or fine-grained spatial-reasoning benchmark, and no safety or human-preference evaluation.

Citation

@misc{tokenizer-umm,
  title         = {Studying Image Tokenizers as Visual Languages in Unified Multimodal Models},
  author        = {Siting Li and Zhengyang Wang and Simon Shaolei Du and Xi Chen and Yang Liu},
  year          = {2026},
  eprint        = {2609.09143},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.09143}
}