A survey · 2026

Towards Next-Generation Architectures of Large Language Models

What the canonical decoder fixes, and what recent architectures change. A reading of open-weight language models as a design space, not a catalogue of modules.

Yao Teng^1,† Guoxuan Chen^1,† Yimeng Chen^2,* Han Shi^3,* Piotr Piękos² Jie Ran³ Yutao Zeng⁴ Chuanyang Zheng³ Jingyao Li⁵ Li Nanbo² Haoli Bai³ Chao Huang¹ Xihui Liu¹ Di He⁶ Weiyang Liu^5,7 Jürgen Schmidhuber^2,8

¹The University of Hong Kong · ²Center of Excellence for Generative AI, KAUST · ³Huawei Technologies · ⁴ByteDance Seed · ⁵The Chinese University of Hong Kong · ⁶Peking University · ⁷Max Planck Institute for Intelligent Systems · ⁸The Swiss AI Lab, IDSIA-USI/SUPSI

^† Equal contribution. ^* Corresponding authors.

Paper PDF GitHub Hugging Face Cite

Abstract

The decoder, read as a set of assumptions

Large language models have advanced rapidly through increases in model scale, data, and computation, while their underlying architectures have also undergone substantial evolution. Beyond the canonical Transformer, recent designs seek greater efficiency, scalability, and flexibility by rethinking how sequence information is processed, represented, and transformed.

We survey efficient attention and state management, modern recurrent and other linear-complexity sequence models, Mixture-of-Experts architectures, and advances in activation functions, normalization, residual connections, positional encoding, and tokenization. We also cover multi-token prediction and adaptive and recurrent depth. Across these areas we highlight trade-offs among representational power, optimization stability, and computational cost.

Complementing this component-wise treatment, we take a system-level view. We trace representative open-weight families to identify recurring patterns in how choices are combined, and synthesize those developments along four directions—processing structure, transformation dynamics, state management, and output commitment—asking which assumptions of the canonical decoder each design preserves, refines, or relaxes.

Section 15

Four directions of change

From the canonical decoder to four directions of architectural change — **Fig. 1** Opening view of the survey. The canonical decoder fixes a pre-norm residual block, a dense feed-forward network, the same depth for every token, and a token-level key–value cache that grows with generation. The four panels are developed in Section 15.

Processing

Flexible and conditional: who runs, and how much work a token receives.

State

Selective and flexible: what history is kept, and how that state is updated.

Lookahead

Delayed commitment: predict further ahead, then decide what to keep.

Dynamics

Refined transforms: the nonlinearity and the path across depth.

Section 1

Where open-weight models stand

MMLU against release date covers roughly 2021–2025. By 2026 that benchmark was nearing saturation, so the survey turns to Humanity’s Last Exam for the recent open-weight generation.

Open-weight model scale versus Humanity's Last Exam — **Fig. 2** HLE versus open-weight model size. Marker area grows with active parameters for mixture-of-experts models. Dashed and dotted curves are observed Pareto frontiers with and without tools. Scores that mix protocols should be compared with caution.

MMLU versus release date — **Fig. 3** Earlier generations on MMLU. Scale still matters at a given moment; newer and smaller models often pass older and much larger ones; sparse capacity reaches the top of the range at lower active cost.

Section 2

Survey map

Sections 1 and 2 frame the tree; they are not a third technical family. Color follows the role of each block in the stack. Scroll the figure to read the full taxonomy.

Survey taxonomy: eight architectural families from foundations through system-level synthesis — **Fig. 4** Taxonomy in Section 2, numbered as in the survey.

3–4 · Foundations

From Shannon’s source to the decoder-only Transformer: attention, feed-forward, position, residual, tokenization, and next-token generation.

5 · Efficient attention

Keep quadratic attention; shrink the KV state along token, head, dimension, and layer.

6 · Linear language models

Replace pairwise mixing with a linear-time operator: RNNs, fast weights, state space, linearized attention.

7 · Mixture of experts

Conditional parameter activation: sparse gating, fine-grained shared experts, systems-oriented variants.

8–9 · Inside the block

Activation functions, then normalization and residual wiring—the path representations take across depth.

10–11 · Sequence interface

Positional encoding and tokenization: the units and coordinates of the sequence.

12–13 · Compute per token

Multi-token prediction and adaptive or recurrent depth: how much work a generated token receives.

14–16 · Synthesis

Released families, a unified design space, and three conclusions about what is still the reference decoder.

Section 15.2

Ten design dimensions

The canonical structure is simple: a fixed tokenization, the same dense blocks, an append-only token-level cache, and one committed next token. The table is the survey’s decomposition of what can change.

Direction	Dimension	Canonical assumption	Alternative
Flexible and conditional processing	Sequence interface	One fixed tokenization and positional scheme	Finer, multi-scale, or adaptive units
	Operator assignment	The same sequence mixer in every layer	Different operators across layers or heads
	Parameter activation	Every token runs the same dense blocks	Input-dependent routing among experts
	Computation allocation	A fixed path and a predetermined amount of work	Early exit, conditional or recurrent depth, drafter–verifier
Refined transformation dynamics	Local nonlinearity	A fixed feed-forward nonlinearity, often SwiGLU	Other gates: squared ReLU, clamped SwiGLU
Refined transformation dynamics	Depth propagation	Fixed residual and normalization pathways	Other placement, residual scaling, or gating
Selective and flexible state	History representation	Individually addressable token states	Selective, compressed, recurrent, or hybrid history
Selective and flexible state	State update	Append-only growth	Eviction, merging, compression, branching, or rollback
Lookahead and delayed commitment	Prediction horizon	Next-token supervision only	Multi-step, blockwise, or feature-level prediction
Lookahead and delayed commitment	Output commitment	One committed token per forward pass	Drafting, parallel verification, delayed commitment

Section 14.2

Patterns in released models

A pattern is a combination. The four are not a partition: one backbone can instantiate more than one.

14.2.1

Canonical dense decoders

Decoder-only stack, dense feed-forward blocks, fixed depth. Pre-RMSNorm, SwiGLU, and rotary positions became the usual fill of that skeleton. LLaMA is the reference instance.

14.2.2

Sparse capacity, compressed attention state

Expert routing and cache compression spend different budgets. DeepSeek-V2 and V3 pair fine-grained shared experts with multi-head latent attention.

14.2.3

Hybrid sequence and state processing

Softmax attention keeps an addressable history; a linear operator compresses it as it is built. Griffin, Jamba, Nemotron, Qwen3-Next, and Kimi Linear interleave the two.

14.2.4

Prediction-augmented backbones

A multi-token objective or module sits on a dense, sparse, or hybrid trunk. DeepSeek-V3, MiMo-7B, LongCat-Flash, and MiMo-V2-Flash are instances.

Section 16

What stays fixed, and what has to be designed together

The survey closes on three claims. Most designs still start from the canonical decoder and relax one assumption. The comparisons that matter are the combinations, and those combinations are bounded by systems as much as by modeling.

The reference remains

A dense decoder with a fixed schedule, Pre-RMSNorm, SwiGLU, and rotary positions is still the baseline. Most of the survey changes one of those choices and keeps the rest.

Compare the joint design

Sparse capacity with a compressed cache, hybrid sequence operators, and a prediction module on top of a trunk show up because the pieces interact. A module list does not capture that.

Systems bound the space

Load balance, kernels, credit assignment, and caching decide what can actually ship. At least one hybrid family returned to full attention when the infrastructure was immature, then later moved on.

Citation

If the survey is useful, please cite it

@article{llmarchsurvey2026,
  doi = {10.20944/preprints202610.1532.v1},
  url = {https://doi.org/10.20944/preprints202610.1532.v1},
  year = 2026,
  month = {October},
  publisher = {Preprints},
  author = {Yao Teng and Guoxuan Chen and Yimeng Chen and Han Shi and Piotr Piękos and Jie Ran and Yutao Zeng and Chuanyang Zheng and Jingyao Li and Li Nanbo and Haoli Bai and Chao Huang and Xihui Liu and Di He and Weiyang Liu and Jürgen Schmidhuber},
  title = {Towards Next-Generation Architectures of Large Language Models: A Survey},
  journal = {Preprints}
}

Paper on Preprints.org · doi:10.20944/preprints202610.1532.v1 · Literature map on GitHub