How LLMs Are Implemented, Algorithmically

Pull apart a decoder-only large model and you find only five modules: attention, normalization, positional encoding, the feed-forward network, plus the prefill / decode scheduling that strings them into a two-phase inference loop. This structure has barely changed from the 2017 Transformer [1] to today, and every “new architecture” is a local replacement inside one of these five modules. This piece gives each module its own chapter, writes out its core formulas, and then uses two real, current models — the latest Qwen3.5 [24] and DeepSeek-V4 [25] — to see what choices each one makes across these modules.

The whole piece uses one consistent set of symbols:

Tensor shapes follow PyTorch convention, written as [B, ..., H]; weight matrices follow the “input dim × output dim” convention, written as $W \in \mathbb{R}^{[\text{in}, \text{out}]}$.

Attention — Q/K/V · Scaled Dot-Product · MHA/MQA/GQA/MLA

Attention is the only module in a Transformer that lets different tokens exchange information; every other module operates on each token independently. It first projects each token’s hidden state $X$ into three copies — query, key, and value:

• $X \in \mathbb{R}^{B \times S \times H}$ — the normalized input
• $W_Q \in \mathbb{R}^{H \times n_q d}$ — Q projection; $W_K, W_V \in \mathbb{R}^{H \times n_{kv} d}$ — K, V projections
• $Q \in \mathbb{R}^{B \times S \times n_q d}$, $K, V \in \mathbb{R}^{B \times S \times n_{kv} d}$, then reshaped to expose the head dimension

Scaled Dot-Product Attention

Within a single head, attention is just “use the similarity between Q and all Ks as weights to average over V”:

• $QK^{\top} \in \mathbb{R}^{S \times S}$ — the similarity matrix for each pair $(q_i, k_j)$
• $\sqrt{d}$ — the scaling factor, keeping the inner products from growing so large that softmax is pushed into a saturated region with near-zero gradients
• $M \in \mathbb{R}^{S \times S}$ — the causal mask, with the upper triangle set to $-\infty$ (position $i$ can only see positions $\le i$)
• softmax normalizes along the K dimension into attention weights, which are then weighted-summed with $V$ back to $[S, d]$

Multi-Head attention replicates this operation $n_q$ times, each in its own $d$-dimensional subspace, then concatenates and projects back to $H$:

Different heads learn different attention patterns (near-neighbor, syntactic, coreference, …), which is where attention’s expressiveness comes from.

Four variants: touching only the K, V side

At inference, $Q$ is used once and discarded, but the $K$, $V$ of every historical position must be kept for later tokens — this is the KV Cache. It grows linearly with the sequence and is the main pressure on long-context memory and decode bandwidth. The difference among the four attention variants lies entirely in “how many independent sets of K, V there are, and whether K, V are low-rank compressed”; the Q side is always $n_q$ independent heads.

MHA (Multi-Head Attention, the original): every Q head gets its own independent set of K, V, for $n_q$ sets total. Most expressive, and the largest cache.

MQA (Multi-Query Attention) [2]: all $n_q$ Q heads share the same single set of K, V, leaving just 1 set:

The cache shrinks to $2d$, $n_q$ times smaller than MHA, but expressiveness is limited and large models applying it directly tend to lose accuracy.

GQA (Grouped-Query Attention) [3]: split the $n_q$ Q heads into $n_{kv}$ groups sharing K, V within each group — a continuous interpolation between MHA and MQA:

$n_{kv} = n_q$ degenerates to MHA, $n_{kv} = 1$ to MQA. The kernel only computes $n_{kv}$ sets of K, V and broadcasts along the group dimension when computing scores, without actually replicating tensors. This is the current default for dense and most MoE models (Llama 3, Qwen3, Mistral).

MLA (Multi-head Latent Attention) [4]: GQA only scales the cache linearly with head count; MLA instead compresses K and V jointly into a low-rank latent vector $\mathbf{c}^{KV}$, then routes RoPE through a separate, shared, small branch. In four steps:

Content branch — K and V share one down-projection, and only the latent vector $\mathbf{c}^{KV}_i$ enters the cache:

The Q side is also low-rank (saving memory during training):

Decoupled RoPE branch — on the K side, only one shared $d_r$-dim vector is computed, used by all heads:

Concatenate and compute attention (the content part and the RoPE part concatenated along the head dimension):

The key point: MLA actually caches only $\mathbf{c}^{KV}_i \in \mathbb{R}^{d_c}$ and $\mathbf{k}^{R}_i \in \mathbb{R}^{d_r}$, for $d_c + d_r$ numbers in total. At inference, the associativity of matrices folds the up-projection $W_K^{U,(h)}$ into $W_Q^{U,(h)}$:

This way the K content part never has to be reconstructed; attention is computed directly on the cached latent vectors. But RoPE’s rotation angle depends on the absolute position and cannot be folded into fixed weights — once it is applied on the reconstructed K, the folding breaks. So the positional signal must be split into an independent $d_r$-dim shallow branch.

KV dimension comparison

In one line: MHA → MQA/GQA saves params, FLOPs, and cache all at once; MLA adds a few projection stages, trading params/FLOPs for cache — shrinking the per-token cache from the KB range down to ~1 KB.

Linear attention and hybrid stacking

All four variants above are still softmax attention: $O(S^2)$ in sequence length, with a cache that grows linearly. Linear attention replaces softmax with an associative feature map $\phi$, rewriting attention as a recurrence over a “state matrix”, dropping the complexity to $O(S)$ and compressing the KV Cache into a state of fixed size, independent of sequence length:

• $S_t \in \mathbb{R}^{d_v \times d_k}$ — the state accumulated up to step $t$, equivalent to “compressing” all historical KV into one matrix
• each decode step only updates and reads $S_t$ once, no longer slowing down as the cache grows

Naive linear attention cannot remember long-range details (the state only grows and cannot be precisely rewritten). Gated DeltaNet [26] adds two things to the recurrence: a gated decay $\alpha_t \in (0,1)$ (borrowed from Mamba2, letting the state forget quickly) and a delta rule ($\beta_t$ controls precise rewriting of a specific key rather than pure accumulation):

In practice no one uses pure linear attention on its own: it is good at handling long sequences but weak at precise retrieval. The mainstream approach is hybrid stacking — most layers use linear attention to cut cost, and every few layers a full-attention layer is inserted to restore retrieval capability. The Qwen3.5 chapter below is exactly this “3 linear + 1 full” ratio.

Sparsification and compression: letting each query see only part of the KV

Full softmax lets every query see all historical KV; to save cache and FLOPs in ultra-long contexts, there are two orthogonal routes:

Compression — aggregate the KV of $m$ consecutive tokens into 1 “compressed entry” in advance, so attention runs over compressed entries and the effective sequence length shrinks to $\frac{1}{m}$:

Sparse selection — no compression, but for each query pick only the most relevant top-$k$ historical entries to attend to, skipping the rest. DeepSeek’s approach is a lightweight lightning indexer that first computes a set of cheap index scores $I_{t,s}$ per query, then takes the top-$k$:

The two routes stack: compress first, then do top-$k$ sparse selection over the compressed entries — that is DeepSeek-V4’s CSA below; push the compression rate to the max, drop sparse selection, and keep dense attention over the compressed entries — that is its HCA. Add a sliding-window branch for local detail and a set of attention sinks that let each head’s attention weights not sum to 1, and you have V4’s hybrid attention.

Normalization — LayerNorm vs RMSNorm · pre-norm

Normalization stabilizes the numerical scale of each layer’s input so that deep networks stay trainable. Standard LayerNorm does zero-mean, unit-variance over the feature dimension, then applies an affine transform:

• $\boldsymbol{\gamma}, \boldsymbol{\beta} \in \mathbb{R}^{H}$ — learnable scale / shift
• $\epsilon$ — small constant to avoid division by zero (typically $10^{-5}$)
• $\odot$ — element-wise multiplication

RMSNorm [5] (the mainstream choice for Llama / Qwen / DeepSeek) drops the mean-centering and the bias, using only root-mean-square scaling:

The denominator is the root mean square of $\mathbf{x}$, hence the name. It drops the mean and also drops $\boldsymbol{\beta}$, roughly halving both compute and parameters, and empirically this is almost lossless for quality.

Placement: pre-norm, not post-norm. The original Transformer was post-norm (norm after the residual addition), which tends to be gradient-unstable at depth; modern models all use pre-norm — the norm sits inside the residual branch, and the trunk jumps straight across:

Pre-norm makes the residual trunk a “highway” undisturbed by the norm, allowing stable training even at hundreds of layers.

Beyond the residual: hyper-connections. Pre-norm adds each layer’s output back to a single residual trunk. Hyper-Connections [27] widen this trunk into $n_{hc}$ parallel residual streams, mixing between layers with learnable (and input-dependent) coefficient matrices $A_l, B_l, C_l$:

This relaxes the “fixed depth order + single residual” constraint, easing the tug-of-war between vanishing gradients and representation collapse. DeepSeek-V4’s mHC further constrains the residual mapping $B_l$ to the manifold of doubly stochastic matrices (spectral norm $\le 1$, non-expansive mapping) in exchange for numerical stability when stacking deep — see the DeepSeek chapter below.

Positional encoding — absolute position → RoPE → length extrapolation

Attention itself is order-insensitive (shuffle the tokens and the result is unchanged), so positional information must be injected explicitly. The original Transformer used absolute positional encoding: build a sinusoidal vector for each position $m$ and add it to the embedding. The downsides are that it only encodes absolute position and extrapolates poorly to lengths never seen in training.

RoPE (Rotary Positional Embedding) [6] takes a different tack: instead of adding a vector, it lets position act on Q and K as a rotation, so the inner product automatically retains only relative position. It slices each head’s $d$ dimensions into $d/2$ two-dimensional subspaces and, at position $m$, multiplies the $k$-th coordinate pair by a 2D rotation matrix:

• $m$ — the token’s absolute position; $k \in \{0, \ldots, d/2 - 1\}$ — the 2D-subspace index
• $\theta_k$ — the angular velocity of the $k$-th subspace, with $\text{base}$ the frequency-decay base (Llama takes $10^4$, Qwen3 takes $10^6$)

Why does rotation encode relative position? Denote the whole $d$-dim rotation as the block-diagonal matrix $\mathbf{R}_m$; it is orthogonal and satisfies $\mathbf{R}_m^{\top}\mathbf{R}_n = \mathbf{R}_{n-m}$, hence:

When attention computes $QK^{\top}$, the score for each pair $(q_i, k_j)$ depends only on the difference $j - i$ — absolute position is canceled in the inner product, leaving relative position. The frequency spectrum $\theta_k = \text{base}^{-2k/d}$ gives the $d/2$ subspaces angular velocities from fast to slow: $k=0$ has period about $2\pi$ tokens handling near-neighbors, $k=d/2-1$ has period tens of thousands of tokens handling long range. RoPE acts only on Q and K, not on V.

Length extrapolation. Training only ever sees $m \le L_{\text{train}}$; once inference length exceeds this, the low-frequency subspaces enter the out-of-distribution region and attention degrades. The common fixes all modify $\theta_k$:

• Position Interpolation: $m \to m/s$, compressing all frequencies by the same ratio — simple, but it sacrifices high-frequency precision.
• NTK-aware: scale only the low frequencies while preserving the high ones, equivalent to enlarging $\text{base}$.
• YaRN [7]: handle frequency bands separately — high frequencies unchanged, low frequencies scaled à la PI, with a smooth transition in between, plus a temperature correction to offset the attention-entropy drift. The long-context extension of Llama 3.1 and Qwen3 both use it.

Activation functions and the FFN — GELU · SwiGLU · GeGLU · MoE

After attention comes the per-token feed-forward network (FFN), responsible for nonlinear transformation and knowledge storage. The classic FFN first projects up, applies an activation, then projects down:

The activation $\phi$ is an element-wise scalar nonlinearity. The mainstream choices:

• $\Phi$ is the standard-normal CDF; GELU [9] is used in GPT-2/3 and BERT
• SiLU (a.k.a. Swish) has a shape close to GELU but simpler, and is the gating activation of the Llama / PaLM line

Gated FFN: the GLU family

The real engineering inflection point is switching the activation from “wrapped on the projection” to “wrapped on the gate”. The GLU family [8] uses two independent up-projections, one as gate and one as value, multiplied element-wise:

Whichever $\phi$ you pick names the variant: $\phi = \text{GELU}$ is GeGLU (T5 v1.1), $\phi = \text{SiLU}$ is the most common SwiGLU (PaLM, Llama, Qwen, DeepSeek):

SwiGLU has one more projection than the classic FFN (three matrices vs two), and to keep the parameter budget aligned, implementations usually set the intermediate dim to $I \approx \tfrac{2}{3}\cdot 4H$.

Mixture-of-Experts (MoE)

In a dense FFN, every token passes through the same pair of weights, paying for all parameters and all FLOPs. MoE [10] replicates the FFN into $N$ “experts”, with each token selecting only the top-$k$ to compute — total parameters scale linearly while the per-token activated parameters and FLOPs stay almost unchanged, trading capacity for knowledge, not for compute:

• $W_g \in \mathbb{R}^{H \times N}$ — the router, mapping the hidden state to the logits of $N$ experts
• $\mathcal{T}_k(\mathbf{x})$ — the selected top-$k$ expert set; $g_i$ is the normalized combine weight
• each $\text{FFN}_i$ is usually just a SwiGLU, with only the weights not shared

A naive router will “collapse” onto a few hot experts, requiring load balancing. GShard/Switch use an auxiliary loss $\mathcal{L}_{\text{aux}} = \alpha N \sum_i f_i \bar{s}_i$ ($f_i$ the routing fraction, $\bar{s}_i$ the average probability) as a soft constraint; DeepSeek instead uses an auxiliary-loss-free scheme [12]: maintain a bias $b_i$ per expert, base TopK on $s_i + b_i$, raise it for underused experts and lower it for overused ones — affecting only selection, without polluting the gradient. These two pieces (fine-grained MoE, loss-free balancing) are central to the DeepSeek chapter below.

Prefill and Decoding — GEMM vs GEMV · compute vs bandwidth

Autoregressive inference splits into two phases with sharply different dimensions and bottlenecks.

Prefill: ingest all $S$ prompt tokens at once, run through every layer in parallel, and produce the first output token plus the full KV Cache. The main tensor shape is $[B, S, H]$, all matrix multiplications are matrix × matrix (GEMM), arithmetic intensity is high, and it is compute-bound, much like a single training forward pass.

Decoding: each step feeds only the 1 token generated last step, reads all historical K, V from the cache, and computes the next token. The main shape is $[B, 1, H]$, matrix multiplication degenerates to matrix × vector (GEMV), most time is spent moving weights from memory into the compute units, and it is memory-bandwidth-bound.

KV Cache shape and growth

Each layer caches a pair of tensors that keep growing as generation proceeds:

The per-token, per-layer cache size (fp16, GQA) is $2 \times n_{kv} \times d \times 2\ \text{bytes}$. With $n_{kv}=8, d=128$ that is 4 KB/layer; a 32-layer, 4096-token request is about 512 MB — under long contexts this is the main memory occupant, and the battleground for all cache optimizations like MLA, KV quantization, and PagedAttention.

The physical ceiling on throughput

Every decode step must scan the entire model’s weights once. With parameter count $P$ in fp16, the weight bytes are about $2P$, so the decode throughput ceiling for a single request ($B=1$):

An 8B model has about 16 GB of fp16 weights; on an H100 @ 3 TB/s, moving the weights alone for one token takes about $16/3000 \approx 5.3$ ms — the physical floor for single-request decode, almost independent of compute. The breakthrough is continuous batching: pack $B$ requests into one big GEMV so the same weights are shared by $B$ requests, multiplying arithmetic intensity by $B$ and raising throughput near-linearly until compute or attention bandwidth hits the wall first.

In computational complexity, having a KV Cache or not differs by three orders of magnitude. With the cache, the total for generating $T$ tokens:

The first term is the per-token linear layers (constant per decode step), the second is attention (growing linearly with the cache). Without the cache it would be $O(L(S+T)^3)$, recomputing all history each step. “The longer you generate, the slower it gets” comes precisely from $\text{cache\_len}$ growing in the second term. In practice FlashAttention [13] (fusing softmax and matmul into one kernel, cutting memory from $O(S^2)$ to $O(S)$), PagedAttention [14] (paged cache management, eliminating fragmentation), and speculative decoding [15] (a small model drafts, the large model verifies $k$ at once) all do local surgery on this same skeleton without changing the formulas.

Case study: Qwen — Qwen3.5-397B-A17B

Qwen3.5 [24] (released 2026, a flagship natively multimodal model) is Alibaba’s latest open-source generation, with flagship Qwen3.5-397B-A17B (397B total parameters, ~17B activated per token). Compared with the previous Qwen3, it switches the attention backbone: no longer uniformly full attention, but a hybrid stack of linear attention (Gated DeltaNet) and full attention. Broken down into the five modules (language backbone only):

Key architecture numbers (from the official config): 60 layers, hidden $=4096$; full-attention layers with $n_q=32$, $n_{kv}=2$, head_dim $=256$ (decoupled from hidden); linear-attention layers with 16 key heads, 64 value heads, head dim $128$, conv kernel width $4$; expert intermediate dim $=1024$, vocabulary $=248320$ (byte-level BPE, expanded from Qwen3’s ~150k to ~250k).

Three module-level highlights:

• Hybrid attention: 3 of every 4 layers use Gated DeltaNet — an $O(S)$ state recurrence whose cache is a fixed-size state matrix, absorbing the cost of long sequences; every 4th layer inserts full attention to restore precise retrieval. The full-sequence attention cost thus grows near-linearly rather than quadratically. See the earlier “Linear attention and hybrid stacking” section for the formulas.
• Gated attention: the full-attention layer’s output passes through a per-channel sigmoid gate $\mathbf{o}' = \mathbf{o} \odot \sigma(\mathbf{x} W_g)$, suppressing attention sinks and massive activations to stabilize large-model training.
• Fine-grained MoE + shared expert: each token selects 10 of 512 routed experts, plus 1 shared expert that every token passes through — compared with Qwen3’s “no shared expert”, Qwen3.5 shifts to a DeepSeek-style shared-expert configuration.

In one line: Qwen3.5 = “linear/full hybrid attention + gated attention + RoPE + SwiGLU + fine-grained MoE (with sharing)”, pushing the cost of long context down on the attention side.

Case study: DeepSeek — DeepSeek-V4-Pro / Flash

DeepSeek-V4 [25] (April 2026, preview) gives two MoE models: V4-Pro (1.6T total / 49B activated) and V4-Flash (284B / 13B activated), both natively supporting a 1-million-token context. Compared with V3 [18] / V3.2 [19], it keeps DeepSeekMoE and MTP but switches to new designs in attention and residuals.

Hybrid attention: CSA + HCA (replacing MLA). V3’s signature was MLA (compressing K, V into low-rank latent vectors); V4 goes further, dropping MLA and instead twisting the “compression” and “sparsity” routes (the earlier “Sparsification and compression” section) into two interleavable attention layer types:

• CSA (Compressed Sparse Attention): first compress every $m=4$ tokens’ KV into 1 entry, then use the lightning indexer to select only the top-$k$ compressed entries per query for attention (Pro $k=1024$, Flash $k=512$) — compression × sparsity stacked.
• HCA (Heavily Compressed Attention): push the compression rate to $m'=128$ (far above CSA), but no longer sparse-select, keeping dense attention over the compressed entries.

The core attention uses shared-KV MQA (compressed entries serve as both K and V), the Q side goes through low-rank compression ($d_c$: Pro $1536$, Flash $1024$), plus an $n_{\text{win}}=128$ sliding-window branch for local dependencies and a set of learnable attention sinks that let each head’s attention weights not sum to 1. RoPE acts only on the last 64 dims of each vector. KV storage uses mixed precision: BF16 for the RoPE dims, FP8 for the rest, cutting the cache nearly in half again. The result: at 1M context V4-Pro needs only ~27% of V3.2’s per-token FLOPs and 10% of its KV cache, and Flash drops further to ~10% FLOPs and 7% KV cache; relative to a BF16 GQA (head_dim 128) baseline, KV cache is squeezed to about 2%.

DeepSeekMoE (fine-grained + shared experts) [11] follows V3’s approach with tweaked configuration: MoE now fills every Transformer layer (no more dense FFNs), the first 3 MoE layers switch to hash routing (assigning experts by a fixed hash of the token id); the routing scoring function changes from V3’s sigmoid to $\text{Sqrt}(\text{Softplus}(\cdot))$; load balancing still uses the auxiliary-loss-free bias method [12], plus a weak sequence-level balancing loss. Pro has 1 shared + 384 routed experts per layer (intermediate dim $3072$), Flash has 1 shared + 256 routed experts (intermediate dim $2048$), both activating 6 routed experts per token.

Two more modules:

• mHC (Manifold-Constrained Hyper-Connections): widen the single residual trunk into $n_{hc}=4$ parallel residual streams, and constrain the residual-mixing matrix to the manifold of doubly stochastic matrices (spectral norm $\le 1$, non-expansive), trading for numerical stability when stacking deep. See the earlier “Beyond the residual” passage for the principle.
• MTP (Multi-Token Prediction): a depth-1 shallow module predicts the “token after next”, giving the backbone a denser supervision signal during training and serving directly as a speculative-decoding draft at inference to boost throughput — same as V3.

Architecture numbers: V4-Pro has 61 layers, hidden $=7168$; V4-Flash has 43 layers, hidden $=4096$; vocabulary $128\text{K}$; both trained in FP8, with post-training FP4 quantization of the MoE expert weights and indexer QK. Placing the three models side by side: Qwen3.5 takes “linear/full hybrid attention + fine-grained MoE”, DeepSeek-V4 takes “compression+sparsity hybrid attention (CSA/HCA) + fine-grained MoE + mHC + MTP” — the same skeleton, different trade-offs.

Summary — five modules, one skeleton

Decoder-only LLM inference is, at bottom, the combination of five modules: attention (who sees whom), normalization (stabilizing values), positional encoding (encoding order), FFN/MoE (storing knowledge), and prefill/decode (scheduling). Each module has one main formula, and the variants just swap in a replacement at one spot — GQA/MLA changes the attention KV structure, RMSNorm changes normalization, RoPE/YaRN changes positional encoding, SwiGLU/MoE changes the FFN. Once you understand these five formulas, the difference between Qwen3.5 and DeepSeek-V4 is nothing but a different selection table on the same skeleton; and any inference-optimization paper you read afterward can be located to whichever module it touches.

References — papers · technical reports

• Vaswani et al. Attention Is All You Need (NeurIPS 2017). arxiv.org/abs/1706.03762
• Shazeer. Fast Transformer Decoding: One Write-Head is All You Need (2019) — MQA. arxiv.org/abs/1911.02150
• Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023). arxiv.org/abs/2305.13245
• DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) — first introduces MLA. arxiv.org/abs/2405.04434
• Zhang & Sennrich. Root Mean Square Layer Normalization (NeurIPS 2019) — RMSNorm. arxiv.org/abs/1910.07467
• Su et al. RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) — RoPE. arxiv.org/abs/2104.09864
• Peng et al. YaRN: Efficient Context Window Extension of Large Language Models (2023). arxiv.org/abs/2309.00071
• Shazeer. GLU Variants Improve Transformer (2020) — SwiGLU / GeGLU. arxiv.org/abs/2002.05202
• Hendrycks & Gimpel. Gaussian Error Linear Units (GELUs) (2016). arxiv.org/abs/1606.08415
• Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017). arxiv.org/abs/1701.06538
• DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024) — fine-grained + shared experts. arxiv.org/abs/2401.06066
• Wang et al. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts (2024). arxiv.org/abs/2408.15664
• Dao et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS 2022). arxiv.org/abs/2205.14135
• Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — vLLM. arxiv.org/abs/2309.06180
• Leviathan et al. Fast Inference from Transformers via Speculative Decoding (ICML 2023). arxiv.org/abs/2211.17192
• Qwen Team. Qwen3 Technical Report (2025). arxiv.org/abs/2505.09388
• Qwen Team. Qwen2.5 Technical Report (2024). arxiv.org/abs/2412.15115
• DeepSeek-AI. DeepSeek-V3 Technical Report (2024) — MLA + MoE + MTP + FP8. arxiv.org/abs/2412.19437
• DeepSeek-AI. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention (2025). github.com/deepseek-ai/DeepSeek-V3.2-Exp
• Meta AI. The Llama 3 Herd of Models (2024). arxiv.org/abs/2407.21783
• Jiang et al. Mixtral of Experts (Mistral AI, 2024). arxiv.org/abs/2401.04088
• Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022). arxiv.org/abs/2101.03961
• Adam Casson. Transformer Inference Arithmetic — back-of-envelope inference FLOPs / KV cache. kipp.ly
• Qwen Team. Qwen3.5-Omni Technical Report (2026) — Gated DeltaNet + gated full-attention hybrid. arxiv.org/abs/2604.15804
• DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (2026) — CSA/HCA + mHC. arxiv.org/abs/2606.19348
• Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025). arxiv.org/abs/2412.06464
• Zhu et al. Hyper-Connections (ICLR 2025, ByteDance). arxiv.org/abs/2409.19606