Low-Precision Data Formats in Large Language Models

If you had to summarize the hardware trend in large-model training and inference over the past three years in one sentence — training is moving from BF16 to FP8, and inference is moving from FP8 to FP4. Every time precision drops one notch, the same silicon area fits twice as many multiply-accumulate (MAC) units, and training throughput and inference tokens/s double outright. Underneath this main line sits a whole spectrum of data formats: FP32 / TF32 / FP16 / BF16 / FP8 (two variants) / FP6 (two variants) / FP4 (MX and NV versions) / INT8 / INT4.

A lot of people’s first reaction to low precision is “it saves memory.” That’s only half right. Saving memory certainly matters — BF16 weights are half the size of FP32, INT4 weights are a quarter of FP16, and for long-context inference the KV cache can take up even more room than the weights. But what really drove NVIDIA, AMD, and Google to pile into low precision is four dividends cashing in at once:

• Memory: half the bit width means the same card holds a bigger model or a longer context.
• Bandwidth: half the bytes moved from memory to the compute units for weights and activations — and the decode phase of LLM inference is almost entirely memory-bandwidth bound.
• Compute: each Tensor Core generation roughly doubles its peak throughput at lower precision — halve the bit width, double the MAC count on the same-generation silicon.
• Cost: the three above stack up and directly drive down the dollar cost per token of training and inference.

Of these, compute is the one most easily underestimated. Line up a few generations of flagship cards and the pattern is unmistakable[13][14]:

Halve the bit width and compute roughly doubles. So low precision isn’t just “the same model uses less memory,” it’s “the same silicon runs faster” — and that is the fundamental reason this road keeps going.

But “precision moving down” has never been as simple as “swap the whole model to one format.” Within a single Transformer block, matrix multiplication may run in FP8, Softmax in FP32, and the KV cache may be squeezed to INT8 — precision is mixed by operation type, not by layer.

This article lays out that spectrum in four steps: first filling in how floating-point and integers are actually represented in bits, then taking each format apart one by one, then using Qwen’s latest generation to see how a real model picks precision, and finally hardware support. Throughout, it covers only formats with native Tensor Core / matrix-engine hardware support — pure software emulation or paper-only formats (INT2, binarization, posits, etc.) are out of scope.

Numeric representation

To understand why BF16 suits training while FP4 is only usable with a bolt-on scale, we have to go back down to “how a number is actually stored in a computer.” Every trade-off in a low-precision format is, at bottom, a re-slicing of two pieces of a fixed bit budget: range and precision.

Floating-point

Any IEEE 754-style floating-point number is made of three fields — sign, exponent, and mantissa[1]:

• S (sign): 1 bit, sets positive vs negative.
• E (exponent): $e$ bits total, storing the “biased” exponent. The real exponent is $E - \text{bias}$, where $\text{bias} = 2^{e-1} - 1$. A bias rather than two’s complement is used so that comparing floats bitwise matches the result of comparing integers.
• M (mantissa): $m$ bits total. A normal number has an implicit leading “1.” before the point, so the mantissa carries $m+1$ bits of significance but costs only $m$ bits to store — that leading bit is free.

A few special cases are worth calling out:

• Normalized: $0 < E < 2^e - 1$, with the implicit leading 1 — how the vast majority of values are represented.
• Subnormal / denormal: when $E = 0$, the leading bit becomes 0 and the value is $(0.M)_2 \times 2^{\,1-\text{bias}}$. This fills the gap around 0, making underflow gradual rather than a cliff-drop straight to zero.
• Special values: when $E = 2^e - 1$ (all ones), the value is $\pm\infty$ or NaN.

The key takeaway: a floating-point format’s entire personality is fixed uniquely by three numbers — total bits, $e$, and $m$. Specifically,

• Dynamic range is set by $e$. The larger $e$, the more orders of magnitude separate the largest and smallest representable numbers.
• Precision is set by $m$. Floats are distributed geometrically, not evenly — relative precision is roughly constant at the machine epsilon $2^{-m}$, independent of magnitude. Each extra mantissa bit halves the relative error.

At a fixed total width, $e$ and $m$ trade off against each other: give the exponent one more bit and the mantissa loses one. “Range-heavy” vs “precision-heavy” is the first-principles question behind every low-precision floating-point format — as we’ll see, BF16 and FP16 are both 16-bit and simply spend that budget in different places.

Integers

An integer format can be seen as the special case “all mantissa, zero exponent” — values are spread evenly and uniformly across a fixed interval, with none of floating-point’s geometric grading.

• An unsigned $n$-bit integer represents $0 \sim 2^n - 1$; a signed one uses two’s complement for $-2^{n-1} \sim 2^{n-1}-1$.
• A fixed-point number is just “an integer plus an agreed, fixed binary-point position,” equivalent to an integer times a constant scale.

An integer has no fractional part and no cross-magnitude range on its own, so to represent the small-yet-wide-ranging weights of a neural network it must bolt on a floating-point scale factor that “stretches” the integer interval onto the target numeric range. That is the core formula of quantization:

where $q$ is the stored integer, $s$ is an FP16 / FP32 scale factor, and $z$ is the zero point:

• Symmetric quantization: $z = 0$, with the integer interval symmetric about 0. Simple to implement, well-suited to roughly symmetric weight distributions.
• Asymmetric quantization: $z \neq 0$, shifting the whole interval to hug a one-sided distribution (for example, activations that are all positive after a ReLU).

The granularity of the scale sets quantization quality: one scale per whole tensor (per-tensor) is cheapest, one per column/channel (per-channel) is more accurate, and one per group of 32 / 64 / 128 numbers (per-group) is the most accurate but carries the most metadata overhead. The lower the bit width, the finer the granularity needed to contain the error — the same rule holds on the floating-point side, just under a different name: microscaling.

Common data precisions

First, lay a few representative floating-point formats out at the same scale, where each cell is one bit. Exponent width sets dynamic range and mantissa width sets precision, and the trade-off between these two axes is the heart of every format’s design:

Now each format taken apart in turn. We’ll write ExMy for $x$ exponent bits and $y$ mantissa bits.

FP32

• Width: 1 + 8 + 23 = 32 bits (E8M23).
• Range and precision: bias = 127, dynamic range roughly $1.2\times10^{-38} \sim 3.4\times10^{38}$, about 7 decimal significant digits.
• Use: the baseline format for every modern CPU / GPU, and the default precision for the accumulator, master weights, and optimizer state in low-precision training.
• Trade-off: precision and range are both ample, but each number takes 4 bytes and is slow to compute. In LLMs it mostly retreats backstage, handling the “safety net” rather than the main compute.

TF32

• Width: 1 + 8 + 10 = 19 effective bits, stored in a 32-bit register (E8M10).
• Range and precision: the exponent is exactly FP32’s (8 bits) and the mantissa is trimmed to FP16’s (10 bits). Dynamic range equals FP32, precision only about 3 decimal digits.
• Use: an engineering trick NVIDIA introduced on the A100 as the default replacement for FP32 GEMMs — because its dynamic range equals FP32, it accelerates training without changing a line of code and without loss scaling.
• Trade-off: the name says “32” but it’s really only 19 bits. Trading precision for speed, it’s the first step down from FP32 toward low precision.

FP16

• Width: 1 + 5 + 10 = 16 bits (E5M10), IEEE 754 half precision.
• Range and precision: bias = 15, dynamic range roughly $6\times10^{-5} \sim 65504$, about 3–4 decimal digits.
• Use: early mixed-precision training and a large body of inference scenarios.
• Trade-off: plenty of mantissa and decent precision, but its dynamic range is too narrow for training — once gradients shrink to the $10^{-7}$ scale they underflow to zero, so it must be paired with loss scaling (scaling the loss up to a non-underflowing magnitude before backprop). That pain point is exactly what gave rise to BF16.

BF16

• Width: 1 + 8 + 7 = 16 bits (E8M7), proposed by Google Brain in the TPU v2 era.
• Range and precision: it keeps all 8 of FP32’s exponent bits and trims the mantissa to 7. Dynamic range is essentially identical to FP32, precision about 2–3 decimal digits (even lower than FP16).
• Use: the de facto standard for large-model training after 2020.
• Trade-off: dynamic range equal to FP32 → training no longer needs loss scaling and BF16 is almost a painless drop-in for FP32; FP32 ↔ BF16 is just truncation/zero-padding → hardware conversion is essentially free. The cost is lower precision, but large-model training is far more sensitive to dynamic range than to precision — a lesson that is the direct prototype for the FP8 design that follows.

FP8 (E4M3 / E5M2)

FP8 was jointly proposed by NVIDIA / Arm / Intel in 2022[2], defining two variants at once, because activations and gradients have very different dynamic ranges:

• E4M3 is precision-heavy, range-light, well-suited to the activations that cluster in a tight range after a LayerNorm. It doesn’t strictly follow IEEE 754: it has no inf, repurposing the encoding that would have been inf to extend the numeric range (max value 448 rather than 240).
• E5M2 is range-heavy, precision-light, used to contain gradients that can span several orders of magnitude. It strictly follows IEEE 754, with both inf and NaN.
• Scaling: 8 bits alone come nowhere near covering the numeric range of training, so FP8 almost always carries a per-tensor FP32 scale (auto-maintained by the H100’s Transformer Engine[15]), the actual value being $\text{FP8\_value} \times \text{scale}$.

FP6

• Width: 6 bits, part of the OCP microscaling spec[4], with two variants: E3M2 (more range, less precision) and E2M3 (less range, more precision).
• Use: a middle rung between FP8 and FP4, mainly for inference. It fills the slot where FP8 is not thrifty enough but the precision drop of FP4 is unacceptable.
• Trade-off: like FP4, it belongs to the microscaling family — a single value expresses little and it must share a per-block scale to be usable. Native hardware support starts only with Blackwell.

FP4 (MXFP4 / NVFP4)

FP4 has only 4 bits (E2M1) and can represent just 16 distinct values (including signed zeros). On its own it’s completely unusable — it runs only thanks to microscaling: pack a group of numbers and share one outer scale. Tensor Cores support two versions:

• MXFP4 has every 32 numbers share one pure-exponent E8M0 scale ($2^E$, no sign, no mantissa), so a block value = $\text{FP4\_value} \times 2^E$. A pure-exponent scale is used so that scaling degenerates into a bit shift in hardware — essentially free.
• NVFP4 shrinks the block to 16, swaps the block scale for the finer FP8 (E4M3), and wraps an FP32 per-tensor scale around the outside — a three-tier scale structure. This makes it noticeably more accurate than MXFP4 for Blackwell inference, at the cost of a bit more metadata. In NVIDIA’s NVFP4 pretraining experiments the relative error was about 1.5%, versus about 2.5% for MXFP4[5].
• Use: the workhorse low-precision format for inference after Blackwell, now also entering pretraining.

INT8

• Width: 8-bit signed integer ($-128 \sim 127$), paired with an FP16 / FP32 scale.
• Range and precision: values are evenly spaced; the dynamic range is set entirely by the external scale.
• Use: the most mature quantization format in LLM inference, where a single per-tensor or per-channel scale is usually enough. Also common for W8A8 (both weights and activations in 8 bits) and KV-cache compression.
• Trade-off: the hard part is outliers — a few extreme activation values blow up the scale and drown out the rest. LLM.int8()[6] pulls the outliers out to handle separately, while SmoothQuant[7] “migrates” activation outliers into the weights to make W8A8 workable.

INT4

• Width: 4-bit signed integer ($-8 \sim 7$), only 16 values.
• Range and precision: group-wise scaling is essentially mandatory — every 32 / 64 / 128 numbers share a scale, or the precision loss is too great.
• Use: the workhorse for weight quantization on the consumer / local-deployment side. GPTQ[8] uses second-order information to solve for the optimal per-layer scale, AWQ[9] picks group-wise scales by activation magnitude — the default path for local stacks like llama.cpp / Ollama.
• Trade-off: 4× compression that can run large models, but it collapses without a suitable group scale. It and FP4 are “two philosophies at the same bit width” — INT4 leans on dense grouping with an external scale, FP4 on a built-in 2-bit exponent plus a block scale.

Rolling nine floating-point formats and two integer formats into one comparison table:

Now stack every floating-point format’s dynamic range on the same log axis, and the “range-heavy vs precision-heavy” split is visible at a glance:

Qwen as a case study

That was all abstract formats — what does it look like on a real model? Alibaba’s Qwen3.5 series, released in February 2026, is the best case to look at — it’s the current open-weights generation with the widest precision coverage[10].

Qwen3.5 spans from 0.8B all the way up to the flagship Qwen3.5-397B-A17B: 397 billion total parameters, 17 billion activated per token, 512 experts, 60 layers, a hybrid structure interleaving Gated DeltaNet linear attention and full softmax attention at roughly 3:1, with native 262K context. For a MoE this large, the precision choice directly decides whether it can even be deployed.

The multi-precision release matrix

What makes Qwen3.5 most illustrative is that the same model ships in multiple precisions of weights, from the official team or partners, forming a “release matrix”:

• BF16: the original weights. Training and alignment were done at this precision, and it’s the reference baseline for every quantized version.
• Official FP8: fine-grained FP8 quantization with a block size of 128 (exactly the per-group-scale idea from earlier, carried over to the floating-point side). The key engineering detail is that the shared expert and the attention layers stay at 16 bits, unquantized, and only the remaining experts’ weights are compressed. Measured accuracy is nearly indistinguishable from BF16[12].
• Official GPTQ INT4: 4-bit weight quantization, likewise keeping the shared expert and attention at 16 bits. Suited to memory-tight deployments.
• Community / partner AWQ, NVFP4, GGUF: AWQ takes the activation-aware group-wise INT4 route; NVIDIA directly released Qwen3.5-397B-A17B-NVFP4, compressing weights to 4-bit FP4 with a block-level FP8 scale, purpose-built for Blackwell inference; GGUF serves the llama.cpp local stack.

A systematic evaluation of Qwen quantization offers several practical conclusions[11]:

• With thinking off, FP8 / INT4 / NVFP4 essentially tie BF16 on MMLU — 4× compression basically for free.
• Quantizing the shared expert drops accuracy markedly — exactly why the official FP8 / INT4 both deliberately keep the shared expert at 16 bits, confirming that “precision is mixed by operation/component, not applied uniformly.”
• With thinking on, small models (9B class) can drop by as much as 33% on some benchmarks — but not entirely from precision loss: the quantized versions tend to generate longer reasoning chains and hit the sequence-length cap, getting truncated. Models at 27B and above are fairly robust to quantization.

FP8 training in practice

Qwen3.5 dares to ship FP8 as a first-class citizen because DeepSeek-V3 cleared the path at the end of 2024[3]. Before it, the field broadly doubted that FP8 — with so many fewer bits than FP16 — could stably train an ultra-large model at all. DeepSeek-V3’s key contribution was fine-grained quantization — 1×128 tile-wise grouping for activations and 128×128 block-wise grouping for weights, using finer scale granularity to fight outliers, ultimately training a 671-billion-parameter MoE stably in FP8.

That “block-level scale + keep sensitive components at high precision” approach is exactly what Qwen3.5’s official FP8 continues, with its block size of 128 and its shared expert and attention held at 16 bits. From format design (FP8’s per-tensor scale) to training framework (DeepSeek-V3’s tile/block grouping) to release matrix (Qwen3.5’s multi-precision weights), it’s the same “fine-grained scale” logic landing at each layer.

Inference, meanwhile, is cleanly “dual-track”: the server side (vLLM, TensorRT-LLM, SGLang) runs FP8 to save memory and run faster[17]; the local side runs INT4 AWQ / GPTQ, because FP4 hardware is still scarce on consumer cards; and as Blackwell rolls out, NVFP4 is becoming the new server-side FP4 option. Qwen3.5’s release matrix happens to cover all three tracks.

Hardware support

The real reason low precision can “take off” is hardware acceleration — without native Tensor Core / matrix-engine support, low precision only saves memory, not time. A format being “natively supported” in hardware means the Tensor Core has multiply-accumulate circuitry at that bit width, plus circuitry to decode the block scale automatically, so it can consume the format directly for matrix multiply rather than first converting up to higher precision in registers.

NVIDIA

• Volta (2017) introduced the Tensor Core, supporting FP16 matrix multiply + FP32 accumulation, kicking off mixed-precision training.
• Ampere (2020) added BF16 and TF32, making the “painless FP32 replacement” possible, and pushed INT8 inference into the mainstream.
• Hopper (2022) introduced the first-generation Transformer Engine and native FP8 Tensor Cores (E4M3 / E5M2), auto-maintaining the per-tensor scale. This is the hardware starting point for FP8 training and inference.
• Blackwell (2024) brought the second-generation Transformer Engine, centered on micro-tensor scaling — block-level scale-decoding circuitry built into the hardware, which is what gives it native FP4 and FP6 support, covering both MXFP4 (block 32, E8M0 scale) and NVFP4 (block 16, E4M3 scale)[13]. On the same card, FP4 Tensor Core throughput is roughly 2× that of FP8[16].
• Blackwell Ultra takes attention-layer acceleration to 2× and overall AI compute up about 1.5×, aimed squarely at long context and MoE.

Microscaling formats need hardware cooperation because the flow “decode a scale every 16 or 32 elements, then do a 4-bit multiply-accumulate” would, if emulated in software, eat up all the time low precision saved. Only by welding the scale-decoding circuitry into the Tensor Core does FP4 actually run faster than FP8. That’s why FP4 became “usable” only with Blackwell rather than back in 2022.

Other accelerators

• AMD: the Instinct MI300 (2023) supports FP8, with format coverage roughly matching NVIDIA’s H100 — about one generation behind; the MI350 (2025) only then adds FP6 / FP4.
• Google TPU: a deliberately “restrained” path — long betting solely on BF16 + INT8 and skipping FP8 / FP4. Because Google is both the hardware designer and the model designer, it can let “software bend to hardware” and validate on its own workloads that BF16 + INT8 is already enough. Gemini’s training stack being built around BF16 is the direct result of that choice.
• Intel Gaudi and various inference ASICs: mostly sit at the FP8 / INT8 tier, with FP4 still the preserve of high-end NVIDIA / AMD GPUs.

One pattern runs throughout: FP32 / FP16 / INT8 are the three universally supported base of the stack — code written in them deploys to any accelerator with no trouble; the further down you go (FP8 → FP6 → FP4) the more it locks in to specific vendors and generations. Choosing a precision isn’t just choosing a precision, it’s choosing how deep your hardware lock-in goes.

Summary

Boiling the whole article down to a few lines:

• The low-precision dividend is four-in-one — memory, bandwidth, compute, and cost all fall at once, and of these “halve the bit width, double the compute” is the fundamental reason the road keeps going.
• A format’s personality is set by three numbers — total width, exponent bits $e$ (range), and mantissa bits $m$ (precision). Training prefers more exponent (BF16, FP8-E5M2) to contain the gradient range; inference prefers more mantissa (FP8-E4M3, FP6-E2M3) to squeeze precision out of a known distribution.
• The lower the bit width, the more it depends on an external scale — FP8 can still get by per-tensor, FP6 / FP4 must go block-level microscaling (MXFP4 / NVFP4), and INT4 must go group-wise. Integers and 4-bit floats are “two philosophies at the same bit width” converging on the same “fine-grained scale.”
• Precision is mixed by operation and component — the official Qwen3.5 FP8 / INT4 both deliberately keep the shared expert and attention at 16 bits, exactly this rule borne out on a real model.
• Each step is gated by a hardware generation — FP8 waited for Hopper, FP4 for Blackwell. When a format becomes “usable” depends on when the Tensor Core welds in its multiply-accumulate and scale-decoding circuitry.

To close in one line: training is on the BF16 → FP8 road, inference is on the FP8 → FP4 road, and Qwen3.5’s weight matrix — shipping BF16 / FP8 / INT4 / NVFP4 all at once — is living proof that this migration is happening right now.

References — Standards · Papers · Engineering reports

• IEEE. IEEE Standard for Floating-Point Arithmetic (IEEE 754-2019) — the authoritative definition of sign / exponent / mantissa, normalized and subnormal numbers, and the biased exponent. standards.ieee.org
• NVIDIA / Arm / Intel. FP8 Formats for Deep Learning (2022) — the design motivation for the E4M3 / E5M2 variants and their training experiments. arxiv.org/abs/2209.05433
• DeepSeek-AI. DeepSeek-V3 Technical Report (2024) — the first ultra-large-scale FP8 training report, detailing fine-grained tile-wise / block-wise scaling and outlier handling. arxiv.org/abs/2412.19437
• Open Compute Project. OCP Microscaling Formats (MX) Specification v1.0 (2023) — the official spec for MXFP4 / MXFP6 / MXFP8 and the E8M0 block scale. opencompute.org · MX v1.0 spec
• NVIDIA. Pretraining Large Language Models with NVFP4 (2025) — the NVFP4 block structure and its convergence/error comparison against MXFP4. arxiv.org/abs/2509.25149
• Dettmers et al. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022) — the first stable use of INT8 at large-model scale, finding and handling activation outliers. arxiv.org/abs/2208.07339
• Xiao et al. SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs (2022) — “migrating” activation outliers into the weights to make W8A8 quantization viable. arxiv.org/abs/2211.10438
• Frantar et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022) — INT4 weight quantization via layer-by-layer optimal-scale solving based on second-order information. arxiv.org/abs/2210.17323
• Lin et al. AWQ: Activation-aware Weight Quantization (2023) — group-wise scale selection based on activation magnitude. arxiv.org/abs/2306.00978
• Qwen Team. Qwen3.5-397B-A17B — the flagship MoE’s scale, hybrid attention structure, and multi-precision release. huggingface.co · Qwen3.5-397B-A17B
• Benjamin Marie. Qwen3.5 Quantization: Similar Accuracy, More Thinking (2026) — a systematic evaluation of FP8 / INT4 / NVFP4 against BF16, with the shared-expert finding. kaitchup.substack.com
• Qwen Team. Qwen3.5-397B-A17B-FP8 — the model card for fine-grained FP8 quantization at block size 128. huggingface.co · Qwen3.5-397B-A17B-FP8
• NVIDIA. NVIDIA Blackwell Architecture — second-generation Transformer Engine, micro-tensor scaling, MXFP4 / NVFP4, and FP4 / FP6 Tensor Cores. nvidia.com · Blackwell
• NVIDIA. NVIDIA Hopper Architecture Whitepaper (2022) — first-generation Transformer Engine, FP8 Tensor Cores, per-tensor scaling mechanism.
• NVIDIA. Transformer Engine — the open-source implementation of FP8 and FP4 training/inference on Hopper / Blackwell, and the delayed-scaling mechanism. github.com/NVIDIA/TransformerEngine
• Luo et al. Microbenchmarking NVIDIA’s Blackwell Architecture (2025) — an architecture-level analysis of measured Tensor Core throughput across precisions on Blackwell. arxiv.org/abs/2512.02189
• vLLM. Quantization — open-source implementation details for INT8 / FP8 / INT4 inference quantization. docs.vllm.ai · Quantization