llm.export — Model Export

Export models to ONNX, TorchScript, and GGUF formats. Supports custom export backends registered via the llm.export_backends entry point group.

Overview

Format	Status
ONNX	Reference implementation, stable API
TorchScript	Entry-point registered
GGUF	Entry-point registered (v1: F16/F32/Q4_0/Q8_0/Q2_K..Q6_K, ADR-011)
Custom	Via EXPORT_REGISTRY plugin points

Export Registry

registry

Export backend registry and bootstrap.

Mirrors the generation/registry.py pattern so third-party export targets (e.g. torch.compile, vLLM, TensorRT-LLM, torch.export, OpenVINO) can plug in via the llm.export_backends setuptools entry-point group without forking export/.

Built-in targets

onnx — wraps the existing export_to_onnx function. This is the canonical reference implementation; the entry-point load raises if a plugin claims the same name, which is intentional (the built-in is the source of truth). torchscript / gguf — registered via the llm.export_backends entry-point group in pyproject.toml.

Usage::

>>> from llm.export.registry import export_model
>>> export_model("onnx", model, "out.onnx", input_shape=(1, 32))  # doctest: +SKIP

Plugin authors can register a target via pyproject.toml:

[project.entry-points."llm.export_backends"]
my_target = "my_pkg.exporters:build_my_target"

build_onnx_exporter

build_onnx_exporter(model, output_path, **kwargs)

Factory for the built-in ONNX export target.

Thin wrapper over :func:llm.export.onnx.export_to_onnx so the registry contract ((model, output_path, **kwargs) -> Path) matches every other target. The wrapper exists purely so the registry doesn't have to special-case keyword forwarding for ONNX's wider surface (opset_version, dynamic_axes, verbose, ...).

源代码位于： src/llm/export/registry.py

def build_onnx_exporter(
    model: nn.Module,
    output_path: str | Path,
    **kwargs: Any,
) -> Path:
    """Factory for the built-in ONNX export target.

    Thin wrapper over :func:`llm.export.onnx.export_to_onnx` so the
    registry contract (``(model, output_path, **kwargs) -> Path``)
    matches every other target. The wrapper exists purely so the
    registry doesn't have to special-case keyword forwarding for
    ONNX's wider surface (``opset_version``, ``dynamic_axes``,
    ``verbose``, ...).
    """
    from llm.export.onnx import export_to_onnx

    return export_to_onnx(model, output_path, **kwargs)

ensure_exporters_registered

ensure_exporters_registered()

Idempotently register built-in exporters and load entry points.

Built-ins are registered BEFORE the entry-point load so a plugin that claims onnx raises loudly — the built-in is the reference implementation. This matches the convention in generation/registry.ensure_backends_registered.

源代码位于： src/llm/export/registry.py

def ensure_exporters_registered() -> None:
    """Idempotently register built-in exporters and load entry points.

    Built-ins are registered BEFORE the entry-point load so a plugin
    that claims ``onnx`` raises loudly — the built-in is the
    reference implementation. This matches the convention in
    ``generation/registry.ensure_backends_registered``.
    """
    global _exporters_registered
    if _exporters_registered:
        return

    # Double-checked locking: fast guard above, re-check inside the lock
    # so two threads cold-starting concurrently can't double-register
    # (the second ``register("onnx")`` would raise) (RIL ISS-119).
    with _exporter_registration_lock:
        if _exporters_registered:
            return

        EXPORT_REGISTRY.register("onnx", build_onnx_exporter)
        load_entry_point_registry("llm.export_backends", EXPORT_REGISTRY)
        _exporters_registered = True

export_model

export_model(name, model, output_path, **kwargs)

Resolve a registered export target and run it.

参数：

名称	类型	描述	默认
name	str	Registered export target name (e.g. "onnx").	必需
model	Module	The model to export.	必需
output_path	str | Path	Where to write the artifact.	必需
**kwargs	Any	Target-specific kwargs forwarded to the factory.	{}

返回：

类型	描述
Path	The resolved output path.

引发：

类型	描述
ValueError	If name is not in :data:EXPORT_REGISTRY.

源代码位于： src/llm/export/registry.py

def export_model(
    name: str,
    model: nn.Module,
    output_path: str | Path,
    **kwargs: Any,
) -> Path:
    """Resolve a registered export target and run it.

    Args:
        name: Registered export target name (e.g. ``"onnx"``).
        model: The model to export.
        output_path: Where to write the artifact.
        **kwargs: Target-specific kwargs forwarded to the factory.

    Returns:
        The resolved output path.

    Raises:
        ValueError: If ``name`` is not in :data:`EXPORT_REGISTRY`.
    """
    ensure_exporters_registered()
    return EXPORT_REGISTRY.get(name)(model=model, output_path=output_path, **kwargs)

ONNX Export

onnx

export_to_onnx

export_to_onnx(model, output_path, input_shape=(1, 32), opset_version=17, dynamic_axes=None, verbose=False)

Export a model to ONNX format.

参数：

名称	类型	描述	默认
model	Module	The model to export (e.g., DecoderModel)	必需
output_path	str | Path	Path to save the ONNX file	必需
input_shape	tuple[int, int]	(batch_size, seq_len) for dummy input	(1, 32)
opset_version	int	ONNX opset version (default: 17)	17
dynamic_axes	dict | None	Dynamic axes for variable-length inputs	None
verbose	bool	Print export details	False

返回：

类型	描述
Path	Path to the exported ONNX file

Example::

>>> model = DecoderModel(vocab_size=1000, hidden_size=64, num_layers=2, num_heads=4)  # doctest: +SKIP
>>> export_to_onnx(model, "model.onnx", input_shape=(1, 32))  # doctest: +SKIP

源代码位于： src/llm/export/onnx.py

def export_to_onnx(
    model: nn.Module,
    output_path: str | Path,
    input_shape: tuple[int, int] = (1, 32),
    opset_version: int = 17,
    dynamic_axes: dict | None = None,
    verbose: bool = False,
) -> Path:
    """
    Export a model to ONNX format.

    Args:
        model: The model to export (e.g., DecoderModel)
        output_path: Path to save the ONNX file
        input_shape: (batch_size, seq_len) for dummy input
        opset_version: ONNX opset version (default: 17)
        dynamic_axes: Dynamic axes for variable-length inputs
        verbose: Print export details

    Returns:
        Path to the exported ONNX file

    Example::

        >>> model = DecoderModel(vocab_size=1000, hidden_size=64, num_layers=2, num_heads=4)  # doctest: +SKIP
        >>> export_to_onnx(model, "model.onnx", input_shape=(1, 32))  # doctest: +SKIP
    """
    output_path = Path(output_path)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    model.eval()
    device = next(model.parameters()).device

    # Wrap model to fix use_cache=False (avoids TracerWarning)
    wrapped_model = ExportCacheWrapper(model)
    wrapped_model.eval()

    # Create dummy input — bounded by the REAL vocab so small-vocab models
    # don't crash the embedding with out-of-range ids (RIL ISS-058).
    dummy_input = dummy_token_ids(model, input_shape, device=device)

    # Default dynamic axes for variable batch and sequence length
    if dynamic_axes is None:
        dynamic_axes = {
            "input_ids": {0: "batch_size", 1: "seq_len"},
            "logits": {0: "batch_size", 1: "seq_len"},
        }

    # Suppress expected warnings:
    # - TracerWarning from positional encoding bounds check
    # - DeprecationWarning from legacy TorchScript ONNX exporter (PyTorch 2.9+)
    with torch.no_grad(), warnings.catch_warnings():
        warnings.filterwarnings("ignore", category=torch.jit.TracerWarning)
        warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*TorchScript.*ONNX.*")
        warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*feature will be removed.*")
        torch.onnx.export(
            wrapped_model,
            (dummy_input,),
            str(output_path),
            input_names=["input_ids"],
            output_names=["logits"],
            dynamic_axes=dynamic_axes,
            opset_version=opset_version,
            do_constant_folding=True,
            verbose=verbose,
            dynamo=False,
        )

    # fp16/bf16: TorchScript ONNX fusion mislabels LayerNormalization X-input
    # types, producing an artifact onnxruntime cannot load (RIL ISS-067).
    # Pure post-processing fixes the type binding in place.
    model_dtype = next(model.parameters()).dtype
    if model_dtype in (torch.float16, torch.bfloat16):
        _normalize_layer_norm_dtypes(output_path)

    if verbose:
        logger.info("Exported model to %s", output_path)

    return output_path

verify_onnx

verify_onnx(onnx_path, model=None, input_shape=(1, 32), rtol=0.001, atol=1e-05)

Verify ONNX model correctness by comparing with PyTorch output.

参数：

名称	类型	描述	默认
onnx_path	str | Path	Path to ONNX file	必需
model	Module | None	Original PyTorch model (optional, for comparison)	None
input_shape	tuple[int, int]	Input shape for verification	(1, 32)
rtol	float	Relative tolerance for comparison	0.001
atol	float	Absolute tolerance for comparison	1e-05

返回：

类型	描述
bool	True if verification passes

引发：

类型	描述
ImportError	If onnxruntime is not installed

源代码位于： src/llm/export/onnx.py

def verify_onnx(
    onnx_path: str | Path,
    model: nn.Module | None = None,
    input_shape: tuple[int, int] = (1, 32),
    rtol: float = 1e-3,
    atol: float = 1e-5,
) -> bool:
    """
    Verify ONNX model correctness by comparing with PyTorch output.

    Args:
        onnx_path: Path to ONNX file
        model: Original PyTorch model (optional, for comparison)
        input_shape: Input shape for verification
        rtol: Relative tolerance for comparison
        atol: Absolute tolerance for comparison

    Returns:
        True if verification passes

    Raises:
        ImportError: If onnxruntime is not installed
    """
    try:
        import onnxruntime as ort
    except ImportError as e:
        raise ImportError("onnxruntime is required: pip install onnxruntime") from e

    onnx_path = Path(onnx_path)

    # Create ONNX Runtime session
    session = ort.InferenceSession(str(onnx_path), providers=["CPUExecutionProvider"])

    # Create test input — bounded by the real vocab when a model is supplied
    # so small-vocab models don't crash the embedding (RIL ISS-058).
    if model is not None:
        test_input = dummy_token_ids(model, input_shape)
    else:
        batch_size, seq_len = input_shape
        test_input = torch.randint(0, 100, (batch_size, seq_len))

    # Run ONNX inference
    onnx_outputs = session.run(None, {"input_ids": test_input.numpy()})

    if model is not None:
        # Compare with PyTorch output.
        model.eval()
        device = next(model.parameters()).device
        with torch.no_grad():
            # Run the comparison input on the MODEL's device. The ONNX
            # session always executes on CPU, but ``model(test_input)``
            # must be fed a tensor on the same device as the model —
            # otherwise a CUDA-resident model crashes with a device
            # mismatch. Then detach/move the result to CPU as float so
            # it can be compared against the (CPU, fp32) ONNX output
            # regardless of the model's native dtype.
            pt_input = test_input.to(device)
            # Handle tuple return (logits, kv_cache) or just logits
            pt_output = model(pt_input)
            if isinstance(pt_output, tuple):
                pt_output = pt_output[0]
            pt_output = pt_output.float().detach().cpu().numpy()

        # Compare
        import numpy as np

        return np.allclose(np.asarray(onnx_outputs[0]), pt_output, rtol=rtol, atol=atol)

    return True

get_onnx_info

get_onnx_info(onnx_path)

Get information about an ONNX model.

参数：

名称	类型	描述	默认
onnx_path	str | Path	Path to ONNX file	必需

返回：

类型	描述
dict	Dictionary with model info (inputs, outputs, opset)

引发：

类型	描述
ImportError	If onnx is not installed

源代码位于： src/llm/export/onnx.py

def get_onnx_info(onnx_path: str | Path) -> dict:
    """
    Get information about an ONNX model.

    Args:
        onnx_path: Path to ONNX file

    Returns:
        Dictionary with model info (inputs, outputs, opset)

    Raises:
        ImportError: If onnx is not installed
    """
    try:
        import onnx
    except ImportError as e:
        raise ImportError("onnx is required: pip install onnx") from e

    onnx_path = Path(onnx_path)
    model = onnx.load(str(onnx_path))

    return {
        "opset_version": model.opset_import[0].version,
        "inputs": [
            {"name": inp.name, "shape": [d.dim_value or d.dim_param for d in inp.type.tensor_type.shape.dim]}
            for inp in model.graph.input
        ],
        "outputs": [
            {"name": out.name, "shape": [d.dim_value or d.dim_param for d in out.type.tensor_type.shape.dim]}
            for out in model.graph.output
        ],
        "file_size_mb": onnx_path.stat().st_size / (1024 * 1024),
    }

TorchScript Export

torchscript

TorchScript export backend.

This is the second target of :data:llm.export.registry.EXPORT_REGISTRY and the first one to register through the llm.export_backends setuptools entry-point group (rather than the in-code registration used by the built-in onnx target). See :func:llm.export._plugins.register_torchscript_exporter for the hook the entry point points at.

TorchScript ships with PyTorch, so this backend adds no runtime dependencies. The exported artifact is a .pt file loadable via torch.jit.load — useful for deployment paths that can't or won't bring up an ONNX runtime.

Two export modes are supported:

• method='trace' (default): records operations with example inputs. Works for any model that is forward-passable with static shapes. The cache wrapper forces use_cache=False so the tracer doesn't record KV-cache branching.
• method='script': compiles the model with the TorchScript compiler. Requires a model whose forward uses only TorchScript-supported constructs. Models with dynamic Python control flow may not script; in that case, fall back to trace.

export_to_torchscript

export_to_torchscript(model, output_path, *, method='trace', input_shape=(1, 32), example_inputs=None, strict=True, **kwargs)

Export a model to TorchScript.

参数：

名称	类型	描述	默认
model	Module	The model to export.	必需
output_path	str | Path	Path to write the .pt artifact to. Parent directories are created automatically.	必需
method	str	"trace" (default) or "script".	'trace'
input_shape	tuple[int, int]	(batch_size, seq_len) for the dummy input used by trace. Ignored if example_inputs is provided.	(1, 32)
example_inputs	Tensor | None	Pre-built dummy tensor. Overrides input_shape when supplied.	None
strict	bool	Forwarded to :func:torch.jit.trace. When False the tracer silently records missing ops instead of raising. Default True.	True
**kwargs	Any	Forwarded to :func:torch.jit.trace or :func:torch.jit.script.	{}

返回：

类型	描述
Path	The resolved output path.

引发：

类型	描述
ValueError	If method is not "trace" or "script".

源代码位于： src/llm/export/torchscript.py

def export_to_torchscript(
    model: nn.Module,
    output_path: str | Path,
    *,
    method: str = "trace",
    input_shape: tuple[int, int] = (1, 32),
    example_inputs: torch.Tensor | None = None,
    strict: bool = True,
    **kwargs: Any,
) -> Path:
    """Export a model to TorchScript.

    Args:
        model: The model to export.
        output_path: Path to write the ``.pt`` artifact to. Parent
            directories are created automatically.
        method: ``"trace"`` (default) or ``"script"``.
        input_shape: ``(batch_size, seq_len)`` for the dummy input
            used by ``trace``. Ignored if ``example_inputs`` is
            provided.
        example_inputs: Pre-built dummy tensor. Overrides
            ``input_shape`` when supplied.
        strict: Forwarded to :func:`torch.jit.trace`. When ``False``
            the tracer silently records missing ops instead of
            raising. Default ``True``.
        **kwargs: Forwarded to :func:`torch.jit.trace` or
            :func:`torch.jit.script`.

    Returns:
        The resolved output path.

    Raises:
        ValueError: If ``method`` is not ``"trace"`` or ``"script"``.
    """
    if method not in {"trace", "script"}:
        raise ValueError(f"method must be 'trace' or 'script', got {method!r}")

    output_path = Path(output_path)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    model.eval()
    wrapped = ExportCacheWrapper(model)
    wrapped.eval()

    if method == "trace":
        if example_inputs is None:
            # Bounded by the REAL vocab so small-vocab models don't crash
            # the embedding with out-of-range ids (RIL ISS-058).
            device = next(model.parameters()).device
            example_inputs = dummy_token_ids(model, input_shape, device=device)
        # Pre-seed every RoPE cos/sin cache to the example sequence length.
        # ``torch.jit.trace`` runs the model TWICE (tracing + a sanity
        # re-check), and ``RotaryPositionEmbedding`` builds its table behind
        # data-dependent ``if seq_len > cached`` control flow — the 1st run
        # builds, the 2nd skips, so the recorded graphs differ and tracing
        # every RoPE model raised ``TracingCheckError`` (RIL ISS-142; the
        # default non-RoPE model never exercised this). Warming the cache up
        # front makes both runs take the identical "populated" path.
        from llm.core.rope import RotaryPositionEmbedding

        example_seq_len = example_inputs.shape[1] if example_inputs.dim() >= 2 else 1
        for module in model.modules():
            if isinstance(module, RotaryPositionEmbedding):
                module._update_cos_sin_cache(
                    example_seq_len, next(model.parameters()).device, next(model.parameters()).dtype
                )
        scripted = torch.jit.trace(wrapped, example_inputs, strict=strict, **kwargs)
    else:  # method == "script"
        scripted = torch.jit.script(wrapped, **kwargs)

    torch.jit.save(scripted, str(output_path))
    return output_path

build_torchscript_exporter

build_torchscript_exporter(model, output_path, **kwargs)

Factory for the TorchScript export target.

Thin wrapper over :func:export_to_torchscript so the registry contract ((model, output_path, **kwargs) -> Path) matches every other target.

源代码位于： src/llm/export/torchscript.py

def build_torchscript_exporter(
    model: nn.Module,
    output_path: str | Path,
    **kwargs: Any,
) -> Path:
    """Factory for the TorchScript export target.

    Thin wrapper over :func:`export_to_torchscript` so the registry
    contract (``(model, output_path, **kwargs) -> Path``) matches
    every other target.
    """
    return export_to_torchscript(model, output_path, **kwargs)

GGUF Export

gguf

GGUF model format module (ADR-011).

Implements the GGUF v3 container — header / typed metadata / tensor info plus reader and writer — and the two GGML block-quantization schemes shipped in v1 (Q4_0 and Q8_0), then exposes the GGUF export target for :data:llm.export.registry.EXPORT_REGISTRY.

Public surface:

• format: :class:GGUFHeader, :class:GGUFTensorInfo, :class:GGUFValueType, :class:GGMLQuantizationType, :class:GGUFError, and the GGUF_* constants;
• I/O: :class:GGUFWriter / :class:GGUFReader;
• quantization: :func:quantize_q4_0 / :func:dequantize_q4_0 and :func:quantize_q8_0 / :func:dequantize_q8_0;
• dequantization (reader side): :func:dequantize_q4_1 / :func:dequantize_q5_0 / :func:dequantize_q5_1 and the K-quant family :func:dequantize_q2_k .. :func:dequantize_q6_k (import real llama.cpp files);
• export: :func:export_to_gguf / :func:build_gguf_exporter;
• load-back: :func:load_gguf_model (re-build a model from a GGUF the exporter wrote with model_config=).

GGUFReader

Parse and read a GGUF file.

属性：

名称	类型	描述
path		The source file path.
header		Parsed :class:GGUFHeader.
metadata		Ordered metadata dict (typed Python values).
tensors		name -> GGUFTensorInfo mapping in file order.

源代码位于： src/llm/export/gguf/reader.py

class GGUFReader:
    """Parse and read a GGUF file.

    Attributes:
        path: The source file path.
        header: Parsed :class:`GGUFHeader`.
        metadata: Ordered metadata dict (typed Python values).
        tensors: ``name -> GGUFTensorInfo`` mapping in file order.
    """

    def __init__(self, path: str | Path) -> None:
        self.path = Path(path)
        size = self.path.stat().st_size
        if size == 0:
            # ``mmap(fd, 0)`` refuses a zero-length file; fall back to empty
            # bytes so the size/truncation checks below fire with a clear error.
            self._mm: mmap.mmap | None = None
            data: Any = b""
        else:
            # Map the whole file read-only. The mmap stays valid after the fd
            # is closed; payload sections page in lazily on access.
            with self.path.open("rb") as fh:
                self._mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
            data = self._mm
        if len(data) < GGUF_HEADER_SIZE:
            raise GGUFError(f"{self.path}: file too small to be GGUF ({len(data)} bytes)")
        magic, version, tensor_count, kv_count = struct.unpack_from("<IIQQ", data, 0)
        if magic != GGUF_MAGIC:
            raise GGUFError(f"{self.path}: bad magic 0x{magic:08x}, not a GGUF file")
        if not 1 <= version <= GGUF_VERSION:
            raise GGUFError(f"{self.path}: unsupported GGUF version {version} (supported 1..{GGUF_VERSION})")
        self.header = GGUFHeader(
            magic=magic,
            version=version,
            tensor_count=tensor_count,
            metadata_kv_count=kv_count,
        )

        pos = GGUF_HEADER_SIZE
        metadata: dict[str, object] = {}
        for _ in range(kv_count):
            key, pos = _read_string_at(data, pos)
            if pos + 4 > len(data):
                raise GGUFError(f"metadata key {key!r}: truncated value type")
            (type_code,) = struct.unpack_from("<I", data, pos)
            pos += 4
            value, consumed = decode_value(type_code, data[pos:])
            metadata[key] = value
            pos += consumed
        self.metadata = metadata

        tensors: dict[str, GGUFTensorInfo] = {}
        for _ in range(tensor_count):
            name, pos = _read_string_at(data, pos)
            if pos + 4 > len(data):
                raise GGUFError(f"tensor {name!r}: truncated dimension count")
            (n_dims,) = struct.unpack_from("<I", data, pos)
            pos += 4
            if n_dims > _MAX_RANK:
                raise GGUFError(f"tensor {name!r}: implausible rank {n_dims}")
            if pos + 8 * n_dims > len(data):
                raise GGUFError(f"tensor {name!r}: truncated dimensions")
            dims = struct.unpack_from(f"<{n_dims}Q", data, pos)
            pos += 8 * n_dims
            if pos + 12 > len(data):
                raise GGUFError(f"tensor {name!r}: truncated type/offset")
            (type_code, offset) = struct.unpack_from("<IQ", data, pos)
            pos += 12
            try:
                ttype = GGMLQuantizationType(type_code)
            except ValueError:
                raise GGUFError(f"tensor {name!r}: unknown GGML type code {type_code}") from None
            if ttype not in SUPPORTED_TENSOR_TYPES:
                raise GGUFError(
                    f"tensor {name!r}: unsupported GGML type {ttype.name} ({type_code}); "
                    f"v1 supports {sorted(t.name for t in SUPPORTED_TENSOR_TYPES)}"
                )
            shape = tuple(reversed(dims))
            tensors[name] = GGUFTensorInfo(
                name=name,
                shape=shape,
                ggml_type=ttype,
                offset=offset,
                data_size=tensor_data_size(ttype, shape),
            )
        self.tensors = tensors

        self._data_start = align_up(pos, GGUF_DEFAULT_ALIGNMENT)
        self._data = data
        for info in tensors.values():
            if info.offset < self._data_start:
                raise GGUFError(
                    f"tensor {info.name!r}: offset {info.offset} precedes the data section start {self._data_start}"
                )
            if info.offset + info.data_size > len(data):
                raise GGUFError(
                    f"tensor {info.name!r}: data range {info.offset}..{info.offset + info.data_size} "
                    f"exceeds file size {len(data)}"
                )

    def close(self) -> None:
        """Release the memory map (safe to call once; idempotent)."""
        if self._mm is not None:
            self._mm.close()
            self._mm = None

    def __del__(self) -> None:
        with suppress(Exception):
            self.close()

    def _info(self, name: str) -> GGUFTensorInfo:
        try:
            return self.tensors[name]
        except KeyError:
            raise KeyError(f"no tensor named {name!r} in {self.path}") from None

    def read_tensor_raw(self, name: str) -> bytes:
        """Return the exact on-disk payload bytes for ``name`` (no dequantization)."""
        info = self._info(name)
        return bytes(self._data[info.offset : info.offset + info.data_size])

    def read_tensor(self, name: str) -> np.ndarray:
        """Read and dequantize ``name`` into a float32 array of its logical shape.

        F32/F16 payloads are returned as-is (F16 widened); Q4_0/Q8_0 and the
        reader-side legacy/K-quant types (Q4_1/Q5_0/Q5_1, Q2_K..Q6_K) are
        dequantized with the reference ggml math.
        """
        info = self._info(name)
        raw = self._data[info.offset : info.offset + info.data_size]
        if info.ggml_type == GGMLQuantizationType.F32:
            return np.frombuffer(raw, dtype="<f4").reshape(info.shape)
        if info.ggml_type == GGMLQuantizationType.F16:
            return np.frombuffer(raw, dtype="<f2").astype(np.float32).reshape(info.shape)

        numel = math.prod(info.shape)
        if numel == 0:
            # An empty quantized tensor has no blocks to dequantize; mirror
            # the F32/F16 empty-array return instead of raising from the
            # block parser (round-75 review LOW).
            return np.empty(info.shape, dtype=np.float32)
        if info.ggml_type in _READER_DEQUANTIZERS:
            return _READER_DEQUANTIZERS[info.ggml_type](np.frombuffer(raw, dtype=np.uint8), numel).reshape(info.shape)

        block_count = numel // GGML_BLOCK_SIZE
        # ggml block layout is interleaved per 32-element block: a 2-byte
        # fp16 scale followed by the packed values (16 bytes for Q4_0, 32
        # for Q8_0). llama.cpp / gguf-py emit and read this layout; the
        # reader must de-interleave it back into per-block scales + body.
        data_per_block = GGML_BLOCK_SIZE // 2 if info.ggml_type == GGMLQuantizationType.Q4_0 else GGML_BLOCK_SIZE
        block_bytes = 2 + data_per_block
        buf = np.frombuffer(raw, dtype=np.uint8).reshape(block_count, block_bytes)
        scales = np.frombuffer(buf[:, :2].reshape(-1).tobytes(), dtype="<f2").astype(np.float32)
        body = buf[:, 2:].reshape(-1).tobytes()
        if info.ggml_type == GGMLQuantizationType.Q4_0:
            return dequantize_q4_0(np.frombuffer(body, dtype=np.uint8), scales, numel).reshape(info.shape)
        if info.ggml_type == GGMLQuantizationType.Q8_0:
            return dequantize_q8_0(np.frombuffer(body, dtype=np.int8), scales, numel).reshape(info.shape)
        raise GGUFError(f"tensor {name!r}: unsupported GGML type {info.ggml_type.name}")  # pragma: no cover

close

close()

Release the memory map (safe to call once; idempotent).

源代码位于： src/llm/export/gguf/reader.py

def close(self) -> None:
    """Release the memory map (safe to call once; idempotent)."""
    if self._mm is not None:
        self._mm.close()
        self._mm = None

read_tensor_raw

read_tensor_raw(name)

Return the exact on-disk payload bytes for name (no dequantization).

源代码位于： src/llm/export/gguf/reader.py

def read_tensor_raw(self, name: str) -> bytes:
    """Return the exact on-disk payload bytes for ``name`` (no dequantization)."""
    info = self._info(name)
    return bytes(self._data[info.offset : info.offset + info.data_size])

read_tensor

read_tensor(name)

Read and dequantize name into a float32 array of its logical shape.

F32/F16 payloads are returned as-is (F16 widened); Q4_0/Q8_0 and the reader-side legacy/K-quant types (Q4_1/Q5_0/Q5_1, Q2_K..Q6_K) are dequantized with the reference ggml math.

源代码位于： src/llm/export/gguf/reader.py

def read_tensor(self, name: str) -> np.ndarray:
    """Read and dequantize ``name`` into a float32 array of its logical shape.

    F32/F16 payloads are returned as-is (F16 widened); Q4_0/Q8_0 and the
    reader-side legacy/K-quant types (Q4_1/Q5_0/Q5_1, Q2_K..Q6_K) are
    dequantized with the reference ggml math.
    """
    info = self._info(name)
    raw = self._data[info.offset : info.offset + info.data_size]
    if info.ggml_type == GGMLQuantizationType.F32:
        return np.frombuffer(raw, dtype="<f4").reshape(info.shape)
    if info.ggml_type == GGMLQuantizationType.F16:
        return np.frombuffer(raw, dtype="<f2").astype(np.float32).reshape(info.shape)

    numel = math.prod(info.shape)
    if numel == 0:
        # An empty quantized tensor has no blocks to dequantize; mirror
        # the F32/F16 empty-array return instead of raising from the
        # block parser (round-75 review LOW).
        return np.empty(info.shape, dtype=np.float32)
    if info.ggml_type in _READER_DEQUANTIZERS:
        return _READER_DEQUANTIZERS[info.ggml_type](np.frombuffer(raw, dtype=np.uint8), numel).reshape(info.shape)

    block_count = numel // GGML_BLOCK_SIZE
    # ggml block layout is interleaved per 32-element block: a 2-byte
    # fp16 scale followed by the packed values (16 bytes for Q4_0, 32
    # for Q8_0). llama.cpp / gguf-py emit and read this layout; the
    # reader must de-interleave it back into per-block scales + body.
    data_per_block = GGML_BLOCK_SIZE // 2 if info.ggml_type == GGMLQuantizationType.Q4_0 else GGML_BLOCK_SIZE
    block_bytes = 2 + data_per_block
    buf = np.frombuffer(raw, dtype=np.uint8).reshape(block_count, block_bytes)
    scales = np.frombuffer(buf[:, :2].reshape(-1).tobytes(), dtype="<f2").astype(np.float32)
    body = buf[:, 2:].reshape(-1).tobytes()
    if info.ggml_type == GGMLQuantizationType.Q4_0:
        return dequantize_q4_0(np.frombuffer(body, dtype=np.uint8), scales, numel).reshape(info.shape)
    if info.ggml_type == GGMLQuantizationType.Q8_0:
        return dequantize_q8_0(np.frombuffer(body, dtype=np.int8), scales, numel).reshape(info.shape)
    raise GGUFError(f"tensor {name!r}: unsupported GGML type {info.ggml_type.name}")  # pragma: no cover

GGMLQuantizationType

Bases: IntEnum

GGML tensor data types as stored in GGUF tensor info (ggml_type).

The integer type codes were renumbered by ggml PR #6050. F32 / F16 / Q4_0 / Q8_0 have stable codes across all versions and are the types this repo exports. Reading additionally supports the legacy 32-wide schemes (Q4_1 / Q5_0 / Q5_1) and the 256-wide K-quant family (Q2_K .. Q6_K) that make up virtually every downloadable llama.cpp GGUF. The remaining values follow the current ggml.h enumeration.

源代码位于： src/llm/export/gguf/spec.py

class GGMLQuantizationType(IntEnum):
    """GGML tensor data types as stored in GGUF tensor info (``ggml_type``).

    The integer type codes were renumbered by ggml PR #6050.  ``F32`` /
    ``F16`` / ``Q4_0`` / ``Q8_0`` have stable codes across all versions and
    are the types this repo *exports*.  Reading additionally supports the
    legacy 32-wide schemes (``Q4_1`` / ``Q5_0`` / ``Q5_1``) and the 256-wide
    K-quant family (``Q2_K`` .. ``Q6_K``) that make up virtually every
    downloadable llama.cpp GGUF.  The remaining values follow the current
    ggml.h enumeration.
    """

    F32 = 0
    F16 = 1
    Q4_0 = 2
    Q4_1 = 3
    Q5_0 = 6
    Q5_1 = 7
    Q8_0 = 8
    Q8_1 = 9
    Q2_K = 10
    Q3_K = 11
    Q4_K = 12
    Q5_K = 13
    Q6_K = 14
    Q8_K = 15
    IQ2_XXS = 16
    IQ2_XS = 17
    IQ3_XXS = 18
    IQ1_S = 19
    IQ4_NL = 20
    IQ3_S = 21
    IQ2_S = 22
    IQ4_XS = 23
    I8 = 24
    I16 = 25
    I32 = 26
    I64 = 27
    F64 = 28

GGUFError

Bases: ValueError

Raised when a GGUF file, metadata blob, or tensor payload is malformed or unsupported.

源代码位于： src/llm/export/gguf/spec.py

class GGUFError(ValueError):
    """Raised when a GGUF file, metadata blob, or tensor payload is malformed or unsupported."""

GGUFHeader dataclass

Parsed GGUF header.

源代码位于： src/llm/export/gguf/spec.py

@dataclass(frozen=True)
class GGUFHeader:
    """Parsed GGUF header."""

    magic: int
    version: int
    tensor_count: int
    metadata_kv_count: int

GGUFTensorInfo dataclass

Parsed GGUF tensor info.

shape is the LOGICAL shape in row-major (PyTorch/NumPy) order — e.g. (out_features, in_features). GGUF stores dimensions in the reverse order on disk; reader and writer translate at the boundary.

源代码位于： src/llm/export/gguf/spec.py

@dataclass(frozen=True)
class GGUFTensorInfo:
    """Parsed GGUF tensor info.

    ``shape`` is the LOGICAL shape in row-major (PyTorch/NumPy) order —
    e.g. ``(out_features, in_features)``. GGUF stores dimensions in the
    reverse order on disk; reader and writer translate at the boundary.
    """

    name: str
    shape: tuple[int, ...]
    ggml_type: GGMLQuantizationType
    offset: int
    data_size: int

GGUFValueType

Bases: IntEnum

GGUF metadata value types (spec §Value Types).

源代码位于： src/llm/export/gguf/spec.py

class GGUFValueType(IntEnum):
    """GGUF metadata value types (spec §Value Types)."""

    UINT8 = 0
    INT8 = 1
    UINT16 = 2
    INT16 = 3
    UINT32 = 4
    INT32 = 5
    FLOAT32 = 6
    BOOL = 7
    STRING = 8
    ARRAY = 9
    UINT64 = 10
    INT64 = 11
    FLOAT64 = 12

GGUFWriter

Incremental GGUF v3 writer.

Usage::

writer = GGUFWriter("model.gguf")
writer.add_metadata("general.name", "tiny")
writer.add_tensor("w", weight_numpy, ggml_type="q8_0")
path = writer.write()

源代码位于： src/llm/export/gguf/writer.py

class GGUFWriter:
    """Incremental GGUF v3 writer.

    Usage::

        writer = GGUFWriter("model.gguf")
        writer.add_metadata("general.name", "tiny")
        writer.add_tensor("w", weight_numpy, ggml_type="q8_0")
        path = writer.write()
    """

    def __init__(
        self,
        output_path: str | Path,
        *,
        version: int = GGUF_VERSION,
        alignment: int = GGUF_DEFAULT_ALIGNMENT,
    ) -> None:
        if not 1 <= version <= GGUF_VERSION:
            raise ValueError(f"unsupported GGUF version {version} (supported 1..{GGUF_VERSION})")
        if alignment <= 0:
            raise ValueError(f"alignment must be positive, got {alignment}")
        if alignment < GGUF_DEFAULT_ALIGNMENT:
            # The GGUF spec fixes tensor-data alignment at
            # ``GGUF_DEFAULT_ALIGNMENT`` (32); the reader hardcodes 32 for
            # ``_data_start``. A smaller writer alignment would emit a file
            # whose own reader (and llama.cpp) rejects every tensor as
            # preceding the data section (RIL ISS-059). Reject it up front.
            raise ValueError(f"alignment must be >= GGUF_DEFAULT_ALIGNMENT ({GGUF_DEFAULT_ALIGNMENT}), got {alignment}")
        self.output_path = Path(output_path)
        self.version = version
        self.alignment = alignment
        self._metadata: dict[str, Any] = {}
        self._tensors: list[tuple[str, GGMLQuantizationType, tuple[int, ...], bytes]] = []

    def add_metadata(self, key: str, value: Any) -> None:
        """Register one metadata KV pair (later duplicates overwrite)."""
        if not isinstance(key, str) or not key:
            raise ValueError(f"metadata key must be a non-empty string, got {key!r}")
        encode_metadata({key: value})  # validate encodability early
        self._metadata[key] = value

    def add_tensor(
        self,
        name: str,
        data: Any,
        ggml_type: GGMLQuantizationType | str,
    ) -> None:
        """Register one tensor with an explicit GGML type.

        Args:
            name: Tensor name (must be unique).
            data: ``numpy`` array or ``torch`` tensor of floating dtype.
            ggml_type: One of ``F32`` / ``F16`` / ``Q4_0`` / ``Q8_0``
                (or a case-insensitive name like ``"q8_0"``).

        Raises:
            ValueError: For duplicate names, non-float input, or a
                block-quantized type whose last dimension is not a
                multiple of 32.
            GGUFError: For unsupported tensor types.
        """
        if not isinstance(name, str) or not name:
            raise ValueError(f"tensor name must be a non-empty string, got {name!r}")
        if any(existing == name for existing, _, _, _ in self._tensors):
            raise ValueError(f"duplicate tensor name {name!r}")
        ttype = ggml_type if isinstance(ggml_type, GGMLQuantizationType) else parse_ggml_type(str(ggml_type))
        if ttype not in EXPORT_TENSOR_TYPES:
            raise GGUFError(
                f"{ttype.name} is reader-supported but not exportable; "
                f"the writer emits {sorted(t.name for t in EXPORT_TENSOR_TYPES)}"
            )
        arr = _as_float32_array(data)
        if arr.ndim == 0:
            raise ValueError(f"tensor {name!r}: scalar tensors are not supported")
        if arr.size == 0:
            # An empty tensor is never a legitimate weight. In the quantized
            # types it also crashed with a raw ``ZeroDivisionError``
            # (``data_per_block = data.size // scales.size`` → ``0 // 0``)
            # deep inside block serialization (GGUF deep-dive finding #3).
            # Reject it up front so the export fails with a clear message.
            raise ValueError(f"tensor {name!r}: empty tensors (0 elements) are not supported")
        shape = tuple(int(d) for d in arr.shape)
        if ttype in (GGMLQuantizationType.Q4_0, GGMLQuantizationType.Q8_0) and not can_quantize_shape(shape):
            raise ValueError(
                f"tensor {name!r}: {ttype.name} requires the last dimension to be a multiple of "
                f"{GGML_BLOCK_SIZE} (got shape {shape})"
            )
        if ttype in (
            GGMLQuantizationType.Q2_K,
            GGMLQuantizationType.Q3_K,
            GGMLQuantizationType.Q4_K,
            GGMLQuantizationType.Q5_K,
            GGMLQuantizationType.Q6_K,
        ) and not can_quantize_k_shape(shape):
            raise ValueError(
                f"tensor {name!r}: {ttype.name} requires the last dimension to be a multiple of "
                f"{GGML_K_BLOCK_SIZE} (got shape {shape})"
            )
        payload = _encode_payload(arr, ttype)
        self._tensors.append((name, ttype, shape, payload))

    def write(self) -> Path:
        """Assemble and atomically write the GGUF file; returns the output path."""
        metadata_bytes = encode_metadata(self._metadata)
        infos_size = sum(_tensor_info_size(name, shape) for name, _, shape, _ in self._tensors)
        data_start = align_up(GGUF_HEADER_SIZE + len(metadata_bytes) + infos_size, self.alignment)

        offset = data_start
        entries: list[tuple[str, GGMLQuantizationType, tuple[int, ...], bytes, int]] = []
        for name, ttype, shape, payload in self._tensors:
            entries.append((name, ttype, shape, payload, offset))
            offset += align_up(len(payload), self.alignment)

        buf = io.BytesIO()
        buf.write(
            struct.pack(
                "<IIQQ",
                GGUF_MAGIC,
                self.version,
                len(self._tensors),
                len(self._metadata),
            )
        )
        buf.write(metadata_bytes)
        for name, ttype, shape, _, off in entries:
            buf.write(_encode_tensor_info(name, shape, ttype, off))
        buf.write(b"\x00" * (data_start - buf.tell()))
        for _, _, _, payload, _ in entries:
            buf.write(payload)
            buf.write(b"\x00" * (align_up(len(payload), self.alignment) - len(payload)))

        self.output_path.parent.mkdir(parents=True, exist_ok=True)
        tmp_path = self.output_path.with_name(self.output_path.name + ".tmp")
        tmp_path.write_bytes(buf.getvalue())
        tmp_path.replace(self.output_path)
        return self.output_path

add_metadata

add_metadata(key, value)

Register one metadata KV pair (later duplicates overwrite).

源代码位于： src/llm/export/gguf/writer.py

def add_metadata(self, key: str, value: Any) -> None:
    """Register one metadata KV pair (later duplicates overwrite)."""
    if not isinstance(key, str) or not key:
        raise ValueError(f"metadata key must be a non-empty string, got {key!r}")
    encode_metadata({key: value})  # validate encodability early
    self._metadata[key] = value

add_tensor

add_tensor(name, data, ggml_type)

Register one tensor with an explicit GGML type.

参数：

名称	类型	描述	默认
name	str	Tensor name (must be unique).	必需
data	Any	numpy array or torch tensor of floating dtype.	必需
ggml_type	GGMLQuantizationType | str	One of F32 / F16 / Q4_0 / Q8_0 (or a case-insensitive name like "q8_0").	必需

引发：

类型	描述
ValueError	For duplicate names, non-float input, or a block-quantized type whose last dimension is not a multiple of 32.
GGUFError	For unsupported tensor types.

源代码位于： src/llm/export/gguf/writer.py

def add_tensor(
    self,
    name: str,
    data: Any,
    ggml_type: GGMLQuantizationType | str,
) -> None:
    """Register one tensor with an explicit GGML type.

    Args:
        name: Tensor name (must be unique).
        data: ``numpy`` array or ``torch`` tensor of floating dtype.
        ggml_type: One of ``F32`` / ``F16`` / ``Q4_0`` / ``Q8_0``
            (or a case-insensitive name like ``"q8_0"``).

    Raises:
        ValueError: For duplicate names, non-float input, or a
            block-quantized type whose last dimension is not a
            multiple of 32.
        GGUFError: For unsupported tensor types.
    """
    if not isinstance(name, str) or not name:
        raise ValueError(f"tensor name must be a non-empty string, got {name!r}")
    if any(existing == name for existing, _, _, _ in self._tensors):
        raise ValueError(f"duplicate tensor name {name!r}")
    ttype = ggml_type if isinstance(ggml_type, GGMLQuantizationType) else parse_ggml_type(str(ggml_type))
    if ttype not in EXPORT_TENSOR_TYPES:
        raise GGUFError(
            f"{ttype.name} is reader-supported but not exportable; "
            f"the writer emits {sorted(t.name for t in EXPORT_TENSOR_TYPES)}"
        )
    arr = _as_float32_array(data)
    if arr.ndim == 0:
        raise ValueError(f"tensor {name!r}: scalar tensors are not supported")
    if arr.size == 0:
        # An empty tensor is never a legitimate weight. In the quantized
        # types it also crashed with a raw ``ZeroDivisionError``
        # (``data_per_block = data.size // scales.size`` → ``0 // 0``)
        # deep inside block serialization (GGUF deep-dive finding #3).
        # Reject it up front so the export fails with a clear message.
        raise ValueError(f"tensor {name!r}: empty tensors (0 elements) are not supported")
    shape = tuple(int(d) for d in arr.shape)
    if ttype in (GGMLQuantizationType.Q4_0, GGMLQuantizationType.Q8_0) and not can_quantize_shape(shape):
        raise ValueError(
            f"tensor {name!r}: {ttype.name} requires the last dimension to be a multiple of "
            f"{GGML_BLOCK_SIZE} (got shape {shape})"
        )
    if ttype in (
        GGMLQuantizationType.Q2_K,
        GGMLQuantizationType.Q3_K,
        GGMLQuantizationType.Q4_K,
        GGMLQuantizationType.Q5_K,
        GGMLQuantizationType.Q6_K,
    ) and not can_quantize_k_shape(shape):
        raise ValueError(
            f"tensor {name!r}: {ttype.name} requires the last dimension to be a multiple of "
            f"{GGML_K_BLOCK_SIZE} (got shape {shape})"
        )
    payload = _encode_payload(arr, ttype)
    self._tensors.append((name, ttype, shape, payload))

write

write()

Assemble and atomically write the GGUF file; returns the output path.

源代码位于： src/llm/export/gguf/writer.py

def write(self) -> Path:
    """Assemble and atomically write the GGUF file; returns the output path."""
    metadata_bytes = encode_metadata(self._metadata)
    infos_size = sum(_tensor_info_size(name, shape) for name, _, shape, _ in self._tensors)
    data_start = align_up(GGUF_HEADER_SIZE + len(metadata_bytes) + infos_size, self.alignment)

    offset = data_start
    entries: list[tuple[str, GGMLQuantizationType, tuple[int, ...], bytes, int]] = []
    for name, ttype, shape, payload in self._tensors:
        entries.append((name, ttype, shape, payload, offset))
        offset += align_up(len(payload), self.alignment)

    buf = io.BytesIO()
    buf.write(
        struct.pack(
            "<IIQQ",
            GGUF_MAGIC,
            self.version,
            len(self._tensors),
            len(self._metadata),
        )
    )
    buf.write(metadata_bytes)
    for name, ttype, shape, _, off in entries:
        buf.write(_encode_tensor_info(name, shape, ttype, off))
    buf.write(b"\x00" * (data_start - buf.tell()))
    for _, _, _, payload, _ in entries:
        buf.write(payload)
        buf.write(b"\x00" * (align_up(len(payload), self.alignment) - len(payload)))

    self.output_path.parent.mkdir(parents=True, exist_ok=True)
    tmp_path = self.output_path.with_name(self.output_path.name + ".tmp")
    tmp_path.write_bytes(buf.getvalue())
    tmp_path.replace(self.output_path)
    return self.output_path

build_gguf_exporter

build_gguf_exporter(model, output_path, **kwargs)

Factory for the GGUF export target (EXPORT_REGISTRY contract).

源代码位于： src/llm/export/gguf/exporter.py

def build_gguf_exporter(
    model: nn.Module,
    output_path: str | Path,
    **kwargs: Any,
) -> Path:
    """Factory for the GGUF export target (``EXPORT_REGISTRY`` contract)."""
    return export_to_gguf(model, output_path, **kwargs)

export_to_gguf

export_to_gguf(model, output_path, *, quantize=None, metadata=None, model_name=None, quantize_min_ndim=2, model_config=None)

Export model.state_dict() to a GGUF v3 file.

参数：

名称	类型	描述	默认
model	Module	The model to export (evaluated state, tensors are detached on CPU).	必需
output_path	str | Path	Destination .gguf path; parent directories are created automatically.	必需
quantize	str | GGMLQuantizationType | None	None (default) or "f16" writes F16 tensors; "f32" writes F32; "q4_0" / "q8_0" block-quantizes eligible weight tensors (ndim >= quantize_min_ndim and last dim a multiple of 32) and keeps everything else F16.	None
metadata	dict[str, Any] | None	Extra general.*-style metadata; overrides the built-in defaults (general.name, general.file_type, ...).	None
model_name	str | None	Override for general.name (defaults to the model class name).	None
quantize_min_ndim	int	Minimum tensor rank eligible for block-quantization.	2
model_config	dict[str, Any] | None	Optional architecture config as a JSON-safe dict (e.g. ModelConfig.model_dump()). When present it is persisted as general.llm_model_config so :func:llm.export.gguf.loader.load_gguf_model can rebuild the exact model — closing the export-only loop (round 71).	None

返回：

类型	描述
Path	The resolved output path.

引发：

类型	描述
NotImplementedError	If the model has a non-floating tensor in its state dict (v1 scope).
ValueError	For unknown quantize values.

源代码位于： src/llm/export/gguf/exporter.py

def export_to_gguf(
    model: nn.Module,
    output_path: str | Path,
    *,
    quantize: str | GGMLQuantizationType | None = None,
    metadata: dict[str, Any] | None = None,
    model_name: str | None = None,
    quantize_min_ndim: int = 2,
    model_config: dict[str, Any] | None = None,
) -> Path:
    """Export ``model.state_dict()`` to a GGUF v3 file.

    Args:
        model: The model to export (evaluated state, tensors are
            detached on CPU).
        output_path: Destination ``.gguf`` path; parent directories are
            created automatically.
        quantize: ``None`` (default) or ``"f16"`` writes F16 tensors;
            ``"f32"`` writes F32; ``"q4_0"`` / ``"q8_0"``
            block-quantizes eligible weight tensors (ndim >=
            ``quantize_min_ndim`` and last dim a multiple of 32) and
            keeps everything else F16.
        metadata: Extra ``general.*``-style metadata; overrides the
            built-in defaults (``general.name``, ``general.file_type``,
            ...).
        model_name: Override for ``general.name`` (defaults to the model
            class name).
        quantize_min_ndim: Minimum tensor rank eligible for
            block-quantization.
        model_config: Optional architecture config as a JSON-safe dict
            (e.g. ``ModelConfig.model_dump()``). When present it is
            persisted as ``general.llm_model_config`` so
            :func:`llm.export.gguf.loader.load_gguf_model` can rebuild
            the exact model — closing the export-only loop (round 71).

    Returns:
        The resolved output path.

    Raises:
        NotImplementedError: If the model has a non-floating tensor in
            its state dict (v1 scope).
        ValueError: For unknown ``quantize`` values.
    """
    quant_type = _resolve_quant_type(quantize)

    writer = GGUFWriter(output_path)
    for key, value in _default_metadata(model, model_name, quant_type, metadata, model_config).items():
        writer.add_metadata(key, value)

    for name, tensor in model.state_dict().items():
        if not tensor.is_floating_point():
            raise NotImplementedError(
                f"GGUF exporter v1 only supports floating-point tensors; {name!r} has dtype {tensor.dtype}"
            )
        arr = tensor.detach().float().cpu().numpy()
        ttype = _pick_tensor_type(arr, quant_type, quantize_min_ndim)
        if ttype != quant_type and quant_type in (
            GGMLQuantizationType.Q4_0,
            GGMLQuantizationType.Q8_0,
            GGMLQuantizationType.Q2_K,
            GGMLQuantizationType.Q3_K,
            GGMLQuantizationType.Q4_K,
            GGMLQuantizationType.Q5_K,
            GGMLQuantizationType.Q6_K,
        ):
            logger.debug("keeping %s as F16 (shape %s not block-quantizable)", name, tuple(arr.shape))
        writer.add_tensor(name, arr, ggml_type=ttype)

    return writer.write()

load_gguf_model

load_gguf_model(path, *, device=None)

Rebuild a model from a GGUF file — self-export or llama.cpp import.

参数：

名称	类型	描述	默认
path	str | Path	GGUF file path.	必需
device	device | str | None	Optional target device (default: CPU).	None

返回：

类型	描述
Module	The rebuilt model in eval() mode with the exported/imported
Module	weights.

引发：

类型	描述
GGUFError	If the file is malformed; if a self-export carries an invalid general.llm_model_config; or if a foreign file is not a supported dense Llama-style architecture, is missing required {arch}.* metadata, carries tensors that do not map into llm state-dict naming, or uses an unsupported feature (RoPE scaling, non-standard head dims).
RuntimeError	If the tensor names/shapes in a self-export do not match the model rebuilt from the embedded config (strict load_state_dict).

Note

F32/F16 files round-trip or import exactly (widened to float32); block-quantized files (Q4_0/Q8_0) come back dequantized and therefore approximately, within the quantizer's expected error.

源代码位于： src/llm/export/gguf/loader.py

def load_gguf_model(
    path: str | Path,
    *,
    device: torch.device | str | None = None,
) -> nn.Module:
    """Rebuild a model from a GGUF file — self-export or llama.cpp import.

    Args:
        path: GGUF file path.
        device: Optional target device (default: CPU).

    Returns:
        The rebuilt model in ``eval()`` mode with the exported/imported
        weights.

    Raises:
        GGUFError: If the file is malformed; if a self-export carries an
            invalid ``general.llm_model_config``; or if a foreign file is not
            a supported dense Llama-style architecture, is missing required
            ``{arch}.*`` metadata, carries tensors that do not map into
            ``llm`` state-dict naming, or uses an unsupported feature (RoPE
            scaling, non-standard head dims).
        RuntimeError: If the tensor names/shapes in a self-export do not match
            the model rebuilt from the embedded config (strict
            ``load_state_dict``).

    Note:
        F32/F16 files round-trip or import exactly (widened to float32);
        *block-quantized* files (Q4_0/Q8_0) come back dequantized and
        therefore approximately, within the quantizer's expected error.
    """
    reader = GGUFReader(path)

    raw_config = reader.metadata.get("general.llm_model_config")
    if isinstance(raw_config, str):
        return _load_self_export(reader, raw_config, device=device)
    return _load_foreign_gguf(reader, device=device)

dequantize_q2_k

dequantize_q2_k(raw, n)

Dequantize Q2_K blocks: 256 2-bit values, 16 per-block (d,s) pairs.

Layout: scales(16) + qs(64) + d(2) + dmin(2). Per 16-element group, dl = d * (scales & 0xF) and ml = dmin * (scales >> 4); value is dl * q2 - ml.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q2_k(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q2_K blocks: 256 2-bit values, 16 per-block (d,s) pairs.

    Layout: scales(16) + qs(64) + d(2) + dmin(2).  Per 16-element group,
    ``dl = d * (scales & 0xF)`` and ``ml = dmin * (scales >> 4)``; value is
    ``dl * q2 - ml``.
    """
    blocks = _as_whole_blocks(raw, 16 + 64 + 4, "Q2_K")
    scales = blocks[:, 0:16]
    qs = blocks[:, 16:80]
    d = _block_f16(blocks, 80)
    dmin = _block_f16(blocks, 82)
    dl = d[:, None] * (scales & 0x0F).astype(np.float32)
    ml = dmin[:, None] * (scales >> 4).astype(np.float32)
    shift = np.array([0, 2, 4, 6], dtype=np.uint8).reshape(1, 1, 4, 1)
    qv = (qs.reshape(-1, 2, 1, 32) >> shift) & np.uint8(3)
    qv = qv.reshape(-1, 16, 16).astype(np.float32)
    out = dl[:, :, None] * qv - ml[:, :, None]
    return _trim(out.reshape(-1, 256), n)

dequantize_q3_k

dequantize_q3_k(raw, n)

Dequantize Q3_K blocks: 256 3-bit values + 16 scale bytes packed 6-bit.

Layout: hmask(32) + qs(64) + scales(12) + d(2). The 16 per-block scales are packed 6 bits each across the 12 scales bytes.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q3_k(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q3_K blocks: 256 3-bit values + 16 scale bytes packed 6-bit.

    Layout: hmask(32) + qs(64) + scales(12) + d(2).  The 16 per-block scales
    are packed 6 bits each across the 12 ``scales`` bytes.
    """
    blocks = _as_whole_blocks(raw, 32 + 64 + 12 + 2, "Q3_K")
    hmask = blocks[:, 0:32]
    qs = blocks[:, 32:96]
    scales = blocks[:, 96:108]
    d = _block_f16(blocks, 108)
    # scales: 8 low bytes pack two 4-bit scale parts, 4 high bytes hold the
    # remaining two bits of each of the 16 scales (ggml aux[4] rearrangement).
    lscales, hscales = np.hsplit(scales, [8])
    lscales = lscales.reshape(-1, 1, 8) >> np.array([0, 4], dtype=np.uint8).reshape(1, 2, 1)
    lscales = lscales.reshape(-1, 16)
    hscales = hscales.reshape(-1, 1, 4) >> np.array([0, 2, 4, 6], dtype=np.uint8).reshape(1, 4, 1)
    hscales = hscales.reshape(-1, 16)
    scales8 = (lscales & 0x0F) | ((hscales & 0x03) << 4)
    scales8 = (scales8.astype(np.int8) - np.int8(32)).astype(np.float32)
    dl = (d[:, None] * scales8).reshape(-1, 16, 1)
    ql = qs.reshape(-1, 2, 1, 32) >> np.array([0, 2, 4, 6], dtype=np.uint8).reshape(1, 1, 4, 1)
    qh = hmask.reshape(-1, 1, 1, 32) >> np.arange(8, dtype=np.uint8).reshape(1, 1, 8, 1)
    ql = ql.reshape(-1, 16, 16) & np.uint8(3)
    qh = (qh.reshape(-1, 16, 16) & np.uint8(1)) ^ np.uint8(1)  # offset 0 when bitmask set
    q = (ql.astype(np.int8) - (qh << np.uint8(2)).astype(np.int8)).astype(np.float32)
    out = dl * q
    return _trim(out.reshape(-1, 256), n)

dequantize_q4_0

dequantize_q4_0(packed, scales, n)

Dequantize Q4_0 blocks back to float32.

Matches ggml's dequantize_row_q4_0: (q - 8) * d with the same (negative) per-block scale used at quantize time.

参数：

名称	类型	描述	默认
packed	ndarray	uint8 nibble bytes (one per two elements).	必需
scales	ndarray	float16/float32 block scales (one per 32 elements).	必需
n	int	Total element count to reconstruct.	必需

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q4_0(packed: np.ndarray, scales: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q4_0 blocks back to float32.

    Matches ggml's ``dequantize_row_q4_0``: ``(q - 8) * d`` with the same
    (negative) per-block scale used at quantize time.

    Args:
        packed: ``uint8`` nibble bytes (one per two elements).
        scales: ``float16``/``float32`` block scales (one per 32 elements).
        n: Total element count to reconstruct.
    """
    p = np.asarray(packed, dtype=np.uint8).reshape(-1)
    s = np.asarray(scales, dtype=np.float32).reshape(-1)
    if p.size % (GGML_BLOCK_SIZE // 2):
        raise ValueError(f"packed Q4_0 data must hold whole blocks, got {p.size} bytes")
    numel = p.size * 2
    if s.size != numel // GGML_BLOCK_SIZE:
        raise ValueError(f"expected {numel // GGML_BLOCK_SIZE} Q4_0 scales, got {s.size}")
    low = (p & 0x0F).astype(np.float32) - 8.0
    high = ((p >> 4) & 0x0F).astype(np.float32) - 8.0
    blocks = np.empty((s.size, GGML_BLOCK_SIZE), dtype=np.float32)
    blocks[:, : GGML_BLOCK_SIZE // 2] = low.reshape(-1, GGML_BLOCK_SIZE // 2)
    blocks[:, GGML_BLOCK_SIZE // 2 :] = high.reshape(-1, GGML_BLOCK_SIZE // 2)
    out = (blocks * s[:, None]).reshape(-1)
    if n < 0 or n > out.size:
        raise ValueError(f"cannot reconstruct {n} elements from {out.size} available")
    return out[:n]

dequantize_q4_1

dequantize_q4_1(raw, n)

Dequantize Q4_1 blocks: value = q * d + m (q in 0..15).

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q4_1(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q4_1 blocks: value = ``q * d + m`` (q in 0..15)."""
    blocks = _as_whole_blocks(raw, 2 + 2 + 16, "Q4_1")
    d = _block_f16(blocks, 0)
    m = _block_f16(blocks, 2)
    qs = blocks[:, 4:]  # (n, 16)
    low = (qs & 0x0F).astype(np.float32)  # elements 0..15
    high = ((qs >> 4) & 0x0F).astype(np.float32)  # elements 16..31
    out = np.concatenate([low, high], axis=1) * d[:, None] + m[:, None]
    return _trim(out, n)

dequantize_q4_k

dequantize_q4_k(raw, n)

Dequantize Q4_K blocks: value = d*sc*q - dmin*m.

Layout: d(2) + dmin(2) + scales(12) + qs(128); 8 (scale, min) pairs cover eight 32-element groups.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q4_k(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q4_K blocks: value = ``d*sc*q - dmin*m``.

    Layout: d(2) + dmin(2) + scales(12) + qs(128); 8 (scale, min) pairs cover
    eight 32-element groups.
    """
    blocks = _as_whole_blocks(raw, 4 + 12 + 128, "Q4_K")
    d = _block_f16(blocks, 0)
    dmin = _block_f16(blocks, 2)
    scales = blocks[:, 4:16]
    qs = blocks[:, 16:144]
    sc, m = _get_scale_min_k4(scales)
    d1 = (d[:, None] * sc.astype(np.float32)).reshape(-1, 8, 1)
    dm = (dmin[:, None] * m.astype(np.float32)).reshape(-1, 8, 1)
    qv = (qs.reshape(-1, 4, 1, 32) >> np.array([0, 4], dtype=np.uint8).reshape(1, 1, 2, 1)) & np.uint8(0x0F)
    qv = qv.reshape(-1, 8, 32).astype(np.float32)
    out = d1 * qv - dm
    return _trim(out.reshape(-1, 256), n)

dequantize_q5_0

dequantize_q5_0(raw, n)

Dequantize Q5_0 blocks: value = (q - 16) * d; element p keeps its fifth bit at qh bit p (file/quantizer layout — see module doc).

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q5_0(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q5_0 blocks: value = ``(q - 16) * d``; element ``p`` keeps
    its fifth bit at ``qh`` bit ``p`` (file/quantizer layout — see module doc)."""
    blocks = _as_whole_blocks(raw, 2 + 4 + 16, "Q5_0")
    d = _block_f16(blocks, 0)
    qh = np.frombuffer(blocks[:, 2:6].reshape(-1).tobytes(), dtype="<u4")  # (n,)
    qs = blocks[:, 6:]  # (n, 16)
    low = qs & 0x0F
    high = (qs >> 4) & 0x0F
    bits = ((qh[:, None] >> np.arange(32, dtype=np.uint32)) & 1).astype(np.uint8) << 4
    vals = (np.concatenate([low, high], axis=1) | bits).astype(np.int32) - 16
    out = vals * d[:, None]
    return _trim(out, n)

dequantize_q5_1

dequantize_q5_1(raw, n)

Dequantize Q5_1 blocks: value = q * d + m with the same qh layout.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q5_1(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q5_1 blocks: value = ``q * d + m`` with the same qh layout."""
    blocks = _as_whole_blocks(raw, 2 + 2 + 4 + 16, "Q5_1")
    d = _block_f16(blocks, 0)
    m = _block_f16(blocks, 2)
    qh = np.frombuffer(blocks[:, 4:8].reshape(-1).tobytes(), dtype="<u4")
    qs = blocks[:, 8:]  # (n, 16)
    low = qs & 0x0F
    high = (qs >> 4) & 0x0F
    bits = ((qh[:, None] >> np.arange(32, dtype=np.uint32)) & 1).astype(np.uint8) << 4
    vals = (np.concatenate([low, high], axis=1) | bits).astype(np.float32)
    out = vals * d[:, None] + m[:, None]
    return _trim(out, n)

dequantize_q5_k

dequantize_q5_k(raw, n)

Dequantize Q5_K blocks: 32-element groups with 5-bit values.

Layout: d(2) + dmin(2) + scales(12) + qh(32) + qs(128); the 5th bit of element p lives in qh bit (p % 32) of byte p // 32.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q5_k(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q5_K blocks: 32-element groups with 5-bit values.

    Layout: d(2) + dmin(2) + scales(12) + qh(32) + qs(128); the 5th bit of
    element ``p`` lives in ``qh`` bit ``(p % 32)`` of byte ``p // 32``.
    """
    blocks = _as_whole_blocks(raw, 4 + 12 + 32 + 128, "Q5_K")
    d = _block_f16(blocks, 0)
    dmin = _block_f16(blocks, 2)
    scales = blocks[:, 4:16]
    qh = blocks[:, 16:48]
    qs = blocks[:, 48:176]
    sc, m = _get_scale_min_k4(scales)
    d1 = (d[:, None] * sc.astype(np.float32)).reshape(-1, 8, 1)
    dm = (dmin[:, None] * m.astype(np.float32)).reshape(-1, 8, 1)
    ql = (qs.reshape(-1, 4, 1, 32) >> np.array([0, 4], dtype=np.uint8).reshape(1, 1, 2, 1)) & np.uint8(0x0F)
    qh = (qh.reshape(-1, 1, 1, 32) >> np.arange(8, dtype=np.uint8).reshape(1, 1, 8, 1)) & np.uint8(0x01)
    q = (ql.reshape(-1, 8, 32) | (qh.reshape(-1, 8, 32) << np.uint8(4))).astype(np.float32)
    out = d1 * q - dm
    return _trim(out.reshape(-1, 256), n)

dequantize_q6_k

dequantize_q6_k(raw, n)

Dequantize Q6_K blocks: 256 6-bit values, 16 per-block fp16*int8 scales.

Layout: ql(128) + qh(64) + scales(16) + d(2); value = d * sc * (q - 32) with sc an int8 per 16-element group.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q6_k(raw: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q6_K blocks: 256 6-bit values, 16 per-block fp16*int8 scales.

    Layout: ql(128) + qh(64) + scales(16) + d(2); value = ``d * sc * (q - 32)``
    with ``sc`` an int8 per 16-element group.
    """
    blocks = _as_whole_blocks(raw, 128 + 64 + 16 + 2, "Q6_K")
    ql = blocks[:, 0:128]
    qh = blocks[:, 128:192]
    scales = blocks[:, 192:208]
    d = _block_f16(blocks, 208)
    sc = np.asarray(scales, dtype=np.uint8).astype(np.int8).astype(np.float32)
    d1 = (d[:, None] * sc).reshape(-1, 16, 1)
    qlv = (ql.reshape(-1, 2, 1, 64) >> np.array([0, 4], dtype=np.uint8).reshape(1, 1, 2, 1)) & np.uint8(0x0F)
    qhv = (qh.reshape(-1, 2, 1, 32) >> np.array([0, 2, 4, 6], dtype=np.uint8).reshape(1, 1, 4, 1)) & np.uint8(0x03)
    qv = qlv.reshape(-1, 8, 32) | (qhv.reshape(-1, 8, 32) << np.uint8(4))
    qv = (qv.astype(np.int8) - np.int8(32)).reshape(-1, 16, 16).astype(np.float32)
    out = d1 * qv
    return _trim(out.reshape(-1, 256), n)

dequantize_q8_0

dequantize_q8_0(values, scales, n)

Dequantize Q8_0 blocks back to float32.

源代码位于： src/llm/export/gguf/quant.py

def dequantize_q8_0(values: np.ndarray, scales: np.ndarray, n: int) -> np.ndarray:
    """Dequantize Q8_0 blocks back to float32."""
    q = np.asarray(values, dtype=np.float32).reshape(-1)
    s = np.asarray(scales, dtype=np.float32).reshape(-1)
    if q.size % GGML_BLOCK_SIZE:
        raise ValueError(f"Q8_0 values must hold whole blocks, got {q.size} elements")
    if s.size != q.size // GGML_BLOCK_SIZE:
        raise ValueError(f"expected {q.size // GGML_BLOCK_SIZE} Q8_0 scales, got {s.size}")
    blocks = q.reshape(-1, GGML_BLOCK_SIZE) * s[:, None]
    out = blocks.reshape(-1)
    if n < 0 or n > out.size:
        raise ValueError(f"cannot reconstruct {n} elements from {out.size} available")
    return out[:n]

quantize_q4_0

quantize_q4_0(data)

Quantize float data to Q4_0 blocks.

Byte-compatible with ggml's quantize_row_q4_0_reference so the packed tensor is readable by llama.cpp / the wider GGUF ecosystem.

参数：

名称	类型	描述	默认
data	ndarray	Float array with a multiple of 32 elements (any shape is flattened row-major).	必需

返回：

类型	描述
ndarray	(packed, scales) where packed is uint8 with one byte
ndarray	per two elements and scales is float16 with one (negative,
tuple[ndarray, ndarray]	per ggml) scale per 32-element block.

源代码位于： src/llm/export/gguf/quant.py

def quantize_q4_0(data: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """Quantize float data to Q4_0 blocks.

    Byte-compatible with ggml's ``quantize_row_q4_0_reference`` so the
    packed tensor is readable by llama.cpp / the wider GGUF ecosystem.

    Args:
        data: Float array with a multiple of 32 elements (any shape is
            flattened row-major).

    Returns:
        ``(packed, scales)`` where ``packed`` is ``uint8`` with one byte
        per two elements and ``scales`` is ``float16`` with one (negative,
        per ggml) scale per 32-element block.
    """
    x = _as_flat_float32(data, "Q4_0")
    blocks = x.reshape(-1, GGML_BLOCK_SIZE)
    scale = _q4_0_block_scales(blocks)  # negative: max / -8
    inv = _safe_inverse(scale)
    scaled = blocks * inv[:, None]
    # ggml: q = (int8_t)(x * (-8/max) + 8.5) == (x / d) + 8.5, clipped to
    # [0, 15]; the C cast truncates toward zero (values here are clipped
    # into the positive range so trunc == floor).
    nibbles = np.trunc(scaled + 8.5).astype(np.int16)
    nibbles = np.clip(nibbles, 0, 15).astype(np.uint8)
    low = nibbles[:, : GGML_BLOCK_SIZE // 2]
    high = nibbles[:, GGML_BLOCK_SIZE // 2 :]
    packed = (low | (high << 4)).reshape(-1)
    return packed, scale.astype(np.float16)

quantize_q8_0

quantize_q8_0(data)

Quantize float data to Q8_0 blocks.

Returns (values, scales) where values is int8 (one per element) and scales is float16 (one per 32-element block).

源代码位于： src/llm/export/gguf/quant.py

def quantize_q8_0(data: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """Quantize float data to Q8_0 blocks.

    Returns ``(values, scales)`` where ``values`` is ``int8`` (one per
    element) and ``scales`` is ``float16`` (one per 32-element block).
    """
    x = _as_flat_float32(data, "Q8_0")
    blocks = x.reshape(-1, GGML_BLOCK_SIZE)
    amax = np.max(np.abs(blocks), axis=1)
    scale = amax / _Q8_0_MAX
    inv = _safe_inverse(scale)
    scaled = blocks * inv[:, None]
    values = np.clip(_round_half_away_from_zero(scaled), -128, 127).astype(np.int8)
    return values.reshape(-1), scale.astype(np.float16)

Shared Wrapper

_wrapper

Shared helpers for export backends.

Right now the only shared piece is the cache-contract wrapper used by every trace-based export target (torch.onnx.export, torch.jit.trace). Both exporters need the model to be called with use_cache=False and to return a single tensor so the tracer doesn't record KV-cache boolean conditionals or shape expressions.

This module is intentionally tiny — it only holds what two or more backends need. Anything specific to a single backend stays in that backend's file.

ExportCacheWrapper

Bases: Module

Wrap a model so trace-based exporters see a clean contract.

Forces use_cache=False (avoiding KV-cache tracer branching) and unwraps the (logits, kv_cache) tuple to just logits so the traced graph's output is a single tensor.

The class is shared across every trace-based backend. script backends don't need it, but using it is harmless — the wrapper is just a thin nn.Module subclass.

源代码位于： src/llm/export/_wrapper.py

class ExportCacheWrapper(nn.Module):
    """Wrap a model so trace-based exporters see a clean contract.

    Forces ``use_cache=False`` (avoiding KV-cache tracer branching)
    and unwraps the ``(logits, kv_cache)`` tuple to just ``logits``
    so the traced graph's output is a single tensor.

    The class is shared across every trace-based backend. ``script``
    backends don't need it, but using it is harmless — the wrapper
    is just a thin ``nn.Module`` subclass.
    """

    def __init__(self, model: nn.Module) -> None:
        super().__init__()
        self.model = model

    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
        output = self.model(input_ids, use_cache=False)
        if isinstance(output, tuple):
            return output[0]
        return output

model_vocab_size

model_vocab_size(model)

Return the model's vocab size (embedding row count), or None.

Trace-based exporters build a random token-id dummy input and must bound it by the REAL vocabulary — a hardcoded randint(0, 100) crashes with IndexError inside the embedding for any model with vocab_size < 100 (RIL ISS-058). This helper resolves the vocab from the common embedding layouts (DecoderModel and friends); returns None when the model exposes no recognizable embedding so callers keep their historical default.

源代码位于： src/llm/export/_wrapper.py

def model_vocab_size(model: nn.Module) -> int | None:
    """Return the model's vocab size (embedding row count), or ``None``.

    Trace-based exporters build a random token-id dummy input and must
    bound it by the REAL vocabulary — a hardcoded ``randint(0, 100)``
    crashes with ``IndexError`` inside the embedding for any model with
    ``vocab_size < 100`` (RIL ISS-058). This helper resolves the vocab
    from the common embedding layouts (``DecoderModel`` and friends);
    returns ``None`` when the model exposes no recognizable embedding so
    callers keep their historical default.
    """
    embedding = getattr(model, "embedding_layer", None)
    token_embeddings = getattr(embedding, "token_embeddings", None)
    num_embeddings = getattr(token_embeddings, "num_embeddings", None)
    if isinstance(num_embeddings, int):
        return num_embeddings
    # Fallbacks for models that expose an HF-style ``get_input_embeddings``.
    getter = getattr(model, "get_input_embeddings", None)
    if callable(getter):
        try:
            emb = getter()
        except Exception:  # noqa: BLE001 - defensive; vocab is best-effort
            return None
        num = getattr(emb, "num_embeddings", None)
        if isinstance(num, int):
            return num
    return None

dummy_token_ids

dummy_token_ids(model, shape, *, device=None)

Build a random token-id dummy input bounded by the model's vocab.

参数：

名称	类型	描述	默认
model	Module	The model being exported.	必需
shape	tuple[int, int]	(batch_size, seq_len).	必需
device	device | str | None	Torch device for the tensor.	None

Uses :func:model_vocab_size; falls back to the historical 100 upper bound when the vocab can't be resolved (keeps existing behaviour for models without a discoverable embedding).

源代码位于： src/llm/export/_wrapper.py

def dummy_token_ids(
    model: nn.Module,
    shape: tuple[int, int],
    *,
    device: torch.device | str | None = None,
) -> torch.Tensor:
    """Build a random token-id dummy input bounded by the model's vocab.

    Args:
        model: The model being exported.
        shape: ``(batch_size, seq_len)``.
        device: Torch device for the tensor.

    Uses :func:`model_vocab_size`; falls back to the historical ``100``
    upper bound when the vocab can't be resolved (keeps existing
    behaviour for models without a discoverable embedding).
    """
    batch_size, seq_len = shape
    upper = model_vocab_size(model)
    if upper is None:
        upper = 100
    # ``randint(0, upper)`` yields ids in ``[0, upper)`` — always < vocab.
    return torch.randint(0, upper, (batch_size, seq_len), device=device)