`llm.evaluation` — Metrics, Tasks, and Harness Adapters

The evaluation subpackage is split into two slices:

Metrics + offline tasks (llm.evaluation.metrics, llm.evaluation.eval_tasks) — pure-Python accuracy/F1/perplexity helpers and the offline task protocol. See metrics.base for the task/metric contract.
lm-evaluation-harness adapter (llm.evaluation.harness) — the thin shim that lets our DecoderModel plug into the upstream lm-evaluation-harness benchmark suite.

This page documents the harness slice. lm_eval is an optional dependency; importing the modules below never crashes on a host that doesn't have it installed — only instantiation raises.

Benchmark Presets

EvalPreset bundles a benchmark name with the kwargs that lm_eval.evaluator.evaluate understands. Three built-in presets ship out of the box; users can construct their own by passing the same fields.

presets

Benchmark presets for the lm-eval-harness pipeline.

A preset bundles a benchmark name with the lm_eval kwargs that evaluator.evaluate expects (num_fewshot, batch_size, limit, etc.). Three common presets ship out of the box; users can extend by constructing :class:EvalPreset directly.

The presets are intentionally decoupled from the lm_eval import so this module is safe to import on hosts that don't have lm_eval installed. Callers that actually want to run the benchmark should use :func:llm.evaluation.harness.adapter.run_preset, which is the boundary that imports lm_eval.

EvalPreset `dataclass`

A benchmark preset for the lm-eval-harness pipeline.

属性：

名称	类型	描述
`task`	`str`	lm_eval task name (e.g. `"mmlu"`, `"arc_easy"`, `"wikitext"`).
`num_fewshot`	`int \| None`	Number of few-shot exemplars. `None` lets lm_eval use the task's default.
`batch_size`	`int`	Per-device evaluation batch size.
`limit`	`int \| None`	Optional cap on the number of samples per task (`None` means run the full benchmark).
`task_kwargs`	`dict[str, Any]`	Extra kwargs forwarded to `lm_eval.tasks` (e.g. `{"subject": "abstract_algebra"}` for a single MMLU subject).
`description`	`str`	Human-readable one-liner for the report.

源代码位于： src/llm/evaluation/harness/presets.py

@dataclass(frozen=True)
class EvalPreset:
    """A benchmark preset for the lm-eval-harness pipeline.

    Attributes:
        task: lm_eval task name (e.g. ``"mmlu"``, ``"arc_easy"``,
            ``"wikitext"``).
        num_fewshot: Number of few-shot exemplars. ``None`` lets
            lm_eval use the task's default.
        batch_size: Per-device evaluation batch size.
        limit: Optional cap on the number of samples per task
            (``None`` means run the full benchmark).
        task_kwargs: Extra kwargs forwarded to ``lm_eval.tasks``
            (e.g. ``{"subject": "abstract_algebra"}`` for a single
            MMLU subject).
        description: Human-readable one-liner for the report.
    """

    task: str
    num_fewshot: int | None = None
    batch_size: int = 8
    limit: int | None = None
    task_kwargs: dict[str, Any] = field(default_factory=dict)
    description: str = ""

    def to_lm_eval_kwargs(self) -> dict[str, Any]:
        """Flatten to the kwargs ``evaluator.evaluate`` understands.

        Always returns a fresh dict so callers can mutate it without
        poisoning the frozen preset.
        """
        kwargs: dict[str, Any] = {"tasks": [self.task], "batch_size": self.batch_size}
        if self.num_fewshot is not None:
            kwargs["num_fewshot"] = self.num_fewshot
        if self.limit is not None:
            kwargs["limit"] = self.limit
        if self.task_kwargs:
            kwargs["task_kwargs"] = dict(self.task_kwargs)
        return kwargs

to_lm_eval_kwargs

to_lm_eval_kwargs()

Flatten to the kwargs evaluator.evaluate understands.

Always returns a fresh dict so callers can mutate it without poisoning the frozen preset.

源代码位于： src/llm/evaluation/harness/presets.py

def to_lm_eval_kwargs(self) -> dict[str, Any]:
    """Flatten to the kwargs ``evaluator.evaluate`` understands.

    Always returns a fresh dict so callers can mutate it without
    poisoning the frozen preset.
    """
    kwargs: dict[str, Any] = {"tasks": [self.task], "batch_size": self.batch_size}
    if self.num_fewshot is not None:
        kwargs["num_fewshot"] = self.num_fewshot
    if self.limit is not None:
        kwargs["limit"] = self.limit
    if self.task_kwargs:
        kwargs["task_kwargs"] = dict(self.task_kwargs)
    return kwargs

get_preset

get_preset(name)

Look up a built-in preset by name.

引发：

类型	描述
`KeyError`	if `name` is not one of the built-in presets.

源代码位于： src/llm/evaluation/harness/presets.py

def get_preset(name: str) -> EvalPreset:
    """Look up a built-in preset by name.

    Raises:
        KeyError: if ``name`` is not one of the built-in presets.
    """
    if name not in BUILTIN_PRESETS:
        available = ", ".join(sorted(BUILTIN_PRESETS))
        raise KeyError(f"unknown preset {name!r}; available built-ins: {available}")
    return BUILTIN_PRESETS[name]

`LlamaLmEvalLM` — `DecoderModel` adapter for lm_eval

Minimal lm_eval.api.model.LM implementation that wraps a DecoderModel + tokenizer. Implements the three protocol methods (loglikelihood, loglikelihood_rolling, generate_until) without pulling in HFLM's HF-only kwargs (prefix_token, backend).

lm_eval_lm

lm_eval LM adapter for our :class:DecoderModel.

lm-evaluation-harness expects model wrappers to implement the lm_eval.api.model.LM protocol (loglikelihood, loglikelihood_rolling, generate_until). This module provides a minimal adapter that conforms to that interface so any trained :class:DecoderModel + tokenizer can be evaluated with lm_eval.evaluator.evaluate(lm=LlamaLmEvalLM(model, tokenizer)).

Soft dependency on lm_eval — this module imports lazily inside __init__ so importing :mod:llm.evaluation.harness.lm_eval_lm never raises on hosts without lm_eval installed. The ImportError fires at __init__ time with the install hint.

Why a dedicated wrapper rather than reusing :class:HFLM? HFLM's surface requires a torch.device and several HF-only arguments (prefix_token, backend) that don't apply here. A 100-line minimal adapter keeps the dependency tree honest and the contract obvious.

LlamaLmEvalLM

Minimal :class:lm_eval.api.model.LM adapter for DecoderModel.

Implements just enough of the lm_eval protocol to run the standard multiple-choice (loglikelihood) and generation (generate_until) tasks. Each request is processed individually with a single forward pass; batch_size controls how many requests are handled between no_grad context rebuilds and Python-side bookkeeping, not the model's batch dimension.

参数：

名称	类型	描述	默认
`model`	`Any`	A trained :class:`llm.models.DecoderModel`.	必需
`tokenizer`	`Any`	Tokenizer with `encode`, `decode`, `bos_token_id` (optional), `eos_token_id`. Must produce token ids accepted by the model's vocabulary.	必需
`batch_size`	`int`	Maximum number of requests per forward pass.	`8`
`max_length`	`int \| None`	Hard cap on sequence length (model's `max_seq_len` is the default).	`None`
`device`	`str \| device \| None`	Target device; defaults to the model's parameter device.	`None`

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py

class LlamaLmEvalLM:
    """Minimal :class:`lm_eval.api.model.LM` adapter for ``DecoderModel``.

    Implements just enough of the lm_eval protocol to run the
    standard multiple-choice (loglikelihood) and generation
    (generate_until) tasks. Each request is processed individually
    with a single forward pass; ``batch_size`` controls how many
    requests are handled between ``no_grad`` context rebuilds and
    Python-side bookkeeping, not the model's batch dimension.

    Args:
        model: A trained :class:`llm.models.DecoderModel`.
        tokenizer: Tokenizer with ``encode``, ``decode``, ``bos_token_id``
            (optional), ``eos_token_id``. Must produce token ids
            accepted by the model's vocabulary.
        batch_size: Maximum number of requests per forward pass.
        max_length: Hard cap on sequence length (model's
            ``max_seq_len`` is the default).
        device: Target device; defaults to the model's parameter device.
    """

    def __init__(
        self,
        model: Any,
        tokenizer: Any,
        *,
        batch_size: int = 8,
        max_length: int | None = None,
        device: str | torch.device | None = None,
    ) -> None:
        _require_lm_eval()
        # We import lazily so this module is import-safe without lm_eval.
        from lm_eval.api.model import LM  # noqa: F401  (validates install)

        self.model = model
        self.tokenizer = tokenizer
        self.batch_size = batch_size
        self.max_length = max_length or getattr(model, "max_seq_len", 2048)
        if device is None:
            device = next(model.parameters()).device
        self.device = torch.device(device) if not isinstance(device, torch.device) else device
        self.model.eval()

    # --- lm_eval protocol surface ------------------------------------------

    def loglikelihood(self, requests):
        """Compute (log_likelihood, is_greedy_match) for each request.

        Each request is an ``lm_eval.api.request.Request`` whose
        ``args`` is ``(context_str, continuation_str)``. Returns a
        list of ``(sum_logprob, is_greedy)`` tuples.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._loglikelihood_batch(batch)
                results.extend(batch_results)
        return results

    def loglikelihood_rolling(self, requests):
        """Compute total log-probability of each string (perplexity-style).

        Returns ``list[float]`` — one scalar sum-log-prob per request,
        matching the lm_eval ``LM.loglikelihood_rolling`` protocol.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._loglikelihood_rolling_batch(batch)
                results.extend(batch_results)
        return results

    def generate_until(self, requests):
        """Greedy generation until ``until`` token sequences appear.

        Each request is an ``lm_eval.api.request.Request`` whose
        ``args`` is ``(context_str, {"until": [...], "max_gen_toks": N})``.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._generate_until_batch(batch)
                results.extend(batch_results)
        return results

    # --- batched implementations -------------------------------------------

    def _loglikelihood_batch(self, batch):
        """Tokenize + forward + per-token log-prob extraction for a batch.

        For each (context, continuation):
        1. Concatenate ``context + continuation`` token ids.
        2. Pad to the longest sequence in the batch.
        3. Forward; collect log-probs at each continuation position.
        4. Sum and compare to greedy argmax for ``is_greedy_match``.
        """
        results = []
        # Tokenize once, defer padding to after we know the longest length.
        encoded: list[tuple[list[int], list[int]]] = []
        for req in batch:
            context, continuation = req.args
            ctx_ids = self._encode(context)
            cont_ids = self._encode(continuation)
            if not cont_ids:
                # lm_eval generally guarantees a non-empty continuation,
                # but guard against pathological inputs.
                cont_ids = [0]
            encoded.append((ctx_ids, cont_ids))

        max_len = min(
            self.max_length,
            max(len(c) + len(k) for c, k in encoded),
        )

        for ctx_ids, cont_ids in encoded:
            full = (ctx_ids + cont_ids)[-max_len:]
            ctx_len = max(0, len(full) - len(cont_ids))
            cont_len = len(cont_ids)

            input_tensor = torch.tensor([full], dtype=torch.long, device=self.device)
            model_out = self.model(input_tensor, use_cache=False)
            logits = model_out[0] if isinstance(model_out, tuple) else model_out
            # Log-probs at each continuation position: logits at
            # ``ctx_len - 1 + i`` predicts ``full[ctx_len + i]``.
            relevant = logits[0, max(0, ctx_len - 1) : ctx_len - 1 + cont_len, :]
            log_probs = torch.log_softmax(relevant, dim=-1)
            cont_tensor = torch.tensor(full[ctx_len:], device=self.device, dtype=torch.long)
            target_log_probs = log_probs[torch.arange(cont_len, device=self.device), cont_tensor]
            sum_logprob = float(target_log_probs.sum().item())

            # Greedy match: argmax at each continuation position
            # equals the continuation token.
            greedy_tokens = relevant.argmax(dim=-1)
            is_greedy = bool(torch.equal(greedy_tokens, cont_tensor))

            results.append((sum_logprob, is_greedy))
        return results

    def _loglikelihood_rolling_batch(self, batch):
        """Sum log-probs across every token of each request's string.

        Returns one ``float`` per request (NOT a tuple) so the result
        is compatible with the lm_eval ``loglikelihood_rolling``
        protocol: it appends each element to ``req.resps`` and
        downstream code (e.g. WikiText perplexity) does
        ``(loglikelihood,) = results`` — unpacking a 1-tuple here
        would corrupt the metric tuples downstream.
        """
        results = []
        for req in batch:
            (text,) = req.args
            ids = self._encode(text)[: self.max_length]
            if len(ids) < 2:
                results.append(0.0)
                continue
            input_tensor = torch.tensor([ids], dtype=torch.long, device=self.device)
            model_out = self.model(input_tensor, use_cache=False)
            logits = model_out[0] if isinstance(model_out, tuple) else model_out
            # Log-probs at positions 0..len-1 predict ids 1..len.
            relevant = logits[0, :-1, :]
            log_probs = torch.log_softmax(relevant, dim=-1)
            targets = torch.tensor(ids[1:], device=self.device, dtype=torch.long)
            token_log_probs = log_probs[torch.arange(len(targets), device=self.device), targets]
            results.append(float(token_log_probs.sum().item()))
        return results

    def _generate_until_batch(self, batch):
        """Greedy ``generate_until`` for a batch of requests."""
        results = []
        for req in batch:
            context, gen_kwargs = req.args
            until = gen_kwargs.get("until", [])
            max_gen_toks = int(gen_kwargs.get("max_gen_toks", 64))

            ctx_ids = self._encode(context)
            generated: list[int] = []
            for _ in range(max_gen_toks):
                full = (ctx_ids + generated)[-self.max_length :]
                input_tensor = torch.tensor([full], dtype=torch.long, device=self.device)
                model_out = self.model(input_tensor, use_cache=False)
                logits = model_out[0] if isinstance(model_out, tuple) else model_out
                next_token = int(logits[0, -1, :].argmax(dim=-1).item())
                generated.append(next_token)
                # Stop if the suffix matches any ``until`` token sequence.
                if self._matches_any_suffix(generated, until):
                    break

            results.append(self.tokenizer.decode(generated))
        return results

    # --- helpers ------------------------------------------------------------

    def _encode(self, text: str) -> list[int]:
        """Encode text using the bound tokenizer, with BOS handling."""
        ids = self.tokenizer.encode(text)
        if not isinstance(ids, list):
            ids = ids.tolist() if hasattr(ids, "tolist") else list(ids)
        return list(ids)

    @staticmethod
    def _matches_any_suffix(generated: list[int], until: list) -> bool:
        """Return True if any of the ``until`` token sequences is a suffix of ``generated``.

        ``until`` entries are expected to be token-id sequences
        (``list[int]``). String entries are skipped silently because
        we can't match by text without a tokenizer round-trip here
        — callers that want string-based stopping should tokenize
        them into id sequences first.
        """
        if not until:
            return False
        for stop in until:
            if isinstance(stop, str):
                # lm_eval sometimes passes strings; we don't decode here,
                # so we can't match by text. Skip string stops rather
                # than failing the whole generate call.
                continue
            stop_ids = list(stop) if not isinstance(stop, list) else stop
            if not stop_ids:
                continue
            if len(generated) >= len(stop_ids) and generated[-len(stop_ids) :] == stop_ids:
                return True
        return False

loglikelihood

loglikelihood(requests)

Compute (log_likelihood, is_greedy_match) for each request.

Each request is an lm_eval.api.request.Request whose args is (context_str, continuation_str). Returns a list of (sum_logprob, is_greedy) tuples.

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py

def loglikelihood(self, requests):
    """Compute (log_likelihood, is_greedy_match) for each request.

    Each request is an ``lm_eval.api.request.Request`` whose
    ``args`` is ``(context_str, continuation_str)``. Returns a
    list of ``(sum_logprob, is_greedy)`` tuples.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._loglikelihood_batch(batch)
            results.extend(batch_results)
    return results

loglikelihood_rolling

loglikelihood_rolling(requests)

Compute total log-probability of each string (perplexity-style).

Returns list[float] — one scalar sum-log-prob per request, matching the lm_eval LM.loglikelihood_rolling protocol.

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py

def loglikelihood_rolling(self, requests):
    """Compute total log-probability of each string (perplexity-style).

    Returns ``list[float]`` — one scalar sum-log-prob per request,
    matching the lm_eval ``LM.loglikelihood_rolling`` protocol.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._loglikelihood_rolling_batch(batch)
            results.extend(batch_results)
    return results

generate_until

generate_until(requests)

Greedy generation until until token sequences appear.

Each request is an lm_eval.api.request.Request whose args is (context_str, {"until": [...], "max_gen_toks": N}).

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py

def generate_until(self, requests):
    """Greedy generation until ``until`` token sequences appear.

    Each request is an ``lm_eval.api.request.Request`` whose
    ``args`` is ``(context_str, {"until": [...], "max_gen_toks": N})``.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._generate_until_batch(batch)
            results.extend(batch_results)
    return results

`LmEvalAdapter` — top-level driver

Preset lookup, kwarg merging, and structured result flattening on top of lm_eval.evaluator.

adapter

Adapter for lm-evaluation-harness.

Two responsibilities:

LmEvalAdapter — thin wrapper around lm_eval.evaluator that adds structured result handling and preset support.
run_preset — convenience that ties a preset, an LM, and the evaluator together.

The :mod:llm.evaluation.harness.presets module is safe to import without lm_eval installed; the lm_eval import boundary lives here so the project's existing [eval] optional-dependency group keeps working.

LmEvalAdapter

Adapter for lm-evaluation-harness.

Adds:

Preset lookup — run_preset(preset_name, lm) resolves a :class:EvalPreset by name (built-in or user-supplied) and runs the benchmark.
Structured result flattening — :meth:summarize extracts acc / acc_norm / perplexity / f1 / etc. from the nested lm_eval result shape into a flat {task_name: {metric: value}} dict.

源代码位于： src/llm/evaluation/harness/adapter.py

class LmEvalAdapter:
    """Adapter for lm-evaluation-harness.

    Adds:

    * **Preset lookup** — ``run_preset(preset_name, lm)`` resolves a
      :class:`EvalPreset` by name (built-in or user-supplied) and
      runs the benchmark.
    * **Structured result flattening** — :meth:`summarize` extracts
      ``acc`` / ``acc_norm`` / ``perplexity`` / ``f1`` / etc. from
      the nested lm_eval result shape into a flat
      ``{task_name: {metric: value}}`` dict.
    """

    def __init__(self) -> None:
        _require_lm_eval()
        from lm_eval.tasks import TaskManager

        self._task_manager = TaskManager()

    def list_tasks(self) -> list[str]:
        """List available benchmark tasks."""
        return sorted(self._task_manager.all_tasks)

    def evaluate(self, model: Any, tasks: list[str] | None = None, **kwargs: Any) -> dict:
        """Run evaluation on specified tasks.

        Mirrors the lm_eval ``evaluator.evaluate`` signature; see
        ``lm_eval`` docs for the kwargs surface.
        """
        _require_lm_eval()
        from lm_eval import evaluator

        return evaluator.evaluate(model=model, tasks=tasks or ["mmlu"], **kwargs)

    def run_preset(
        self,
        preset: EvalPreset | str,
        model: Any,
        **kwargs: Any,
    ) -> dict:
        """Run a benchmark by preset (name or :class:`EvalPreset`).

        Merges the preset's :meth:`EvalPreset.to_lm_eval_kwargs` with
        any caller-supplied ``kwargs`` (caller wins on conflicts).
        """
        if isinstance(preset, str):
            preset = get_preset(preset)
        merged = preset.to_lm_eval_kwargs()
        merged.update(kwargs)
        return self.evaluate(model, **merged)

    def run_benchmark(self, model: Any, benchmark: str, **kwargs: Any) -> dict:
        """Run a single benchmark task by name (no preset lookup)."""
        return self.evaluate(model, tasks=[benchmark], **kwargs)

    @staticmethod
    def summarize(results: dict[str, Any]) -> dict[str, dict[str, float]]:
        """Flatten lm_eval's nested result tree into a flat metric map.

        lm_eval's ``evaluator.simple_evaluate`` returns:

        .. code-block:: python

            {
                "results": {
                    "task_name": {
                        "acc,none": 0.42,
                        "acc_norm,none": 0.45,
                        ...
                    },
                    ...
                },
                "groups": {...},
                "configs": {...},
            }

        This helper extracts the ``results`` block and splits each
        comma-separated key (``"acc,none"`` -> metric ``"acc"``,
        subset ``"none"``) so callers can serialize it as a flat
        dict.

        Notes:

        - Only the ``results`` block is flattened; ``groups`` and
          ``configs`` are intentionally ignored.
        - Booleans are dropped (Python's ``bool`` is a subclass of
          ``int``, but a metric of value ``True`` is almost certainly
          a bug — string aliases are usually what you'd see).
        - Non-numeric values (e.g. string aliases) are silently
          skipped, keeping the output strictly numeric.
        """
        flat: dict[str, dict[str, float]] = {}
        for task_name, metrics in results.get("results", {}).items():
            flat[task_name] = {}
            for key, value in metrics.items():
                metric_name = key.split(",", 1)[0] if "," in key else key
                if isinstance(value, bool):
                    # bool is technically int — skip to avoid ``True`` -> 1.0.
                    continue
                if isinstance(value, (int, float)):
                    flat[task_name][metric_name] = float(value)
        return flat

list_tasks

list_tasks()

List available benchmark tasks.

源代码位于： src/llm/evaluation/harness/adapter.py

def list_tasks(self) -> list[str]:
    """List available benchmark tasks."""
    return sorted(self._task_manager.all_tasks)

evaluate

evaluate(model, tasks=None, **kwargs)

Run evaluation on specified tasks.

Mirrors the lm_eval evaluator.evaluate signature; see lm_eval docs for the kwargs surface.

源代码位于： src/llm/evaluation/harness/adapter.py

def evaluate(self, model: Any, tasks: list[str] | None = None, **kwargs: Any) -> dict:
    """Run evaluation on specified tasks.

    Mirrors the lm_eval ``evaluator.evaluate`` signature; see
    ``lm_eval`` docs for the kwargs surface.
    """
    _require_lm_eval()
    from lm_eval import evaluator

    return evaluator.evaluate(model=model, tasks=tasks or ["mmlu"], **kwargs)

run_preset

run_preset(preset, model, **kwargs)

Run a benchmark by preset (name or :class:EvalPreset).

Merges the preset's :meth:EvalPreset.to_lm_eval_kwargs with any caller-supplied kwargs (caller wins on conflicts).

源代码位于： src/llm/evaluation/harness/adapter.py

def run_preset(
    self,
    preset: EvalPreset | str,
    model: Any,
    **kwargs: Any,
) -> dict:
    """Run a benchmark by preset (name or :class:`EvalPreset`).

    Merges the preset's :meth:`EvalPreset.to_lm_eval_kwargs` with
    any caller-supplied ``kwargs`` (caller wins on conflicts).
    """
    if isinstance(preset, str):
        preset = get_preset(preset)
    merged = preset.to_lm_eval_kwargs()
    merged.update(kwargs)
    return self.evaluate(model, **merged)

run_benchmark

run_benchmark(model, benchmark, **kwargs)

Run a single benchmark task by name (no preset lookup).

源代码位于： src/llm/evaluation/harness/adapter.py

def run_benchmark(self, model: Any, benchmark: str, **kwargs: Any) -> dict:
    """Run a single benchmark task by name (no preset lookup)."""
    return self.evaluate(model, tasks=[benchmark], **kwargs)

summarize `staticmethod`

summarize(results)

Flatten lm_eval's nested result tree into a flat metric map.

lm_eval's evaluator.simple_evaluate returns:

.. code-block:: python

{
    "results": {
        "task_name": {
            "acc,none": 0.42,
            "acc_norm,none": 0.45,
            ...
        },
        ...
    },
    "groups": {...},
    "configs": {...},
}

This helper extracts the results block and splits each comma-separated key ("acc,none" -> metric "acc", subset "none") so callers can serialize it as a flat dict.

Notes:

Only the results block is flattened; groups and configs are intentionally ignored.
Booleans are dropped (Python's bool is a subclass of int, but a metric of value True is almost certainly a bug — string aliases are usually what you'd see).
Non-numeric values (e.g. string aliases) are silently skipped, keeping the output strictly numeric.

源代码位于： src/llm/evaluation/harness/adapter.py

@staticmethod
def summarize(results: dict[str, Any]) -> dict[str, dict[str, float]]:
    """Flatten lm_eval's nested result tree into a flat metric map.

    lm_eval's ``evaluator.simple_evaluate`` returns:

    .. code-block:: python

        {
            "results": {
                "task_name": {
                    "acc,none": 0.42,
                    "acc_norm,none": 0.45,
                    ...
                },
                ...
            },
            "groups": {...},
            "configs": {...},
        }

    This helper extracts the ``results`` block and splits each
    comma-separated key (``"acc,none"`` -> metric ``"acc"``,
    subset ``"none"``) so callers can serialize it as a flat
    dict.

    Notes:

    - Only the ``results`` block is flattened; ``groups`` and
      ``configs`` are intentionally ignored.
    - Booleans are dropped (Python's ``bool`` is a subclass of
      ``int``, but a metric of value ``True`` is almost certainly
      a bug — string aliases are usually what you'd see).
    - Non-numeric values (e.g. string aliases) are silently
      skipped, keeping the output strictly numeric.
    """
    flat: dict[str, dict[str, float]] = {}
    for task_name, metrics in results.get("results", {}).items():
        flat[task_name] = {}
        for key, value in metrics.items():
            metric_name = key.split(",", 1)[0] if "," in key else key
            if isinstance(value, bool):
                # bool is technically int — skip to avoid ``True`` -> 1.0.
                continue
            if isinstance(value, (int, float)):
                flat[task_name][metric_name] = float(value)
    return flat

run_preset

run_preset(preset, model, **kwargs)

Convenience entry point: build an adapter and run a preset.

源代码位于： src/llm/evaluation/harness/adapter.py

def run_preset(
    preset: EvalPreset | str,
    model: Any,
    **kwargs: Any,
) -> dict:
    """Convenience entry point: build an adapter and run a preset."""
    return LmEvalAdapter().run_preset(preset, model, **kwargs)

End-to-end usage

See the Evaluation guide for a worked example (preset selection, result flattening, soft-dependency contract).

llm.evaluation — Metrics, Tasks, and Harness Adapters

Benchmark Presets

presets

EvalPreset dataclass

to_lm_eval_kwargs

get_preset

LlamaLmEvalLM — DecoderModel adapter for lm_eval

lm_eval_lm

LlamaLmEvalLM

loglikelihood

loglikelihood_rolling

generate_until

LmEvalAdapter — top-level driver

adapter

LmEvalAdapter

list_tasks

evaluate

run_preset

run_benchmark

summarize staticmethod

run_preset

End-to-end usage

`llm.evaluation` — Metrics, Tasks, and Harness Adapters

EvalPreset `dataclass`

`LlamaLmEvalLM` — `DecoderModel` adapter for lm_eval

`LmEvalAdapter` — top-level driver

summarize `staticmethod`