跳转至

llm.evaluation — Metrics, Tasks, and Harness Adapters

The evaluation subpackage is split into two slices:

  • Metrics + offline tasks (llm.evaluation.metrics, llm.evaluation.eval_tasks) — pure-Python accuracy/F1/perplexity helpers and the offline task protocol. See below for the full API reference.
  • lm-evaluation-harness adapter (llm.evaluation.harness) — the thin shim that lets our DecoderModel plug into the upstream lm-evaluation-harness benchmark suite.

The metrics and task bases have no optional dependencies; the harness slice is gated behind the lm_eval optional dependency — importing the harness modules below never crashes on a host that doesn't have it installed, only instantiation raises.

Metrics

Abstract base class and concrete scoring implementations for evaluation.

base

BaseMetric

Bases: ABC

Base class for all evaluation metrics.

源代码位于： src/llm/evaluation/metrics/base.py 
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
class BaseMetric(ABC):
    """Base class for all evaluation metrics."""

    name: str

    @abstractmethod
    def compute(self, predictions: Any, references: Any) -> dict:
        """Compute metric score.

        Args:
            predictions: Model outputs
            references: Ground truth

        Returns:
            Dictionary with metric name and score
        """
        pass

compute abstractmethod 

compute(predictions, references)

Compute metric score.

参数：

名称 类型 描述 默认
predictions Any

Model outputs

 必需
references Any

Ground truth

 必需

返回：

类型 描述
dict

Dictionary with metric name and score
源代码位于： src/llm/evaluation/metrics/base.py 
10
11
12
13
14
15
16
17
18
19
20
21
@abstractmethod
def compute(self, predictions: Any, references: Any) -> dict:
    """Compute metric score.

    Args:
        predictions: Model outputs
        references: Ground truth

    Returns:
        Dictionary with metric name and score
    """
    pass

accuracy

AccuracyMetric

Bases: BaseMetric

Accuracy metric for classification tasks.

源代码位于： src/llm/evaluation/metrics/accuracy.py 
 6
 7
 8
 9
10
11
12
13
14
15
class AccuracyMetric(BaseMetric):
    """Accuracy metric for classification tasks."""

    name = "accuracy"

    def compute(self, predictions: list, references: list) -> dict:
        """Compute accuracy."""
        correct = sum(p == r for p, r in zip(predictions, references, strict=True))
        acc = correct / len(predictions) if predictions else 0.0
        return {"accuracy": acc}

compute 

compute(predictions, references)

Compute accuracy.

源代码位于： src/llm/evaluation/metrics/accuracy.py 
11
12
13
14
15
def compute(self, predictions: list, references: list) -> dict:
    """Compute accuracy."""
    correct = sum(p == r for p, r in zip(predictions, references, strict=True))
    acc = correct / len(predictions) if predictions else 0.0
    return {"accuracy": acc}

F1Metric

Bases: BaseMetric

F1 score metric for classification tasks.

Requires the scikit-learn package, available via the [eval] extra.

源代码位于： src/llm/evaluation/metrics/accuracy.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
class F1Metric(BaseMetric):
    """F1 score metric for classification tasks.

    Requires the ``scikit-learn`` package, available via the ``[eval]`` extra.
    """

    name = "f1"

    def __init__(self, average: str = "macro"):
        self.average = average

    def compute(self, predictions: list, references: list) -> dict:
        """Compute F1 score using sklearn.

        Returns ``{"f1": 0.0}`` for empty inputs, matching
        :meth:`AccuracyMetric.compute`'s convention.
        """
        if not predictions:
            return {"f1": 0.0}

        sklearn_metrics = import_module("sklearn.metrics")
        f1 = sklearn_metrics.f1_score(references, predictions, average=self.average, zero_division=0)
        return {"f1": f1}

compute 

compute(predictions, references)

Compute F1 score using sklearn.

Returns {"f1": 0.0} for empty inputs, matching :meth:AccuracyMetric.compute's convention.

源代码位于： src/llm/evaluation/metrics/accuracy.py 
29
30
31
32
33
34
35
36
37
38
39
40
def compute(self, predictions: list, references: list) -> dict:
    """Compute F1 score using sklearn.

    Returns ``{"f1": 0.0}`` for empty inputs, matching
    :meth:`AccuracyMetric.compute`'s convention.
    """
    if not predictions:
        return {"f1": 0.0}

    sklearn_metrics = import_module("sklearn.metrics")
    f1 = sklearn_metrics.f1_score(references, predictions, average=self.average, zero_division=0)
    return {"f1": f1}

generation

RougeMetric

Bases: BaseMetric

ROUGE metric for generation tasks.

Requires the rouge-score package, available via the [eval] extra (pip install llm[eval]).

The rouge_score import is deferred to :meth:compute (and the scorer is built lazily on first use) so the class can be instantiated on hosts without rouge-score installed — the same soft-dependency contract as :class:BleuMetric and :class:ChrFMetric.

源代码位于： src/llm/evaluation/metrics/generation.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
class RougeMetric(BaseMetric):
    """ROUGE metric for generation tasks.

    Requires the ``rouge-score`` package, available via the ``[eval]`` extra
    (``pip install llm[eval]``).

    The ``rouge_score`` import is deferred to :meth:`compute` (and the
    scorer is built lazily on first use) so the class can be instantiated
    on hosts without ``rouge-score`` installed — the same soft-dependency
    contract as :class:`BleuMetric` and :class:`ChrFMetric`.
    """

    name = "rouge"

    def __init__(self, rouge_types=None):
        self.rouge_types = rouge_types or ["rouge1", "rouge2", "rougeL"]
        self._scorer = None

    @staticmethod
    def _build_scorer(rouge_types: list[str]):
        """Import ``rouge_score`` lazily and build a ``RougeScorer``.

        Raises:
            ImportError: with an actionable install hint if
                ``rouge-score`` is not installed.
        """
        try:
            # Read the submodule off the parent package first so callers
            # can patch ``rouge_score`` in sys.modules (test contract),
            # falling back to a real submodule import on first use.
            rouge_module = import_module("rouge_score")
            rouge_scorer = rouge_module.__dict__.get("rouge_scorer") or import_module("rouge_score.rouge_scorer")
        except ImportError as exc:
            raise ImportError(
                "rouge-score is an optional evaluation dependency. Install with `pip install 'llm[eval]'`."
            ) from exc
        return rouge_scorer.RougeScorer(rouge_types, use_stemmer=True)

    def compute(self, predictions: list, references: list) -> dict:
        # Empty inputs — nothing to score, and we shouldn't require the
        # optional dependency just to short-circuit. Every sibling metric
        # reports ``0.0`` on empty input (``BleuMetric`` -> ``{"bleu": 0.0}``,
        # ``AccuracyMetric``/``F1Metric`` -> 0.0); an empty ``{}`` made the
        # per-dimension keys silently vanish from eval output and consumers
        # doing ``results["rouge-1"]`` hit a KeyError (round-73 FINDING 5).
        if not predictions:
            return {t.replace("rouge", "rouge-").lower(): 0.0 for t in self.rouge_types}

        if self._scorer is None:
            self._scorer = self._build_scorer(self.rouge_types)

        results = {}
        for pred, ref in zip(predictions, references, strict=True):
            scores = self._scorer.score(ref, pred)
            for rouge_type in self.rouge_types:
                key = rouge_type.replace("rouge", "rouge-").lower()
                if key not in results:
                    results[key] = []
                results[key].append(scores[rouge_type].fmeasure)

        return {k: sum(v) / len(v) for k, v in results.items()}

BleuMetric

Bases: BaseMetric

BLEU metric for generation tasks.

Requires the sacrebleu package, available via the [eval] extra (pip install llm[eval]). The import is deferred to :meth:compute so the class can be instantiated on hosts without sacrebleu installed — the same soft-dependency contract as :class:RougeMetric.

源代码位于： src/llm/evaluation/metrics/generation.py 
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
class BleuMetric(BaseMetric):
    """BLEU metric for generation tasks.

    Requires the ``sacrebleu`` package, available via the ``[eval]`` extra
    (``pip install llm[eval]``).  The import is deferred to :meth:`compute`
    so the class can be instantiated on hosts without ``sacrebleu``
    installed — the same soft-dependency contract as
    :class:`RougeMetric`.
    """

    name = "bleu"

    def compute(self, predictions: list, references: list) -> dict:
        # Empty inputs — nothing to score, and we shouldn't require the
        # optional dependency just to short-circuit (sacrebleu raises on an
        # empty corpus). Matches the ``0.0`` convention of
        # :class:`AccuracyMetric` / :class:`F1Metric`.
        if not predictions:
            return {"bleu": 0.0}

        try:
            sacrebleu = import_module("sacrebleu")
        except ImportError as exc:
            raise ImportError(
                "sacrebleu is an optional evaluation dependency. Install with `pip install 'llm[eval]'`."
            ) from exc

        refs = [[r] for r in references]
        bleu = sacrebleu.corpus_bleu(predictions, refs)
        return {"bleu": bleu.score}

ChrFMetric

Bases: BaseMetric

chrF metric for generation tasks.

Requires the sacrebleu package, available via the [eval] extra (pip install llm[eval]). The import is deferred to :meth:compute so the class can be instantiated on hosts without sacrebleu installed — the same soft-dependency contract as :class:RougeMetric.

源代码位于： src/llm/evaluation/metrics/generation.py 
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
class ChrFMetric(BaseMetric):
    """chrF metric for generation tasks.

    Requires the ``sacrebleu`` package, available via the ``[eval]`` extra
    (``pip install llm[eval]``).  The import is deferred to :meth:`compute`
    so the class can be instantiated on hosts without ``sacrebleu``
    installed — the same soft-dependency contract as
    :class:`RougeMetric`.
    """

    name = "chrf"

    def compute(self, predictions: list, references: list) -> dict:
        # Empty inputs — nothing to score, and we shouldn't require the
        # optional dependency just to short-circuit (sacrebleu raises on an
        # empty corpus). Matches the ``0.0`` convention of
        # :class:`AccuracyMetric` / :class:`F1Metric`.
        if not predictions:
            return {"chrf": 0.0}

        try:
            sacrebleu = import_module("sacrebleu")
        except ImportError as exc:
            raise ImportError(
                "sacrebleu is an optional evaluation dependency. Install with `pip install 'llm[eval]'`."
            ) from exc

        refs = [[r] for r in references]
        chrf = sacrebleu.corpus_chrf(predictions, refs)
        return {"chrf": chrf.score}

perplexity

PerplexityMetric

Bases: BaseMetric

Perplexity metric for language modeling evaluation.

源代码位于： src/llm/evaluation/metrics/perplexity.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
class PerplexityMetric(BaseMetric):
    """Perplexity metric for language modeling evaluation."""

    name = "perplexity"

    def __init__(self, ignore_index: int | None = None) -> None:
        """Create the metric.

        Args:
            ignore_index: Token id to exclude from the loss (typically
                the tokenizer's ``pad_token_id``). When set, positions
                whose label equals ``ignore_index`` are skipped so padded
                sequences are scored only over real tokens. ``None``
                (the default) scores every position.
        """
        self.ignore_index = ignore_index

    def compute(self, predictions: torch.Tensor, references: torch.Tensor | list) -> dict:
        """Compute perplexity.

        Args:
            predictions: Logits tensor of shape (batch, seq, vocab)
            references: Target token IDs of shape (batch, seq). A list
                of equal-length token sequences is coerced to a tensor
                so the metric works through both :meth:`EvaluationRunner.run`
                (raw path) and :meth:`EvaluationRunner.evaluate`.

        Returns:
            Dictionary with perplexity score.  ``inf`` is returned when
            the batch is empty or no shift-targets are available (e.g.
            ``seq == 1``), since perplexity is undefined in those cases.
        """
        if not isinstance(references, torch.Tensor):
            # ``run()`` passes raw (non-tensor) references: LMTask yields
            # equal-length padded sequences, so a list of tensors stacks
            # cleanly; plain lists are coerced elementwise.
            if references and isinstance(references[0], torch.Tensor):
                references = torch.stack(references)
            else:
                try:
                    references = torch.as_tensor(references, dtype=torch.long)
                except (ValueError, TypeError) as exc:
                    # Ragged (ragged-nested-list) references raise torch's raw
                    # ValueError here; surface a clear metric-level error with
                    # the actual fix instead of an opaque stack trace deep in
                    # cross_entropy (eval deep-dive F1).
                    raise ValueError(
                        "perplexity references must be a rectangular tensor/list of token "
                        f"ids, got {type(references).__name__}; ragged sequences are not "
                        "supported — pad/truncate the references to equal length first."
                    ) from exc

        # A single-sequence reference (1-D) must broadcast to one batch row,
        # not crash on ``references.shape[1]`` below with an opaque IndexError.
        references = torch.atleast_2d(references)

        batch_size = predictions.shape[0]
        if batch_size == 0:
            return {"perplexity": float("inf")}

        _batch, _seq_len, vocab_size = predictions.shape

        logits = predictions[:, :-1, :].contiguous().view(-1, vocab_size)

        # RIL ISS-192: ``LMTask.predict`` clamps inputs to the model's
        # context window (``min(max_seq_len, model.max_seq_len)``) while the
        # references are padded to the dataset's ``max_seq_len`` — for a
        # small-context model the predictions are *narrower* than the
        # references, so the naive ``references[:, 1:]`` yields more label
        # positions than logits rows and ``cross_entropy`` raises a shape
        # error mid-evaluation. Slice the labels to the prediction horizon;
        # the truncated tail was never scored anyway.
        label_width = min(references.shape[1] - 1, _seq_len - 1)
        labels = references[:, 1 : 1 + label_width].contiguous().view(-1)

        if logits.shape[0] == 0 or labels.numel() == 0:
            return {"perplexity": float("inf")}

        if self.ignore_index is not None and labels.numel() > 0 and bool((labels == self.ignore_index).all().item()):
            # Every shift-target is ignored (e.g. a 1-token corpus whose
            # shifted labels are all -100): ``cross_entropy`` with
            # ``reduction='mean'`` and ``ignore_index`` averages over ZERO
            # valid elements and returns NaN. Return the documented
            # ``inf`` (undefined perplexity) instead — NaN would serialize
            # to JSON ``null`` and poison the report (RIL ISS-055).
            return {"perplexity": float("inf")}

        kwargs = {"ignore_index": self.ignore_index} if self.ignore_index is not None else {}
        loss = functional.cross_entropy(logits, labels, reduction="mean", **kwargs)
        perplexity = torch.exp(loss).item()

        return {"perplexity": perplexity}

compute 

compute(predictions, references)

Compute perplexity.

参数：

名称 类型 描述 默认
predictions Tensor

Logits tensor of shape (batch, seq, vocab)

 必需
references Tensor | list

Target token IDs of shape (batch, seq). A list of equal-length token sequences is coerced to a tensor so the metric works through both :meth:EvaluationRunner.run (raw path) and :meth:EvaluationRunner.evaluate.

 必需

返回：

类型 描述
dict

Dictionary with perplexity score. inf is returned when
dict

the batch is empty or no shift-targets are available (e.g.
dict

seq == 1), since perplexity is undefined in those cases.
源代码位于： src/llm/evaluation/metrics/perplexity.py 
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
def compute(self, predictions: torch.Tensor, references: torch.Tensor | list) -> dict:
    """Compute perplexity.

    Args:
        predictions: Logits tensor of shape (batch, seq, vocab)
        references: Target token IDs of shape (batch, seq). A list
            of equal-length token sequences is coerced to a tensor
            so the metric works through both :meth:`EvaluationRunner.run`
            (raw path) and :meth:`EvaluationRunner.evaluate`.

    Returns:
        Dictionary with perplexity score.  ``inf`` is returned when
        the batch is empty or no shift-targets are available (e.g.
        ``seq == 1``), since perplexity is undefined in those cases.
    """
    if not isinstance(references, torch.Tensor):
        # ``run()`` passes raw (non-tensor) references: LMTask yields
        # equal-length padded sequences, so a list of tensors stacks
        # cleanly; plain lists are coerced elementwise.
        if references and isinstance(references[0], torch.Tensor):
            references = torch.stack(references)
        else:
            try:
                references = torch.as_tensor(references, dtype=torch.long)
            except (ValueError, TypeError) as exc:
                # Ragged (ragged-nested-list) references raise torch's raw
                # ValueError here; surface a clear metric-level error with
                # the actual fix instead of an opaque stack trace deep in
                # cross_entropy (eval deep-dive F1).
                raise ValueError(
                    "perplexity references must be a rectangular tensor/list of token "
                    f"ids, got {type(references).__name__}; ragged sequences are not "
                    "supported — pad/truncate the references to equal length first."
                ) from exc

    # A single-sequence reference (1-D) must broadcast to one batch row,
    # not crash on ``references.shape[1]`` below with an opaque IndexError.
    references = torch.atleast_2d(references)

    batch_size = predictions.shape[0]
    if batch_size == 0:
        return {"perplexity": float("inf")}

    _batch, _seq_len, vocab_size = predictions.shape

    logits = predictions[:, :-1, :].contiguous().view(-1, vocab_size)

    # RIL ISS-192: ``LMTask.predict`` clamps inputs to the model's
    # context window (``min(max_seq_len, model.max_seq_len)``) while the
    # references are padded to the dataset's ``max_seq_len`` — for a
    # small-context model the predictions are *narrower* than the
    # references, so the naive ``references[:, 1:]`` yields more label
    # positions than logits rows and ``cross_entropy`` raises a shape
    # error mid-evaluation. Slice the labels to the prediction horizon;
    # the truncated tail was never scored anyway.
    label_width = min(references.shape[1] - 1, _seq_len - 1)
    labels = references[:, 1 : 1 + label_width].contiguous().view(-1)

    if logits.shape[0] == 0 or labels.numel() == 0:
        return {"perplexity": float("inf")}

    if self.ignore_index is not None and labels.numel() > 0 and bool((labels == self.ignore_index).all().item()):
        # Every shift-target is ignored (e.g. a 1-token corpus whose
        # shifted labels are all -100): ``cross_entropy`` with
        # ``reduction='mean'`` and ``ignore_index`` averages over ZERO
        # valid elements and returns NaN. Return the documented
        # ``inf`` (undefined perplexity) instead — NaN would serialize
        # to JSON ``null`` and poison the report (RIL ISS-055).
        return {"perplexity": float("inf")}

    kwargs = {"ignore_index": self.ignore_index} if self.ignore_index is not None else {}
    loss = functional.cross_entropy(logits, labels, reduction="mean", **kwargs)
    perplexity = torch.exp(loss).item()

    return {"perplexity": perplexity}

Evaluation Tasks

Abstract base class and concrete task implementations for offline evaluation.

base

BaseTask

Bases: ABC

Base class for all evaluation tasks.

源代码位于： src/llm/evaluation/eval_tasks/base.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class BaseTask(ABC):
    """Base class for all evaluation tasks."""

    name: str
    metrics: list[Any]

    @abstractmethod
    def prepare_data(self, split: str) -> tuple[list[str], list[str]]:
        """Prepare inputs and references for evaluation.

        Args:
            split: Data split (e.g., 'train', 'test')

        Returns:
            Tuple of (inputs, references)
        """
        pass

    @abstractmethod
    def predict(self, model: Any, inputs: list[str] | list[torch.Tensor]) -> list[str]:
        """Run model on inputs to get predictions.

        Args:
            model: Model to use for prediction
            inputs: List of input texts

        Returns:
            List of predicted outputs
        """
        pass

prepare_data abstractmethod 

prepare_data(split)

Prepare inputs and references for evaluation.

参数：

名称 类型 描述 默认
split str

Data split (e.g., 'train', 'test')

 必需

返回：

类型 描述
tuple[list[str], list[str]]

Tuple of (inputs, references)
源代码位于： src/llm/evaluation/eval_tasks/base.py 
13
14
15
16
17
18
19
20
21
22
23
@abstractmethod
def prepare_data(self, split: str) -> tuple[list[str], list[str]]:
    """Prepare inputs and references for evaluation.

    Args:
        split: Data split (e.g., 'train', 'test')

    Returns:
        Tuple of (inputs, references)
    """
    pass

predict abstractmethod 

predict(model, inputs)

Run model on inputs to get predictions.

参数：

名称 类型 描述 默认
model Any

Model to use for prediction

 必需
inputs list[str] | list[Tensor]

List of input texts

 必需

返回：

类型 描述
list[str]

List of predicted outputs
源代码位于： src/llm/evaluation/eval_tasks/base.py 
25
26
27
28
29
30
31
32
33
34
35
36
@abstractmethod
def predict(self, model: Any, inputs: list[str] | list[torch.Tensor]) -> list[str]:
    """Run model on inputs to get predictions.

    Args:
        model: Model to use for prediction
        inputs: List of input texts

    Returns:
        List of predicted outputs
    """
    pass

lm_task

LMTask

Bases: BaseTask

源代码位于： src/llm/evaluation/eval_tasks/lm_task.py 
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
class LMTask(BaseTask):
    name = "lm"

    def __init__(
        self,
        dataset_path: str,
        batch_size: int = 8,
        max_seq_len: int | None = None,
        tokenizer: BaseTokenizer | None = None,
    ):
        """Perplexity evaluation on a text corpus.

        Args:
            dataset_path: Text corpus file to evaluate.
            batch_size: Batch size for the forward pass.
            max_seq_len: Context window the sequences are truncated to. MUST
                be at least the model's ``max_seq_len`` when the caller knows
                it — the old hardcoded 128 silently crashed any model with a
                smaller context ("Sequence endpoint 128 exceeds maximum
                sequence length", RIL ISS-130). Defaults to 128 for backward
                compatibility with callers that never tuned it.
            tokenizer: The tokenizer bound to the model being evaluated. When
                provided it replaces the corpus-derived one so the vocab ids
                fed to the model actually match its trained vocabulary (RIL
                ISS-195: the old behavior always re-derived a character
                tokenizer from the eval corpus, so any model trained with a
                real tokenizer was scored with mismatched ids). Defaults to a
                corpus-derived simple tokenizer for backward compatibility.
        """
        self.dataset_path = dataset_path
        self.batch_size = batch_size
        self.max_seq_len = max_seq_len or 128
        self.tokenizer = tokenizer or TokenizerFactory.from_dataset_text(dataset_path)
        # Mask padded positions so short trailing sequences are scored
        # only over real tokens. TextDataset marks padded label slots with
        # the standard ignore index -100 (never a real token id), so the
        # metric must ignore -100 — passing the tokenizer's pad_token_id
        # instead crashes cross_entropy with "Target -100 is out of bounds"
        # and silently scores pad tokens when the pad id collides (RIL
        # ISS-041, regression from the ISS-040 label-masking fix).
        self.metrics = [PerplexityMetric(ignore_index=-100)]
        self.pad_token_id = getattr(self.tokenizer, "pad_token_id", None)

        self.val_dataset = TextDataset(
            file_path=dataset_path,
            tokenizer=cast(BaseTokenizer, self.tokenizer),
            max_seq_len=self.max_seq_len,
        )

    def prepare_data(self, split: str):
        # ``split`` is part of the BaseTask contract but a single-corpus LM
        # task has exactly one (validation) dataset, so train/val/test are
        # the same file by design. Deliberately not filtered.
        inputs = []
        references = []

        for item in self.val_dataset:
            inputs.append(item["input_ids"])
            references.append(item["labels"])

        return inputs, references

    def predict(self, model, inputs: list):
        results = []
        # Pad every batch to the GLOBAL max sequence length (not the
        # batch-local max): the final ``torch.cat(results, dim=0)`` requires
        # a uniform seq dim, and per-batch padding previously crashed on any
        # input whose lengths differ across batches (the standard flow only
        # escaped because TextDataset pre-pads to 128).
        global_max_len = max((len(x) for x in inputs), default=0)
        # Clamp to the model's own context window (RIL ISS-130). The dataset
        # is pre-truncated to ``self.max_seq_len``, but a caller that tuned
        # ``LMTask(max_seq_len=128)`` while handing it a smaller model would
        # still forward 128-token rows into a smaller positional-encoding
        # table and crash with "Sequence endpoint N exceeds maximum sequence
        # length". Never let the batch exceed what the model can attend to.
        model_capacity = getattr(model, "max_seq_len", None)
        if model_capacity is not None:
            global_max_len = min(global_max_len, int(model_capacity))
            # Truncate sequences longer than the model can attend to (rather
            # than letting them overrun the table): the tail beyond capacity
            # simply cannot be scored by this model.
            inputs = [x[:global_max_len] for x in inputs]
        pad_id = self.pad_token_id if self.pad_token_id is not None else 0

        for i in range(0, len(inputs), self.batch_size):
            batch = inputs[i : i + self.batch_size]
            lengths = [len(x) for x in batch]
            max_len = global_max_len
            padded = torch.stack(
                [
                    torch.cat(
                        [
                            torch.as_tensor(x, dtype=torch.long),
                            torch.full((max_len - len(x),), pad_id, dtype=torch.long),
                        ]
                    )
                    for x in batch
                ]
            )

            # Padding mask (True = mask out, the ``sdpa`` wrapper's
            # convention). Only built when the tokenizer has a dedicated
            # pad id and the batch actually contains padding; a literal
            # pad token in the text is indistinguishable, so masking is
            # disabled for tokenizers without a pad id.
            attn_mask = None
            if self.pad_token_id is not None and any(length < max_len for length in lengths):
                attn_mask = (padded == self.pad_token_id).unsqueeze(1).unsqueeze(2)  # [B, 1, 1, S]

            with torch.no_grad():
                logits = model(padded, attn_mask=attn_mask)
            results.append(logits)

        if not results:
            # Empty eval set (e.g. an empty corpus file): return an
            # empty 3-D prediction instead of crashing on
            # ``torch.cat([])``. The metric layer already reports ``inf``
            # for a zero-size batch (perplexity is undefined), so an
            # appropriately-shaped empty tensor keeps the whole
            # evaluation pipeline (run() + evaluate()) alive.
            return torch.empty(0, 0, 0)
        return torch.cat(results, dim=0)

Benchmark Presets

EvalPreset bundles a benchmark name with the kwargs that lm_eval.evaluator.evaluate understands. Three built-in presets ship out of the box; users can construct their own by passing the same fields.

presets

Benchmark presets for the lm-eval-harness pipeline.

A preset bundles a benchmark name with the lm_eval kwargs that evaluator.evaluate expects (num_fewshot, batch_size, limit, etc.). Three common presets ship out of the box; users can extend by constructing :class:EvalPreset directly.

The presets are intentionally decoupled from the lm_eval import so this module is safe to import on hosts that don't have lm_eval installed. Callers that actually want to run the benchmark should use :func:llm.evaluation.harness.adapter.run_preset, which is the boundary that imports lm_eval.

EvalPreset dataclass

A benchmark preset for the lm-eval-harness pipeline.

属性：

名称 类型 描述
task str

lm_eval task name (e.g. "mmlu", "arc_easy", "wikitext").
num_fewshot int | None

Number of few-shot exemplars. None lets lm_eval use the task's default.
batch_size int

Per-device evaluation batch size.
limit int | None

Optional cap on the number of samples per task (None means run the full benchmark).
task_kwargs dict[str, Any]

Extra kwargs forwarded to lm_eval.tasks (e.g. {"subject": "abstract_algebra"} for a single MMLU subject).
description str

Human-readable one-liner for the report.
源代码位于： src/llm/evaluation/harness/presets.py 
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
@dataclass(frozen=True)
class EvalPreset:
    """A benchmark preset for the lm-eval-harness pipeline.

    Attributes:
        task: lm_eval task name (e.g. ``"mmlu"``, ``"arc_easy"``,
            ``"wikitext"``).
        num_fewshot: Number of few-shot exemplars. ``None`` lets
            lm_eval use the task's default.
        batch_size: Per-device evaluation batch size.
        limit: Optional cap on the number of samples per task
            (``None`` means run the full benchmark).
        task_kwargs: Extra kwargs forwarded to ``lm_eval.tasks``
            (e.g. ``{"subject": "abstract_algebra"}`` for a single
            MMLU subject).
        description: Human-readable one-liner for the report.
    """

    task: str
    num_fewshot: int | None = None
    batch_size: int = 8
    limit: int | None = None
    task_kwargs: dict[str, Any] = field(default_factory=dict)
    description: str = ""

    def to_lm_eval_kwargs(self) -> dict[str, Any]:
        """Flatten to the kwargs ``evaluator.evaluate`` understands.

        Always returns a fresh dict so callers can mutate it without
        poisoning the frozen preset.
        """
        kwargs: dict[str, Any] = {"tasks": [self.task], "batch_size": self.batch_size}
        if self.num_fewshot is not None:
            kwargs["num_fewshot"] = self.num_fewshot
        if self.limit is not None:
            kwargs["limit"] = self.limit
        if self.task_kwargs:
            kwargs["task_kwargs"] = dict(self.task_kwargs)
        return kwargs

to_lm_eval_kwargs 

to_lm_eval_kwargs()

Flatten to the kwargs evaluator.evaluate understands.

Always returns a fresh dict so callers can mutate it without poisoning the frozen preset.

源代码位于： src/llm/evaluation/harness/presets.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
def to_lm_eval_kwargs(self) -> dict[str, Any]:
    """Flatten to the kwargs ``evaluator.evaluate`` understands.

    Always returns a fresh dict so callers can mutate it without
    poisoning the frozen preset.
    """
    kwargs: dict[str, Any] = {"tasks": [self.task], "batch_size": self.batch_size}
    if self.num_fewshot is not None:
        kwargs["num_fewshot"] = self.num_fewshot
    if self.limit is not None:
        kwargs["limit"] = self.limit
    if self.task_kwargs:
        kwargs["task_kwargs"] = dict(self.task_kwargs)
    return kwargs

get_preset 

get_preset(name)

Look up a built-in preset by name.

引发：

类型 描述
KeyError

if name is not one of the built-in presets.
源代码位于： src/llm/evaluation/harness/presets.py 
 96
 97
 98
 99
100
101
102
103
104
105
def get_preset(name: str) -> EvalPreset:
    """Look up a built-in preset by name.

    Raises:
        KeyError: if ``name`` is not one of the built-in presets.
    """
    if name not in BUILTIN_PRESETS:
        available = ", ".join(sorted(BUILTIN_PRESETS))
        raise KeyError(f"unknown preset {name!r}; available built-ins: {available}")
    return BUILTIN_PRESETS[name]

LlamaLmEvalLM — DecoderModel adapter for lm_eval

Minimal lm_eval.api.model.LM implementation that wraps a DecoderModel + tokenizer. Implements the three protocol methods (loglikelihood, loglikelihood_rolling, generate_until) without pulling in HFLM's HF-only kwargs (prefix_token, backend).

lm_eval_lm

lm_eval LM adapter for our :class:DecoderModel.

lm-evaluation-harness expects model wrappers to implement the lm_eval.api.model.LM protocol (loglikelihood, loglikelihood_rolling, generate_until). This module provides a minimal adapter that conforms to that interface so any trained :class:DecoderModel + tokenizer can be evaluated with lm_eval.evaluator.evaluate(lm=LlamaLmEvalLM(model, tokenizer)).

Soft dependency on lm_eval — this module imports lazily inside __init__ so importing :mod:llm.evaluation.harness.lm_eval_lm never raises on hosts without lm_eval installed. The ImportError fires at __init__ time with the install hint.

Why a dedicated wrapper rather than reusing :class:HFLM? HFLM's surface requires a torch.device and several HF-only arguments (prefix_token, backend) that don't apply here. A 100-line minimal adapter keeps the dependency tree honest and the contract obvious.

LlamaLmEvalLM

Minimal :class:lm_eval.api.model.LM adapter for DecoderModel.

Implements just enough of the lm_eval protocol to run the standard multiple-choice (loglikelihood) and generation (generate_until) tasks. Each request is processed individually with a single forward pass; batch_size controls how many requests are handled between no_grad context rebuilds and Python-side bookkeeping, not the model's batch dimension.

参数：

名称 类型 描述 默认
model Any

A trained :class:llm.models.DecoderModel.

 必需
tokenizer Any

Tokenizer with encode, decode, bos_token_id (optional), eos_token_id. Must produce token ids accepted by the model's vocabulary.

 必需
batch_size int

Maximum number of requests per forward pass.

 8
max_length int | None

Hard cap on sequence length (model's max_seq_len is the default).

 None
device str | device | None

Target device; defaults to the model's parameter device.

 None
源代码位于： src/llm/evaluation/harness/lm_eval_lm.py 
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
class LlamaLmEvalLM:
    """Minimal :class:`lm_eval.api.model.LM` adapter for ``DecoderModel``.

    Implements just enough of the lm_eval protocol to run the
    standard multiple-choice (loglikelihood) and generation
    (generate_until) tasks. Each request is processed individually
    with a single forward pass; ``batch_size`` controls how many
    requests are handled between ``no_grad`` context rebuilds and
    Python-side bookkeeping, not the model's batch dimension.

    Args:
        model: A trained :class:`llm.models.DecoderModel`.
        tokenizer: Tokenizer with ``encode``, ``decode``, ``bos_token_id``
            (optional), ``eos_token_id``. Must produce token ids
            accepted by the model's vocabulary.
        batch_size: Maximum number of requests per forward pass.
        max_length: Hard cap on sequence length (model's
            ``max_seq_len`` is the default).
        device: Target device; defaults to the model's parameter device.
    """

    def __init__(
        self,
        model: Any,
        tokenizer: Any,
        *,
        batch_size: int = 8,
        max_length: int | None = None,
        device: str | torch.device | None = None,
    ) -> None:
        _require_lm_eval()
        # We import lazily so this module is import-safe without lm_eval.
        import_module("lm_eval.api.model")  # validates install

        self.model = model
        self.tokenizer = tokenizer
        self.batch_size = batch_size
        self.max_length = max_length or getattr(model, "max_seq_len", 2048)
        if device is None:
            device = next(model.parameters()).device
        self.device = torch.device(device) if not isinstance(device, torch.device) else device
        self.model.eval()

    # --- lm_eval protocol surface ------------------------------------------

    def loglikelihood(self, requests):
        """Compute (log_likelihood, is_greedy_match) for each request.

        Each request is an ``lm_eval.api.request.Request`` whose
        ``args`` is ``(context_str, continuation_str)``. Returns a
        list of ``(sum_logprob, is_greedy)`` tuples.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._loglikelihood_batch(batch)
                results.extend(batch_results)
        return results

    def loglikelihood_rolling(self, requests):
        """Compute total log-probability of each string (perplexity-style).

        Returns ``list[float]`` — one scalar sum-log-prob per request,
        matching the lm_eval ``LM.loglikelihood_rolling`` protocol.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._loglikelihood_rolling_batch(batch)
                results.extend(batch_results)
        return results

    def generate_until(self, requests):
        """Greedy generation until ``until`` token sequences appear.

        Each request is an ``lm_eval.api.request.Request`` whose
        ``args`` is ``(context_str, {"until": [...], "max_gen_toks": N})``.
        """
        results = []
        with torch.no_grad():
            for batch_start in range(0, len(requests), self.batch_size):
                batch = requests[batch_start : batch_start + self.batch_size]
                batch_results = self._generate_until_batch(batch)
                results.extend(batch_results)
        return results

    # --- batched implementations -------------------------------------------

    def _loglikelihood_batch(self, batch):
        """Tokenize + forward + per-token log-prob extraction for a batch.

        For each (context, continuation):
        1. Concatenate ``context + continuation`` token ids.
        2. Pad to the longest sequence in the batch.
        3. Forward; collect log-probs at each continuation position.
        4. Sum and compare to greedy argmax for ``is_greedy_match``.
        """
        results = []
        # Tokenize once, defer padding to after we know the longest length.
        encoded: list[tuple[list[int], list[int]]] = []
        for req in batch:
            context, continuation = req.args
            ctx_ids = self._encode(context)
            cont_ids = self._encode(continuation)
            if not cont_ids:
                # lm_eval generally guarantees a non-empty continuation,
                # but guard against pathological inputs.
                cont_ids = [0]
            encoded.append((ctx_ids, cont_ids))

        max_len = min(
            self.max_length,
            max(len(c) + len(k) for c, k in encoded),
        )

        for ctx_ids, cont_ids in encoded:
            full = (ctx_ids + cont_ids)[-max_len:]
            # ``full`` keeps the LAST ``max_len`` tokens, so when a request
            # exceeds the window we may drop context, continuation, or both.
            # ``ctx_len`` = how many of ``full``'s tokens are context; the
            # continuation tokens actually present in the window are
            # ``full[ctx_len:]`` (NOT the full ``cont_ids`` — those can
            # extend past the window, RIL ISS-050).
            ctx_len = max(0, len(full) - len(cont_ids))

            # Tokens are scored shift-by-one: the token at window position
            # ``p`` is predicted by logit row ``p - 1``, which exists only
            # for ``p >= 1``. The first scorable token is at position
            # ``max(ctx_len, 1)`` of ``full``:
            #   - ``ctx_len >= 1``: all continuation tokens are scorable,
            #     their predictors are rows ``ctx_len - 1 ... len(full) - 2``;
            #   - ``ctx_len == 0`` (continuation alone fills/exceeds the
            #     window): ``full[0]`` is the first continuation token whose
            #     predictor (at ``full[-1]``) was truncated away, so the
            #     first continuation token is unscorable and we start at
            #     ``full[1]`` (predictors rows ``0 ... len(full) - 2``).
            first_scorable = max(ctx_len, 1)
            scorable = full[first_scorable:]
            if not scorable:
                # No scorable continuation token in the window (a single-token
                # window with no predicting logit).
                results.append((0.0, False))
                continue

            start_row = first_scorable - 1
            input_tensor = torch.tensor([full], dtype=torch.long, device=self.device)
            model_out = self.model(input_tensor, use_cache=False)
            logits = model_out[0] if isinstance(model_out, tuple) else model_out
            relevant = logits[0, start_row : start_row + len(scorable), :]
            log_probs = torch.log_softmax(relevant, dim=-1)
            # Row ``start_row + i`` predicts ``full[first_scorable + i]``.
            cont_tensor = torch.tensor(scorable, device=self.device, dtype=torch.long)
            row_index = torch.arange(len(scorable), device=self.device)
            target_log_probs = log_probs[row_index, cont_tensor]
            sum_logprob = float(target_log_probs.sum().item())

            # Greedy match: argmax at each continuation position
            # equals the continuation token.
            greedy_tokens = relevant.argmax(dim=-1)
            is_greedy = bool(torch.equal(greedy_tokens, cont_tensor))

            results.append((sum_logprob, is_greedy))
        return results

    def _loglikelihood_rolling_batch(self, batch):
        """Sum log-probs across every token of each request's string.

        Returns one ``float`` per request (NOT a tuple) so the result
        is compatible with the lm_eval ``loglikelihood_rolling``
        protocol: it appends each element to ``req.resps`` and
        downstream code (e.g. WikiText perplexity) does
        ``(loglikelihood,) = results`` — unpacking a 1-tuple here
        would corrupt the metric tuples downstream.
        """
        results = []
        for req in batch:
            (text,) = req.args
            ids = self._encode(text)
            if len(ids) < 2:
                results.append(0.0)
                continue

            if len(ids) <= self.max_length:
                # Single pass over the whole (short) doc.
                input_tensor = torch.tensor([ids], dtype=torch.long, device=self.device)
                model_out = self.model(input_tensor, use_cache=False)
                logits = model_out[0] if isinstance(model_out, tuple) else model_out
                # Log-probs at positions 0..len-1 predict ids 1..len.
                relevant = logits[0, :-1, :]
                log_probs = torch.log_softmax(relevant, dim=-1)
                targets = torch.tensor(ids[1:], device=self.device, dtype=torch.long)
                token_log_probs = log_probs[torch.arange(len(targets), device=self.device), targets]
                results.append(float(token_log_probs.sum().item()))
                continue

            # Long doc: score it with STRIDED windows so every token — not just
            # the first ``max_length`` — contributes. lm_eval's downstream
            # word-perplexity divides the returned sum by the FULL doc word
            # count; truncating the numerator here silently deflates the metric
            # on any document longer than ``max_seq_len`` (RIL ISS-128).
            #
            # Each window carries ``max_length`` tokens through the model and
            # sums the log-probs of the *new* tokens (those after the previous
            # window's overlap), so no token is double-counted and the whole
            # doc is covered. This mirrors lm_eval's canonical HFLM rolling
            # perplexity (strided context windows).
            total = 0.0
            start = 0
            while start < len(ids) - 1:
                end = min(start + self.max_length, len(ids))
                window = ids[start:end]
                input_tensor = torch.tensor([window], dtype=torch.long, device=self.device)
                model_out = self.model(input_tensor, use_cache=False)
                logits = model_out[0] if isinstance(model_out, tuple) else model_out
                # Score only the *new* tokens of this window. The first window
                # scores ids[1:end]; each later window's window[0] is the
                # overlap context token already scored as the previous
                # window's last target.
                target_ids = ids[start + 1 : end]
                if len(target_ids) == 0:
                    break
                log_probs = torch.log_softmax(logits[0, :, :], dim=-1)
                # Row ``j`` predicts ``window[j+1] = ids[start+j+1]``, which is
                # exactly ``target_ids[j]`` — so the mask is j (0-based), NOT
                # j+1 (that would predict one token too far and short-change
                # the first target).
                position_mask = torch.arange(len(target_ids), device=self.device)
                token_log_probs = log_probs[position_mask, torch.tensor(target_ids, device=self.device)]
                total += float(token_log_probs.sum().item())
                # Only stop once the LAST token (``ids[-1]``) has been a
                # target. Breaking at ``end >= len(ids) - 1`` terminates one
                # window early whenever a boundary lands exactly on
                # ``len(ids) - 1`` (doc length ≡ 2 mod (max_length-1)), so
                # ``ids[-1]`` is never scored while the downstream metric
                # divides by the full doc word count — a silently deflated
                # perplexity (RIL ISS-193).
                if end >= len(ids):
                    break
                start = end - 1  # overlap one context token for the next window
            results.append(total)
        return results

    def _generate_until_batch(self, batch):
        """Greedy ``generate_until`` for a batch of requests."""
        results = []
        for req in batch:
            context, gen_kwargs = req.args
            until = gen_kwargs.get("until", [])
            max_gen_toks = int(gen_kwargs.get("max_gen_toks", 64))

            # lm_eval's ``handle_stop_sequences`` (models/utils.py) appends
            # the tokenizer's EOS text to ``until`` before generation; the
            # upstream LMs then strip stops with ``postprocess_generated_text``
            # before returning. Do the same here, otherwise an EOS-emitting
            # model runs to ``max_gen_toks`` and every generation answer keeps
            # its trailing delimiter (systematically wrong exact_match/acc).
            #
            # ``until`` may be a scalar string or None (a task YAML like
            # ``until: "END"``). A bare ``list(until)`` would split a
            # multi-char scalar into per-char stops, so ``_strip_stop_strings``
            # trims at the FIRST occurrence of any single char — massively
            # over-trimming completions (RIL ISS-132). Normalize exactly like
            # lm_eval's handle_stop_sequences (str -> [str], None -> []).
            if isinstance(until, str):
                until = [until]
            elif until is None:
                until = []
            stops = list(until)
            eos_id = getattr(self.tokenizer, "eos_token_id", None)
            if eos_id is not None:
                eos_str = self.tokenizer.decode([eos_id])
                if eos_str and eos_str not in stops:
                    stops.append(eos_str)
                # Register the EOS as a token-id-sequence stop too. HF
                # tokenizers decode special tokens to "" by default
                # (``skip_special_tokens=True``), so a decoded-string stop
                # never fires and ``generate_until`` over-ran to
                # ``max_gen_toks`` past the EOS — the ISS-049 regression the
                # string guard intended to solve (RIL ISS-226 / round-73
                # FINDING 6). The id sequence is token-precise and
                # decode-independent.
                if [eos_id] not in stops:
                    stops.append([eos_id])

            ctx_ids = self._encode(context)
            generated: list[int] = []
            for _ in range(max_gen_toks):
                full = (ctx_ids + generated)[-self.max_length :]
                input_tensor = torch.tensor([full], dtype=torch.long, device=self.device)
                model_out = self.model(input_tensor, use_cache=False)
                logits = model_out[0] if isinstance(model_out, tuple) else model_out
                next_token = int(logits[0, -1, :].argmax(dim=-1).item())
                generated.append(next_token)
                # Stop if the suffix matches any ``until`` token sequence
                # (id lists) or the decoded text ends with any string stop.
                if self._matches_any_stop(generated, stops):
                    generated = self._truncate_id_stop_suffix(generated, stops)
                    break

            results.append(self._strip_stop_strings(self.tokenizer.decode(generated), stops))
        return results

    @staticmethod
    def _strip_stop_strings(text: str, stops: list) -> str:
        """Cut ``text`` at the earliest occurrence of any string stop.

        Mirrors lm_eval's ``postprocess_generated_text``: a generation
        answer must not contain the stop delimiter (upstream LMs strip it
        before returning). Token-id ``until`` entries are matched pre-decode
        (as suffixes) and never appear as text, so only strings are
        post-processed here.

        RIL ISS-194: take the MINIMUM index across all string stops, not the
        first stop in ``stops`` list order — list order is not occurrence
        order. E.g. ``stops=["END", "ANSWER"]`` on text ``"foo ANSWER bar
        END"`` must cut at ``"ANSWER"`` (index 4), not at ``"END"`` (index
        14), or the returned completion still contains a delimiter.
        """
        if not stops:
            return text
        indices = [text.find(s) for s in stops if isinstance(s, str) and s]
        occurrences = [i for i in indices if i != -1]
        if not occurrences:
            return text
        return text[: min(occurrences)]

    def _matches_any_stop(self, generated: list[int], until: list) -> bool:
        """Return True when ``generated`` should stop w.r.t. ``until``.

        Handles both stop forms lm_eval emits:

        - token-id sequences (``list[int]``) matched as a suffix of
          ``generated`` by :meth:`_matches_any_suffix`;
        - **strings** — the standard lm_eval form for every generation
          task (``truthfulqa``, ``humaneval``, ...). String stops are
          matched against the *decoded* generated text (a tokenizer is
          available here), so early stopping actually fires instead of
          generating the full ``max_gen_toks`` with the delimiter
          embedded (RIL ISS-049).
        """
        if self._matches_any_suffix(generated, until):
            return True
        decoded = self.tokenizer.decode(generated)
        return any(isinstance(stop, str) and stop and decoded.endswith(stop) for stop in until)

    # --- helpers ------------------------------------------------------------

    def _encode(self, text: str) -> list[int]:
        """Encode text using the bound tokenizer, with BOS handling."""
        ids = self.tokenizer.encode(text)
        if not isinstance(ids, list):
            ids = ids.tolist() if hasattr(ids, "tolist") else list(ids)
        return list(ids)

    @staticmethod
    def _truncate_id_stop_suffix(generated: list[int], until: list) -> list[int]:
        """Drop the matched token-id-sequence stop suffix before decoding.

        ``_matches_any_stop`` halts generation the moment an ``until``
        id-sequence becomes a *suffix* of ``generated``, but the stop ids are
        still in ``generated`` — decoding the whole run embeds the stop's
        text in the returned completion (RIL ISS-106). String stops have the
        post-decode :meth:`_strip_stop_strings` cut; id-sequence stops need
        an exact token-boundary cut here (lossless, unlike a
        ``decode(stop_ids)`` text round-trip). String entries are skipped —
        they are stripped post-decode.
        """
        for stop in until:
            if isinstance(stop, str):
                continue
            stop_ids = list(stop) if not isinstance(stop, list) else stop
            if stop_ids and len(generated) >= len(stop_ids) and generated[-len(stop_ids) :] == stop_ids:
                return generated[: -len(stop_ids)]
        return generated

    @staticmethod
    def _matches_any_suffix(generated: list[int], until: list) -> bool:
        """Return True if any ``until`` *id-sequence* is a suffix of ``generated``.

        Only token-id-sequence entries (``list[int]``) are matched here.
        String entries are handled by :meth:`_matches_any_stop` (which has
        the tokenizer for a text round-trip); this static helper can't
        match by text so it skips strings (they are not failures — a later
        branch handles them).
        """
        if not until:
            return False
        for stop in until:
            if isinstance(stop, str):
                # String stops are matched against decoded text in
                # ``_matches_any_stop`` (we have the tokenizer there).
                continue
            stop_ids = list(stop) if not isinstance(stop, list) else stop
            if not stop_ids:
                continue
            if len(generated) >= len(stop_ids) and generated[-len(stop_ids) :] == stop_ids:
                return True
        return False

loglikelihood 

loglikelihood(requests)

Compute (log_likelihood, is_greedy_match) for each request.

Each request is an lm_eval.api.request.Request whose args is (context_str, continuation_str). Returns a list of (sum_logprob, is_greedy) tuples.

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def loglikelihood(self, requests):
    """Compute (log_likelihood, is_greedy_match) for each request.

    Each request is an ``lm_eval.api.request.Request`` whose
    ``args`` is ``(context_str, continuation_str)``. Returns a
    list of ``(sum_logprob, is_greedy)`` tuples.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._loglikelihood_batch(batch)
            results.extend(batch_results)
    return results

loglikelihood_rolling 

loglikelihood_rolling(requests)

Compute total log-probability of each string (perplexity-style).

Returns list[float] — one scalar sum-log-prob per request, matching the lm_eval LM.loglikelihood_rolling protocol.

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py 
103
104
105
106
107
108
109
110
111
112
113
114
115
def loglikelihood_rolling(self, requests):
    """Compute total log-probability of each string (perplexity-style).

    Returns ``list[float]`` — one scalar sum-log-prob per request,
    matching the lm_eval ``LM.loglikelihood_rolling`` protocol.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._loglikelihood_rolling_batch(batch)
            results.extend(batch_results)
    return results

generate_until 

generate_until(requests)

Greedy generation until until token sequences appear.

Each request is an lm_eval.api.request.Request whose args is (context_str, {"until": [...], "max_gen_toks": N}).

源代码位于： src/llm/evaluation/harness/lm_eval_lm.py 
117
118
119
120
121
122
123
124
125
126
127
128
129
def generate_until(self, requests):
    """Greedy generation until ``until`` token sequences appear.

    Each request is an ``lm_eval.api.request.Request`` whose
    ``args`` is ``(context_str, {"until": [...], "max_gen_toks": N})``.
    """
    results = []
    with torch.no_grad():
        for batch_start in range(0, len(requests), self.batch_size):
            batch = requests[batch_start : batch_start + self.batch_size]
            batch_results = self._generate_until_batch(batch)
            results.extend(batch_results)
    return results

LmEvalAdapter — top-level driver

Preset lookup, kwarg merging, and structured result flattening on top of lm_eval.evaluator.

adapter

Adapter for lm-evaluation-harness.

Two responsibilities:

  1. LmEvalAdapter — thin wrapper around lm_eval.evaluator that adds structured result handling and preset support.
  2. run_preset — convenience that ties a preset, an LM, and the evaluator together.

The :mod:llm.evaluation.harness.presets module is safe to import without lm_eval installed; the lm_eval import boundary lives here so the project's existing [eval] optional-dependency group keeps working.

LmEvalAdapter

Adapter for lm-evaluation-harness.

Adds:

  • Preset lookup — run_preset(preset_name, lm) resolves a :class:EvalPreset by name (built-in or user-supplied) and runs the benchmark.
  • Structured result flattening — :meth:summarize extracts acc / acc_norm / perplexity / f1 / etc. from the nested lm_eval result shape into a flat {task_name: {metric: value}} dict.
源代码位于： src/llm/evaluation/harness/adapter.py 
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
class LmEvalAdapter:
    """Adapter for lm-evaluation-harness.

    Adds:

    * **Preset lookup** — ``run_preset(preset_name, lm)`` resolves a
      :class:`EvalPreset` by name (built-in or user-supplied) and
      runs the benchmark.
    * **Structured result flattening** — :meth:`summarize` extracts
      ``acc`` / ``acc_norm`` / ``perplexity`` / ``f1`` / etc. from
      the nested lm_eval result shape into a flat
      ``{task_name: {metric: value}}`` dict.
    """

    def __init__(self) -> None:
        _require_lm_eval()
        self._task_manager = import_module("lm_eval.tasks").TaskManager()

    def list_tasks(self) -> list[str]:
        """List available benchmark tasks."""
        return sorted(self._task_manager.all_tasks)

    def evaluate(self, model: Any, tasks: list[str] | None = None, **kwargs: Any) -> dict:
        """Run evaluation on specified tasks.

        Mirrors the lm_eval ``evaluator.evaluate`` signature; see
        ``lm_eval`` docs for the kwargs surface.
        """
        _require_lm_eval()
        # Read the attribute off the parent package so callers can patch
        # ``lm_eval.evaluator`` (the mock contract used by the tests),
        # falling back to importing the real submodule on first use.
        lm_eval = import_module("lm_eval")
        evaluator = lm_eval.__dict__.get("evaluator") or import_module("lm_eval.evaluator")
        return evaluator.evaluate(model=model, tasks=tasks or ["mmlu"], **kwargs)

    def run_preset(
        self,
        preset: EvalPreset | str,
        model: Any,
        **kwargs: Any,
    ) -> dict:
        """Run a benchmark by preset (name or :class:`EvalPreset`).

        Merges the preset's :meth:`EvalPreset.to_lm_eval_kwargs` with
        any caller-supplied ``kwargs`` (caller wins on conflicts).
        """
        if isinstance(preset, str):
            preset = get_preset(preset)
        merged = preset.to_lm_eval_kwargs()
        merged.update(kwargs)
        return self.evaluate(model, **merged)

    def run_benchmark(self, model: Any, benchmark: str, **kwargs: Any) -> dict:
        """Run a single benchmark task by name (no preset lookup)."""
        return self.evaluate(model, tasks=[benchmark], **kwargs)

    @staticmethod
    def summarize(results: dict[str, Any]) -> dict[str, dict[str, float]]:
        """Flatten lm_eval's nested result tree into a flat metric map.

        lm_eval's ``evaluator.simple_evaluate`` returns:

        .. code-block:: python

            {
                "results": {
                    "task_name": {
                        "acc,none": 0.42,
                        "acc_norm,none": 0.45,
                        ...
                    },
                    ...
                },
                "groups": {...},
                "configs": {...},
            }

        This helper extracts the ``results`` block and splits each
        comma-separated key (``"acc,none"`` -> metric ``"acc"``,
        subset ``"none"``) so callers can serialize it as a flat
        dict.

        Notes:

        - Only the ``results`` block is flattened; ``groups`` and
          ``configs`` are intentionally ignored.
        - Booleans are dropped (Python's ``bool`` is a subclass of
          ``int``, but a metric of value ``True`` is almost certainly
          a bug — string aliases are usually what you'd see).
        - Non-numeric values (e.g. string aliases) are silently
          skipped, keeping the output strictly numeric.
        """
        flat: dict[str, dict[str, float]] = {}
        for task_name, metrics in results.get("results", {}).items():
            flat[task_name] = {}
            for key, value in metrics.items():
                metric_name = key.split(",", 1)[0] if "," in key else key
                if isinstance(value, bool):
                    # bool is technically int — skip to avoid ``True`` -> 1.0.
                    continue
                if isinstance(value, (int, float)):
                    flat[task_name][metric_name] = float(value)
        return flat

list_tasks 

list_tasks()

List available benchmark tasks.

源代码位于： src/llm/evaluation/harness/adapter.py 
57
58
59
def list_tasks(self) -> list[str]:
    """List available benchmark tasks."""
    return sorted(self._task_manager.all_tasks)

evaluate 

evaluate(model, tasks=None, **kwargs)

Run evaluation on specified tasks.

Mirrors the lm_eval evaluator.evaluate signature; see lm_eval docs for the kwargs surface.

源代码位于： src/llm/evaluation/harness/adapter.py 
61
62
63
64
65
66
67
68
69
70
71
72
73
def evaluate(self, model: Any, tasks: list[str] | None = None, **kwargs: Any) -> dict:
    """Run evaluation on specified tasks.

    Mirrors the lm_eval ``evaluator.evaluate`` signature; see
    ``lm_eval`` docs for the kwargs surface.
    """
    _require_lm_eval()
    # Read the attribute off the parent package so callers can patch
    # ``lm_eval.evaluator`` (the mock contract used by the tests),
    # falling back to importing the real submodule on first use.
    lm_eval = import_module("lm_eval")
    evaluator = lm_eval.__dict__.get("evaluator") or import_module("lm_eval.evaluator")
    return evaluator.evaluate(model=model, tasks=tasks or ["mmlu"], **kwargs)

run_preset 

run_preset(preset, model, **kwargs)

Run a benchmark by preset (name or :class:EvalPreset).

Merges the preset's :meth:EvalPreset.to_lm_eval_kwargs with any caller-supplied kwargs (caller wins on conflicts).

源代码位于： src/llm/evaluation/harness/adapter.py 
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
def run_preset(
    self,
    preset: EvalPreset | str,
    model: Any,
    **kwargs: Any,
) -> dict:
    """Run a benchmark by preset (name or :class:`EvalPreset`).

    Merges the preset's :meth:`EvalPreset.to_lm_eval_kwargs` with
    any caller-supplied ``kwargs`` (caller wins on conflicts).
    """
    if isinstance(preset, str):
        preset = get_preset(preset)
    merged = preset.to_lm_eval_kwargs()
    merged.update(kwargs)
    return self.evaluate(model, **merged)

run_benchmark 

run_benchmark(model, benchmark, **kwargs)

Run a single benchmark task by name (no preset lookup).

源代码位于： src/llm/evaluation/harness/adapter.py 
92
93
94
def run_benchmark(self, model: Any, benchmark: str, **kwargs: Any) -> dict:
    """Run a single benchmark task by name (no preset lookup)."""
    return self.evaluate(model, tasks=[benchmark], **kwargs)

summarize staticmethod 

summarize(results)

Flatten lm_eval's nested result tree into a flat metric map.

lm_eval's evaluator.simple_evaluate returns:

.. code-block:: python

{
    "results": {
        "task_name": {
            "acc,none": 0.42,
            "acc_norm,none": 0.45,
            ...
        },
        ...
    },
    "groups": {...},
    "configs": {...},
}

This helper extracts the results block and splits each comma-separated key ("acc,none" -> metric "acc", subset "none") so callers can serialize it as a flat dict.

Notes:

  • Only the results block is flattened; groups and configs are intentionally ignored.
  • Booleans are dropped (Python's bool is a subclass of int, but a metric of value True is almost certainly a bug — string aliases are usually what you'd see).
  • Non-numeric values (e.g. string aliases) are silently skipped, keeping the output strictly numeric.
源代码位于： src/llm/evaluation/harness/adapter.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
@staticmethod
def summarize(results: dict[str, Any]) -> dict[str, dict[str, float]]:
    """Flatten lm_eval's nested result tree into a flat metric map.

    lm_eval's ``evaluator.simple_evaluate`` returns:

    .. code-block:: python

        {
            "results": {
                "task_name": {
                    "acc,none": 0.42,
                    "acc_norm,none": 0.45,
                    ...
                },
                ...
            },
            "groups": {...},
            "configs": {...},
        }

    This helper extracts the ``results`` block and splits each
    comma-separated key (``"acc,none"`` -> metric ``"acc"``,
    subset ``"none"``) so callers can serialize it as a flat
    dict.

    Notes:

    - Only the ``results`` block is flattened; ``groups`` and
      ``configs`` are intentionally ignored.
    - Booleans are dropped (Python's ``bool`` is a subclass of
      ``int``, but a metric of value ``True`` is almost certainly
      a bug — string aliases are usually what you'd see).
    - Non-numeric values (e.g. string aliases) are silently
      skipped, keeping the output strictly numeric.
    """
    flat: dict[str, dict[str, float]] = {}
    for task_name, metrics in results.get("results", {}).items():
        flat[task_name] = {}
        for key, value in metrics.items():
            metric_name = key.split(",", 1)[0] if "," in key else key
            if isinstance(value, bool):
                # bool is technically int — skip to avoid ``True`` -> 1.0.
                continue
            if isinstance(value, (int, float)):
                flat[task_name][metric_name] = float(value)
    return flat

run_preset 

run_preset(preset, model, **kwargs)

Convenience entry point: build an adapter and run a preset.

源代码位于： src/llm/evaluation/harness/adapter.py 
145
146
147
148
149
150
151
def run_preset(
    preset: EvalPreset | str,
    model: Any,
    **kwargs: Any,
) -> dict:
    """Convenience entry point: build an adapter and run a preset."""
    return LmEvalAdapter().run_preset(preset, model, **kwargs)

End-to-end usage

See the Evaluation guide for a worked example (preset selection, result flattening, soft-dependency contract).