The Two MMLU Scores: What a Benchmark Name Does Not Fix

Date: September 6, 2026 · Author: Dmitrii Zatona 

TL;DR

• Two MMLU accuracies, 0.781 and 0.79, for two builds of one model family under the same benchmark name; for a score-delta query the verifier returns incomparable (Sections 1 and 5).
• mmlu fixes a name. The split, the implementation, the prompt format, the grader and the runner’s network access stay open, and where published measurements exist for them the differences are points of accuracy, not thousandths (Section 2).
• Comparability is a property of the reference the results are traceable to, not of the number (Section 3).
• Under the APL AI-Eval profile the frame is a content-addressed object and the claim carries its hash; the two frames differ, and subset: "all" and an omitted key are different scopes by canonical bytes (Section 4).
• apl-valid is a statement about structure and says nothing about whether either score is correct (Section 6).

Two evaluation records appear in the same table. One reports mmlu accuracy 0.781 for build 42; the other reports 0.79 for build 44. The claims declare the same provider, model family, metric identifier, unit and benchmark name. The arithmetic difference is +0.009.

The records are structurally valid. Their frames declare different runners, graders and dataset splits. The shared mmlu label identifies a dataset family, not a full measurement procedure. For the score-delta query shown below, the verifier returns incomparable.

What follows is that pair taken apart: what stays open once mmlu is fixed and what published numbers say each open variable is worth; why comparability attaches to the reference rather than to the number; and what the verifier in the apl-ai-eval crate outputs once the conditions are a hashed object, with a bridge and without one. Every frame and claim below is copied from the crate’s test vectors; the verifier outputs were recorded by running the crate’s public verification functions against those vectors.

1. What the two numbers say

Here is claim A as it exists on the wire, in the metadata.apl position of a log entry:

{"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-42","build_id":"42","artifact_digest":"sha256:4242424242424242424242424242424242424242424242424242424242424242","provider":"acme","model_family":"acme-gpt-7b"},"aspect_refs":["accuracy"],"statement":{"predicate":"score","content":{"benchmark_id":"mmlu","metric_id":"accuracy","value":0.781,"unit":"fraction"}}},"frame_ref":{"hash":"sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab"}}}

Claim B, two builds later:

{"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-44","build_id":"44","artifact_digest":"sha256:4444444444444444444444444444444444444444444444444444444444444444","provider":"acme","model_family":"acme-gpt-7b"},"aspect_refs":["accuracy"],"statement":{"predicate":"score","content":{"benchmark_id":"mmlu","metric_id":"accuracy","value":0.79,"unit":"fraction"}}},"frame_ref":{"hash":"sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e"}}}

The subject differs, which is the point: two builds of one family. Each claim declares an artifact_digest, which the profile treats as the immutable identity anchor of the evaluated artifact. The statement is identical in shape and vocabulary — predicate score, benchmark mmlu, metric accuracy, unit fraction. Claim B writes the value as 0.79; a presentation may display that JSON value as 0.790, and under RFC 8785 canonical number serialization the trailing zero does not change the value.

The claim-level pointers differ: frame_ref.hash is c7b88426… in one record and c93a9c55… in the other. Resolving the two frames shows differences in both procedure and scope; the hash is the whole signal at the claim level.

The relation someone wants over this pair is also an object:

{"left_aspects":["accuracy"],"right_aspects":["accuracy"],"predicate":"score","relation_type":"score-delta"}

score-delta is a request to subtract. What has to hold for that request to have an answer is Sections 2 and 3; what the verifier returns when it does not is Section 5.

2. What “MMLU” does not fix

benchmark_id: mmlu fixes a name. Five things it leaves open, and what the record says each is worth.

2.1 The split

The MMLU paper reports 15,908 questions split into a few-shot development set of 5 questions per subject across 57 subjects, a validation set of 1,540 and a test set of 14,079 (Hendrycks et al., arXiv:2009.03300 , §3). The Hugging Face dataset most runners load, cais/mmlu config all, reports test 14,042, validation 1,531, dev 285 (cais/mmlu ). The archive linked from the hendrycks/test  README was not retrievable when checked in September 2026 (HTTP 403 after redirect). “The MMLU test set” names two objects of different sizes, and no reviewed document explains the difference.

Frame A scores on dev, the 285 questions the paper defines as the source of its fixed few-shot examples (§4.1), so scoring on it is a choice, and the kind of choice a frame should make visible. Frame B scores on test-lite, the label used in the crate’s own test vectors for a reduced set; a check of the Hugging Face datasets and models APIs and of GitHub repository search found no published artifact under that name, and it is not tinyMMLU . The label tells a reader a private slice was used. It does not tell them which questions.

2.2 The implementation

One published measurement of implementation variance is the June 2023 Hugging Face post on the Open LLM Leaderboard. Three harnesses — HELM, the Eleuther harness, the original code — run the same dataset, all 5-shot, and score llama-65b at 0.637, 0.488 and 0.636; falcon-40b at 0.571, 0.527 and 0.558. The post concludes that the three results are not comparable despite the shared MMLU label (What’s going on with the Open LLM Leaderboard? ). The mechanism is scoring. The original code compares the probabilities of the four answer letters; HELM generates from the next-token output and compares to expected text; the harness scores the full answer sequence including the option text. Rank order moves with it — falcon-40b sits above llama-65b under the harness and below it under the other two.

The runner is not one object either. The lm-evaluation-harness MMLU README describes mmlu, mmlu_continuation (cloze-style) and mmlu_generative (the model produces the answer letter) as three tasks over the same data, at different task versions (lm-evaluation-harness , commit b954108c). Its task guide treats the YAML config plus the codebase commit hash as the unit another researcher needs to replicate a setup; num_fewshot defaults to 0, and the MMLU YAML sets none — so “5-shot MMLU” is a command-line flag, not a property of the task. Version 0.3.0 asked users to report each task’s version; current main carries no such request.

2.3 The prompt format

Anthropic’s 2023 account of evaluation reports that formatting alone — option labels, parentheses, an extra space before the answer — moves MMLU accuracy by about 5% (Challenges in evaluating AI systems ). Answer position moves more. Zheng et al. report that on MMLU, moving the correct answers to position D lowers gpt-3.5-turbo from 67.2 to 60.9, and that moving them to A lifts llama-30b by 15.2 points to 68.2 against gpt-3.5-turbo’s 65.3, reversing the original 53.1 against 67.2 (arXiv:2309.03882 ).

Both frames here declare prompt_protocol: zero-shot-mcq-v1, so this variable is held. A held variable is only visibly held if it is written down.

2.4 The grader

Frame A grades with exact-match-v1. Frame B grades with llm-judge-v3. Those are not two implementations of one function. Zheng et al. found that judge models “exhibit strong position bias”, that only GPT-4 stayed consistent in more than 60% of cases — 65.0%, against 46.2% for GPT-3.5 and 23.8% for Claude-v1 — and describe a judgement that flips when two responses swap positions (arXiv:2306.05685 , Table 2). The same paper puts GPT-4–human agreement at 85% against 81% human–human. It is a different grading procedure with a different documented failure mode.

2.5 What the runner could reach

Another variable is the environment the runner was allowed to touch. CAISI published an account of finding, after the fact, that it had been running SWE-bench Verified with internet access while other evaluators ran without it, and that it learned this from transcripts other evaluators had posted rather than from the benchmark’s documentation (Cheating on AI Agent Evaluations , December 2025). Its new policy for coding evaluations is “fully offline”. Reachability matters for static benchmarks too: Scale’s search-time contamination work found roughly 3% of questions retrievable with labels from Hugging Face, and blocking that source cut accuracy on the contaminated subset by about 15 points (Search-Time Data Contamination ).

Where CAISI compares its own results to self-reported ones — SWE-bench Verified at 63.0 against 74.9 self-reported for one model — it lists possible sources of the differences, including dataset differences, agent setup and API sampling parameters such as temperature and top_p (CAISI Evaluation of DeepSeek AI Models , Appendix A8).

None of this is exotic. A 2026 survey of 101,843 published results across 5,816 models and 635 benchmarks found that 48,698 (96.5%) lack at least one field from its minimal reproducibility sub-schema, that temperature is absent from 93.9%, and recorded one model reported at a 20.9% score by Hugging Face and 61.8% by a second evaluator on MMLU-Pro (Evaluation Cards, arXiv:2606.09809 ). The survey documents incomplete reporting at scale; it does not say how often this exact combination of differences occurs.

3. Comparability is a property of the reference, not of the number

VIM §2.46 defines metrological comparability of measurement results through traceability to a common reference, as comparability of results, for quantities of a given kind, “that are metrologically traceable to the same reference” (JCGM 200:2012). Its §2.41 note permits a measurement procedure to serve as that reference for a non-ordinal quantity. The AI-Eval profile turns that into a narrower operational rule: a bridge is applicable only when the frames meet its exact aspect, scope and procedure constraints, and mmlu alone does not meet them.

Metrological compatibility, §2.47, is a separate VIM term: it concerns whether the difference between two results is small relative to the uncertainty of that difference. It does not establish that two results share a reference.

Fields outside AI have already paid for this. HbA1c has two reference methods, and NGSP reports IFCC results running 1.5 to 2% HbA1c lower across the range (NGSP ); in 2007 the IFCC value was given its own unit, mmol/mol, which NGSP says avoids confusion between the two systems. Cloud availability has the same shape: AWS EC2 derives Monthly Uptime Percentage from minutes of Unavailability, defined as loss of external connectivity (AWS EC2 SLA ), while Google Compute counts loss of persistent disk access as Downtime and does not count intermittent downtime shorter than one minute (Google Compute Engine SLA ). Two contracts print “99.99%” over different measurands.

The AI-specific documents arrive at the same statement without the metrology words. NIST’s draft practices for automated benchmark evaluations tie comparability to consistency of protocol between systems (§2.1) and describe breaking changes under semantic versioning as points after which results on either side are no longer properly comparable (NIST AI 800-2 ipd , January 2026). Practice 2.3 lists grouping together the evaluation logs that are meant to be compared as an emerging practice, and Practice 3.2 notes that an interoperable schema for sharing evaluation details may ease replication. The NAAIMES network best practice of July 2026 splits the property in two, the evaluation of outputs (scoring criteria, rubrics, graders) and the generation of outputs (prompt format, inference settings), holds that the first should always be comparable, and treats the second as depending on the goals of the evaluation (§C.1.1).

grader_id records the grading procedure, while prompt_protocol records part of output generation; the two frames differ in both of those fields and in runner_id. An earlier post here put the general form in one line — a trail that records “accuracy 0.78” without the frame records a number, not an observation. The two MMLU scores are that line’s specific case with the hashes filled in.

4. Binding the number to its frame

Under APL, the frame is not documentation attached to a claim but a separate content-addressed object, and the claim carries only its hash. The AI-Eval profile — the vocabulary for benchmark observations about a model build — fixes what that object must contain. Frame A, indented for reading:

{
  "version": "0.1",
  "observer": { "id": "acme-eval-lab" },
  "procedure": {
    "runner_id": "lm-eval-harness@0.4.2",
    "grader_id": "exact-match-v1",
    "prompt_protocol": "zero-shot-mcq-v1"
  },
  "aspect": ["accuracy"],
  "scope": {
    "benchmark_id": "mmlu",
    "benchmark_variant": "default",
    "dataset_split": "dev",
    "subset": "all"
  },
  "invariance": ["score-object-serialization"],
  "exclusions": [
    "no-production-readiness-claim",
    "no-deployment-safety-claim",
    "no-out-of-scope-generalization-claim"
  ]
}

Its hash is sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab, and the indentation above is display only: the hash is SHA-256 over the RFC 8785 canonical bytes (src/core/jcs.rs:167, canonical_hash), so whitespace never reaches the digest. Frame B is the same object with runner_id at custom-runner@2.1, grader_id at llm-judge-v3, dataset_split at test-lite and no subset key, hashing to sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e.

The profile validator turns those fields from convention into a requirement. procedure must be an object with a non-empty runner_id (frame.rs:35) and grader_id (frame.rs:36); scope must carry benchmark_id, benchmark_variant and dataset_split (frame.rs:48–50); aspect must hold exactly one entry from a closed list (frame.rs:51–57); exclusions must contain all three markers, which is how a frame states what it is not evidence of (frame.rs:69–76). On the claim side, content.benchmark_id is required (claim.rs:265) and must equal frame.scope.benchmark_id (claim.rs:106–121), and content.metric_id is required and constrained by the aspect — accuracy admits accuracy, exact-match or f1 (claim.rs:268, 315–318).

The distance between a description and a validator is visible in the project’s own history. The design document that introduced this pair writes procedure as a human-readable string, invariance as “score object serialization” with spaces, the exclusions as two sentences of prose, the claim with content.benchmark and no metric_id, and the hashes as placeholders. The document’s examples use fields the current crate rejects. A prose value has no controlled vocabulary; different strings serialize differently and cannot satisfy exact equality.

A subtler case sits inside the two frames. Frame A writes subset: "all"; frame B omits the key. A reader might expect the comparison to normalize that, an absent optional field standing for its default. It does not. Scope equality is decided by canonical bytes (bridge.rs:114–118 calling canonical_equal, implemented at src/core/jcs.rs:84–87 as canonical_bytes(a) == canonical_bytes(b)), and serde’s derived PartialEq is explicitly not used for it. Under that rule “all” and absent are two scopes. The rule leaves defaulting to the tool that emits the frame rather than to the comparison step.

5. What the verifier says

apl-ai-eval is a Rust crate, version 0.3.1, Apache-2.0. Given a receipt or a pair of receipts and a relation query, it decides whether the claims are well-formed under the profile and whether the relation is evaluable. The case in this post lives in its test vectors as two_mmlu_scores_incomparable_with_ai_eval_profile, with the pair in test_data/vectors/pairwise/incomparable-no-bridge.json and two sibling vector files containing bridge cases. cargo test -p apl-ai-eval runs 109 unit tests, 3 vector integration tests and 2 doc-tests; all 114 are green.

Verify claim A on its own against the AI-Eval profile and the output is:

{"core_outcome":"apl-valid","relation_outcome":"relation-not-evaluated","failure_classes":[],"diagnostics":["carrier-valid","apl-present","apl-frame-bound","apl-valid","same-frame","transformation-missing"]}

Claim B alone produces the same shape. apl-valid here means well-formed and frame-bound, and says nothing about the accuracy of 0.781; relation-not-evaluated is the answer to a question that was not asked.

Now the pair, with the score-delta query from Section 1 and no bridge:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"incomparable","diagnostics":["apl-cross-frame","apl-bridge-not-found"]}

Both claims have core_outcome: "apl-valid" and empty failure classes; the relation outcome is incomparable. apl-cross-frame states that the two frame_ref hashes differ; apl-bridge-not-found, that nothing was supplied to license a comparison across that difference. The two axes are separate by design: this is not a broken record and not a passing comparison. It is a subtraction with no defined result, reported as such.

The third vector is a separate adversarial case, not a bridge between Frames A and B above. Its source frame declares aspect accuracy and its target frame declares aspect judge-score. The bridge is itself a content-addressed object, and this one is well-formed:

{"version":"0.1","bridge_kind":"runner-equivalence","source_frame":{"hash":"<FRAME_HASH:0>"},"target_frame":{"hash":"<FRAME_HASH:1>"},"comparison_scope":{"source_aspects":["accuracy"],"target_aspects":["judge-score"],"relation_type":"score-delta"},"assumptions":["all MMLU results are interchangeable"],"losses":[]}

The <FRAME_HASH:n> tokens are placeholders in the recorded vector; the harness substitutes the two frame hashes at load time. The bridge parses, points at the right two frames, declares an empty losses array and states its assumption in plain text. The verifier’s answer:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"incomparable","diagnostics":["apl-cross-frame","apl-ai-eval-bridge-aspect-family-mismatch"]}

The reported diagnostic is apl-ai-eval-bridge-aspect-family-mismatch. The bridge’s comparison_scope declares accuracy on the source side and judge-score on the target side, matching the two frames; under the profile a bridge may translate between conditions within one aspect family and may not join frames that measure different aspects.

The fourth vector is a bridge that works — same grader exact-match-v1 on both sides, same scope mmlu / default / dev / all, only runner_id differing:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"bridged-comparable","diagnostics":["apl-cross-frame","apl-bridge-applicable"]}

bridged-comparable indicates that the frames differ and that an applicable bridge, with its stated assumptions and losses, was supplied.

The profile permits a repeatability bridge only for the repeatability-check relation type; it does not license score-delta. Repeatability of a procedure says nothing about a difference between two subjects measured under it. The code pins repeatability bridges to the repeatability-check relation type, rejecting anything else with apl-ai-eval-relation-type-invalid (bridge.rs:167–172), and confines runner- and grader-equivalence bridges to score-delta (bridge.rs:129–164).

Malformed records sit on a separate axis. A claim whose subject lacks type and artifact_digest, whose procedure and scope are strings rather than objects, and whose exclusions are incomplete returns apl-invalid, failure_classes: ["claim-structure-failure"], diagnostic apl-subject-invalid. Invalid means the record is not a well-formed observation; incomparable means two well-formed observations cannot be subtracted.

The crate is at apl-protocol.org/implementations/apl-ai-eval ; the source, the vectors and the three pairwise JSON files quoted above are at github.com/evidentum-io/apl-core/tree/main/apl-ai-eval .

6. What this does not prove

The frame settles a narrow question and leaves the rest open.

• That either score is correct. apl-valid is a statement about structure. MMLU-Redux re-annotated 5,700 questions and estimates “that 6.49% of MMLU questions contain errors” (arXiv:2406.04127 ). A frame binds a number to its declared conditions; it does not establish that those conditions were executed as declared.
• That two identical identifiers name the same thing. Byte equality knows that custom-runner@2.1 equals custom-runner@2.1. It does not know whether the two runs shared a commit, an accelerator, a tokenizer build or sampling defaults. The frame moves the trust boundary to whoever mints identifiers; it does not remove it.
• That the dataset was the same dataset. The frame identifies data by name and split, not by digest. The 14079-against-14042 gap in Section 2.1 confirms an open question left in the earlier post: if a name resolves to two different sets of questions, whether a dataset digest should be recorded remains an open design question.
• Anything about thresholds, or when they were set. A frame records the procedure, not a release threshold or when it was chosen. NIST’s Practice 1.2 addresses documenting what an evaluation measures before conducting it.
• Anything about uncertainty. Profile v0.1 specifies no uncertainty field, and neither claim records sample_count. For illustration, a proportion estimated from 285 independent items has a wider binomial interval than one estimated from 14,042 items; that comparison does not establish the uncertainty of either displayed claim. ISO/IEC 17025:2017, 7.8.3.1 c), calls for reporting measurement uncertainty where relevant. AI-Eval v0.1 specifies no field for it.
• That the split labels mean what a reader assumes. dev is a real MMLU split and, by the paper’s design, the source of its few-shot examples; scoring on it is a decision the frame makes visible and does not defend. test-lite is the label in the crate’s test vectors and names no published artifact.
• That the vocabulary is stable. AI-Eval is version 0.1, a draft. NIST describes a breaking change in an evaluation as a point after which results on either side are no longer properly comparable.

What the frame establishes is narrower: it binds a stated procedure to a stated claim by hash. Under AI-Eval v0.1, the supplied frames and this score-delta query do not satisfy the conditions for an applicable bridge, so the verifier returns incomparable rather than a delta.

If you are building evaluation infrastructure where the numbers will be compared across teams, vendors or quarters, and the comparison has to survive being questioned, that is contract work I take on.