Verification Report

This report documents the independent verification of the article rho = 0.772 in BRCA1 BRCT: What 5,949 Cancer Variants Reveal About ESM-2’s Limits.

---

Validation Methodology

This is a two-candidate comparison post: BRCA1 (saturation genome editing, Findlay et al. 2018) and PTEN (VAMP-seq, Matreyek et al. 2018), evaluated against ESM-2 650M as a combined hereditary cancer VUS benchmark. Each candidate is kept at its natural grain — one paper, one DMS dataset per candidate — with the framing-against-prose alignment handled at the level of the combined post.

BRCA1 DMS source: Findlay et al. 2018, Nature 562(7726):217–222 (doi:10.1038/s41586-018-0461-z). Saturation genome editing (SGE) via CRISPR/Cas9-introduced single-nucleotide variants at the endogenous BRCA1 locus in HAP1 cells; cell viability selection. 1,837 missense variants across the RING (residues 1-101, zinc-binding E3 ubiquitin ligase) and BRCT (residues 1631-1863, phosphopeptide-binding) domains. Data accessed via ProteinGym (Notin et al. 2023, NeurIPS) standardized substitution DMS leaderboard.

PTEN DMS source: Matreyek et al. 2018, Nature Genetics 50(6):874–882 (doi:10.1038/s41588-018-0122-z). VAMP-seq (Variant Abundance by Massively Parallel sequencing) measuring intracellular protein abundance for 4,112 missense variants (53.8% of all possible single amino acid substitutions across PTEN’s 403 amino acids, UniProt P60484). Data accessed via MaveDB.

ESM-2 scoring: 650M-parameter masked-marginal scoring on the NeuroAutomata platform. BRCA1 RING (101 amino acids, PDB 1JM7) and BRCT (233 amino acids, PDB 1T29) were scored as independent fragments — BRCA1’s full length (1,863 amino acids) exceeds the platform’s 1,024 amino acid input limit, but both domains fold independently. PTEN was scored full-length (403 amino acids). We cross-referenced variants against published DMS scores per protein.

BRCA1 global results. BRCT Spearman rho = 0.534 (N = 1,262, p = 9.8 × 10⁻⁹⁴, categorical agreement 67.7%); RING Spearman rho = 0.409 (N = 575, p = 1.4 × 10⁻²⁴, categorical agreement 75.5%). The BRCT domain — which harbors the majority of clinically actionable BRCA1 VUS — shows the stronger overall correlation. The RING result is consistent with the 101 amino acid fragment giving ESM-2 less evolutionary context than the 233 amino acid BRCT tandem repeat.

BRCA1 BRCT regional breakdown. Non-active-site annotated positions: rho = 0.772 (N = 59, p = 8.6 × 10⁻¹³) — the second strongest domain-level correlation across our benchmarks (behind CYP2C9’s heme-binding rho = 0.811). Active-site phosphopeptide-binding: rho = 0.700 (N = 9, categorical agreement 88.9%). Unannotated: rho = 0.512 (N = 1,194, p = 7.2 × 10⁻⁸¹).

BRCA1 RING zinc-binding result. At the eight conserved zinc-coordinating residues (C24, C27, C39, H41, C44, C47, C61, C64), ESM-2 and SGE reach 100% categorical agreement (49/49 variants identified as deleterious by both methods). The Spearman rho within the zinc-binding set is 0.020 (non-significant) — because all 49 variants are uniformly deleterious, rank-order is not meaningful within the set. The 100% categorical agreement is the clinically relevant result; the C61G founder pathogenic variant (ClinVar VCV000017661) is correctly scored as non-functional by both methods (SGE function score = -1.738).

PTEN global and regional results. Global Spearman rho = 0.484 (N = 4,112, p = 4.6 × 10⁻²⁴⁰); non-active-site annotated: rho = 0.536 (N = 205, p = 1.3 × 10⁻¹⁶). The active-site CX5R phosphatase motif (residues C124-R130): rho = −0.011 (N = 84, p = 0.92, not significant). This zero-correlation at the catalytic center is the methodologically load-bearing finding: ESM-2 correctly identifies C124 and R130 as evolutionarily invariant (deep negative scores), but VAMP-seq measures abundance, not catalytic activity — and catalytic-dead variants remain structurally stable, so VAMP-seq scores them near-WT. The mismatch is not an ESM-2 failure mode but an assay-coverage gap: ESM-2 captures catalytic constraint; VAMP-seq does not.

A note on PTEN baselines and which publication each number comes from. The PTEN benchmark we report on this page (Spearman rho = 0.484, n = 4,112 variants) is calculated against Matreyek et al. 2018 (Nature Genetics) — the original VAMP-seq abundance deposit (MaveDB urn:mavedb:00000013-a-1). The ProteinGym leaderboard, by contrast, scores PTEN against an expanded 2021 dataset (Matreyek et al., Genome Medicine; ProteinGym’s processed cut: n = 5,083 variants, vs the paper’s 4,721-filtered / 4,387-missense headline counts), which yields a different per-protein baseline (rho = 0.465).

These are not the same number reported by different sources. They are two different variant sets — the 2021 dataset expanded on the 2018 deposit, and our score and the ProteinGym leaderboard score cannot be directly compared one-to-one. The VAMP-seq assay was introduced in the 2018 paper, which is the methodology we report against; the 2021 paper is the source of the leaderboard’s number. Both citations are correct in their own context.

How to read the Spearman correlation. As the article notes, rho measures agreement on ordering, not per-variant accuracy. For the BRCT rho = 0.534 result, the concordance probability (1 + 0.534) / 2 ≈ 0.77 — ESM-2 agrees with SGE on the ordering of variant effects roughly 77% of the time. For clinical use: ESM-2 helps prioritize which VUS to investigate first; it does not independently classify them. The published prose frames all VUS-related claims as descriptive (rank prioritization), not as ACMG/AMP variant classifications.

---

Summary

Metric	Value
Claims	17 total
Result	All checked; the page passed verification
Per-claim outcome	8 verified · 5 single-source measurements · 4 reference facts (0 contradicted, 0 uncheckable)
Numbers checked for arithmetic consistency	8 of 17
Categorical classifications made	0 (none authored — see “What Was Checked”)
Source data	BRCA1/PTEN validation datasets (versions recorded below at build time)
Verified	2026-05-26
Verified by	Veritas

---

What Was Checked

Source-against-claim review (all 17 claims). Every claim was checked end-to-end against the primary source it cites — whether the source actually says what we claim, and whether the scope matches. The stricter framing-fidelity check — whether the prose framing is faithful to what the cited authors reported — ran additionally on the verified subset only.

Cited primary sources (independently resolved and checked figure-by-figure, with single-source measurements disclosed as such): Findlay et al. 2018 Nature (BRCA1 SGE); Matreyek et al. 2018 Nat Genet (PTEN VAMP-seq); the MaveDB record urn:mavedb:00000013-a-1 (PTEN); the ProteinGym substitution DMS leaderboard; and ClinVar (for the variant-concordance references). Of the eight verified claims, all are ESM-2’s own measurements compared against the source DMS — five against the Findlay 2018 SGE data, three against the Matreyek 2018 VAMP-seq data — and each additionally passed the framing-fidelity check.

Context sources (referenced in the article for background, outside the formal verification scope): PDB 1JM7 (BRCA1 RING structure), PDB 1T29 (BRCA1 BRCT structure), UniProt P60484 (PTEN sequence reference), UniProt P38398 (BRCA1 sequence reference), ClinVar VCV000017661 (BRCA1 C61G founder pathogenic variant), and Chen et al. 2023 JAMA Network Open (the 41% multi-specialty VUS-rate clinical-context figure).

Arithmetic-consistency check (8 of 17 quantitative claims). Every Spearman rho, sample size, p-value, and categorical-agreement percentage was checked for internal consistency: rho values within the valid range (−1 to 1) across BRCT (the 0.534 + 0.772 + 0.700 + 0.512 sub-regions), RING (0.409 global + 0.020 zinc-binding set), and PTEN (0.484 + 0.536 + −0.011 sub-regions); sample-size accounting; p-value bounds; the concordance-probability derivation ((1 + ρ) / 2); the categorical-agreement percentages; and the 49/49 zinc-binding-set 100%-agreement count.

Five single-source measurements. A further 5 claims are single-source DMS values — specific Spearman rho values for sub-regions, taken from a single DMS record or a single leaderboard row, with no independent second source for the same variant-specific value. We disclose these as exactly that — measured once, not independently corroborated. This is an honest, bounded category, distinct from a verified claim; it is not a coverage gap. The remaining 4 claims (two methodological framing claims, one existence claim, and one structural-comparison claim) are reference facts confirmed by source-against-claim review only, with no arithmetic in them to check.

Categorical-classification check — 0 claims, none authored. All BRCA1 and PTEN variant claims are framed as descriptive measurements — rank prioritization, regional correlation, categorical agreement rates — not as ACMG/AMP variant classifications. The article states it plainly: “ESM-2 helps prioritize which VUS to investigate first — it does not independently classify them.” The underlying record matches that framing throughout.

---

How This Page Works

This is the first verification page covering two candidates in one combined argument. Each candidate (BRCA1-Findlay and PTEN-Matreyek) keeps its own evidence trail at its natural grain — one paper, one DMS dataset — while a single combined claim list sits at the level of the published article, because the article reads as one comparative argument. Each claim in the list records which of the two candidates it is sourced against, so the per-claim source is auditable; the claim-by-claim table below shows that mapping.

The exact source-data versions and verification-result version used at this page’s most recent build are shown in the footer below — an evidence chain back to a specific, reproducible state.

---

Verification independence

Verification is performed by Veritas, an independent agent with no involvement in producing the content it checks.