We Ran a 10× Bigger Protein Model. It Didn't Rank Variants Any Better.
On this page
The bigger model didn’t win
We benchmarked ESM-2A protein language model by Meta AI trained on 250 million protein sequences. Predicts how amino acid mutations affect protein function from sequence alone — no structure required.Full definition 650M against ESM-C, EvolutionaryScale’s newer protein language modelA deep learning model trained on millions of protein sequences to predict how mutations affect function. NeuroAutomata uses ESM-2, a PLM developed by Meta AI.Full definition family, at two sizes, 600M and 6B parameters, to answer one practical question: does a 10× larger model rank protein variants better?
It does not. On 201 ProteinGymA standardized benchmark suite for protein variant effect predictors, covering 217 deep mutational scanning assays across diverse protein families.Full definition substitution assays scored identically across all three models, our grouped-average Spearman rank correlationA rank correlation coefficient (−1 to +1) that measures whether two variables agree in order, not magnitude. The primary metric for variant effect benchmarks.Full definition from this internal validation comes out as ESM-2 650M 0.431, ESM-C 600M 0.425, ESM-C 6B 0.419 CLM-ESMC6B-0004Verified. The point estimates decrease as ESM-C scales up. Every pairwise difference is statistically non-significant, so the honest read is a three-way dead heat: scaling ESM-C from 600 million to 6 billion parameters bought no zero-shot gain over ESM-2 650M, and if anything a slight regression.
If you are ranking mutations today, the takeaway is short: for zero-shot variant-effect ranking, ESM-2 650M holds. There is no accuracy reason to switch to ESM-C or to pay for the 6B model.
NOTE
What “zero-shot ranking” means here. We are measuring rank quality: whether a model orders variants from most- to least-damaging the same way a wet-lab experiment does, with no task-specific fine-tuning. This is a ranking-and-triage signal, not a damaging-versus-benign classifier, and Spearman rho is a rank correlation: higher means the model’s ordering agrees more with the experiment. It says nothing about ESM-C’s strengths elsewhere (structure, representation learning, supervised probes). It speaks only to zero-shot variant ranking on this suite.
What we measured
We scored each model with masked-marginal scoringESM-2's zero-shot method for scoring variant effects. It masks each position and measures how surprising the mutant amino acid is relative to wild-type.Full definition on the same set of deep mutational scanningA lab technique that measures the functional effect of every possible single amino acid substitution across a protein. The gold standard for variant effect data.Full definition assays drawn from ProteinGym, then computed Spearman rank correlation between each model’s scores and the experimental effect for every assay. To keep the comparison honest, all three models were run through the identical harness and scored on the 201 assays common to all three: same variants, same protocol, same grouped (five-category weighted) averaging that ProteinGym uses to avoid letting a few large assays dominate the headline. The public model rankings for this benchmark are published on the live ProteinGym leaderboard CSV (accessed 2026-05-08).
Because the head-to-head is restricted to that shared 201-assay subset, ESM-2 650M’s grouped average here (0.431) is computed on those 201 assays specifically. That is a different, scope-bound subset from our full-suite published ESM-2 baseline, not directly comparable to it. Every number in this post is the 201-assay common-set value.
This is internal validation: we ran each model against our own implementation of the scoring harness and compared to published experimental DMS data distributed through ProteinGym (Notin et al. 2023). ESM-C is a model we were evaluating for our own adoption decision (see the disclosure below).
ESM-C 6B vs ESM-2 650M: a non-significant regression
Against ESM-2 650M on the shared 201 assays, ESM-C 6B’s grouped average is 0.419 versus 0.431 CLM-ESMC6B-0001Verified. Run as a paired, per-assay comparison, the mean difference is −0.0081 (SD 0.136, SE 0.0096), a two-sided t of −0.846 at p = 0.40, well short of significance. Across the 201 assays, ESM-C 6B ranks variants better than ESM-2 on 101 of them (50.2%) CLM-ESMC6B-0001Verified, a coin flip.
In plain terms: the larger, newer model and the older, smaller one are indistinguishable on this task, and the small gap that does exist points the wrong way for the bigger model.
Scaling ESM-C didn’t help within its own family
The cleaner test of “does scale help” holds the architecture fixed and only changes size: ESM-C 600M versus ESM-C 6B. Here the 6B model scores 0.419 against the 600M model’s 0.425 CLM-ESMC6B-0002Verified, again below, with a paired mean difference of −0.0044 (SD 0.094, SE 0.0066), t = −0.663, p = 0.51. Non-significant, but the direction is consistent with the cross-family result.
And on the head-to-head win rate, the 6B model beats its own 600M sibling on only 94 of 201 assays (46.8%) CLM-ESMC6B-0003Verified, below the 50% break-even. A model with ten times the parameters loses to the smaller version of itself more often than it wins.
| Model | Parameters | Grouped Spearman rho (201 assays) | ESM-C 6B win rate vs this model |
|---|---|---|---|
| ESM-2 650M | 650M | 0.431 | 101 / 201 (50.2%) |
| ESM-C 600M | 600M | 0.425 | 94 / 201 (46.8%) |
| ESM-C 6B | 6B | 0.419 | n/a |
So should you switch?
The bench question that motivated this was concrete: “should I move my variant-ranking pipeline to ESM-C, or pay for the 6B model?” The evidence says no zero-shot accuracy reason to. On the task of ordering single-substitution variants by predicted effect, ESM-2 650M and ESM-C (both sizes) sit in a statistical tie, and the extra compute of a 6B model buys nothing measurable for this use. That is a useful negative result: it lets a lab keep a smaller, cheaper model in production without giving up ranking quality.
The broader lesson is one we keep re-learning in public: bigger and newer is a hypothesis, not a result. We ran the frontier model on our own benchmark, on the same assays and the same protocol, and reported what came back, including when it argued against the more exciting story.
Disclosures
Research use only. NeuroAutomata’s benchmark outputs are for research use, not clinical or diagnostic decision-making.
First-party benchmark. We ran this evaluation to inform NeuroAutomata’s own model-adoption decision. It is not an independent, third-party benchmark.
Against-interest result. The finding is negative: it argues against adopting the larger, more expensive ESM-C 6B model, and therefore against our own commercial incentive to sell more compute. We report it because it is what the data showed.