Materials Decoded Lab · Artifact 01
Can I trust this number?
Every AI-for-materials system is racing to generate candidates faster. Almost nobody is checking whether the data underneath can carry the weight. This is a benchmark that audits the data instead of serving it.
26.2%
of formation energies agree
within 0.05 eV/atom, the conventional threshold
0.119eV/atom
mean absolute difference
more than double the threshold
85.2%
of band gaps agree
within 0.5 eV, a looser bar
344
structure-matched pairs
from 287 compositions audited
What was actually compared
One mistake makes this whole exercise worthless, and most of the work went into avoiding it. A chemical formula does not identify a material. Diamond and graphite are both carbon. Join two databases on formula and you can compare a calculation of one crystal structure against a calculation of an entirely different one, then report the mismatch as a disagreement.
That is not a small effect: for a single formula, computed band gaps can differ by more than4 eV across polymorphs, several times larger than the disagreement being measured. So nothing here is compared until the crystal structures have been matched, at pymatgen's default tolerances, deliberately not loosened. Records with no structure are never grouped with anything.
287 compositions in. Found in the Materials Project: 284. In OQMD: 223. In both: 222. After structure matching, 344 pairs genuinely describe the same material. Those are the only comparisons that mean anything.
Formation energies: 26.2% agreement
The conventional threshold for two DFT formation energies to be called equal is0.05 eV/atom. Of 344 structure-matched pairs, 90 met it. The mean absolute difference is 0.119 eV/atom, and the Materials Project sits 0.090 eV/atom below OQMD on average.
For scale, 0.05 eV/atom is roughly the accuracy people lean on when deciding whether a hypothetical compound is stable enough to be worth making.
The part where the hypothesis failed
The benchmark was built to test one idea: that Hubbard U policy explains the disagreement. The two databases apply +U to different elements under different rules, and +U moves energies substantially.
Band gaps: the idea holds
0.483 eV where policies differ
0.212 eV where they agree
More than a factor of two, exactly as predicted.
Formation energies: backwards
0.090 eV/atom where policies differ
0.130 eV/atom where they agree
Disagreement is larger where the policies match. The hypothesis is dead.
So what is really going on? The signed differences give it away. Where the policies agree there is a consistent one-directional offset; where they differ, the mean sits near zero but the spread widens. That points at the elemental reference energies and anion corrections each database fits independently. Every formation energy is defined against reference states, each project fits its own, and the resulting offset applies to ordinary main-group compounds and never cancels. Where +U policies also differ, the two effects partly offset, which makes that group look better on average while being less predictable.
The dominant term is not exotic physics. It is bookkeeping, which is much harder to notice.
Computed against measured
Separately, 200 of these materials have laboratory-measured band gaps. The Materials Project underestimates the measured value in 84.5% of cases, by0.83 eV on average.
This is expected and well understood: standard PBE calculations underestimate band gaps by construction. It is worth stating plainly anyway, because a model trained on this data predictsDFT band gaps, not measurements, and the distinction matters when someone spends money on the shortlist.
How confident can any of it be?
Every comparison is graded. The result is deliberately unflattering:0 of 2,592 records reach the high band.
A comparison whose base band is high agreed within half the disagreement threshold. If it does not end at high, the limit was provenance rather than disagreement.
Most records are not assessable because only one database reported them, so agreement cannot be measured at all. That is the honest answer, and it is more useful than a confident score built on a single unchecked source.
What the warnings say
| Flag | Records | What it means |
|---|---|---|
POLYMORPH_AMBIGUOUS | 2,322 | Structure could not be established, so identity is unproven. |
FUNCTIONAL_INFERRED | 2,121 | The functional was inferred from documentation, not stated per record. |
MAGNETIC_INFERRED | 2,117 | Magnetic ordering inferred rather than reported. |
HYPOTHETICAL | 1,929 | Not an experimentally observed structure. |
SINGLE_SOURCE | 1,904 | Only one database reported it, so agreement is unmeasurable. |
MAGNETIC_UNKNOWN | 378 | Magnetic ordering not available. |
CORRECTION_SCHEME_MISMATCH | 344 | Same functional, different correction schemes. |
LARGE_DISAGREEMENT | 305 | Sources differ by more than the threshold. |
HUBBARD_U_MISMATCH | 186 | The documented +U policies differ. |
FUNCTIONAL_MISMATCH | 104 | The two sources used different functionals. |
MAGNETIC_MISMATCH | 84 | The two sources assumed different magnetic orderings. |
Look up a material
All 287 audited compositions, with the +U attribution where the policies differ.
Loading the assayed set...
Run a live audit
The results above are a fixed benchmark run. This runs the same comparisonnow, on any composition you type, against the live Materials Project and OQMD APIs.
It needs your own free Materials Project API key, frommaterialsproject.org. The key is sent only with your audit request, is never stored, and every call runs against your quota rather than mine. No key, no live numbers: the Materials Project does not answer without one.
Explain the result in plain English (optional)
This is the only part of the page that involves a language model, and it is deliberately optional. Every number above is produced by the deterministic audit. The model may only describe what that audit already computed, and a guard checks each number in its answer against the tool output. If it states a number the audit did not produce, the answer is marked as failing rather than shown as sound.
Plates
What this is not
- It covers 287 compositions and two databases. Enough to establish the effect and attribute it, not enough to characterise all of chemistry.
- The material set deliberately over-samples compositions where +U policies differ, so the raw agreement fraction is not a random sample of either database.
- Structure matching at default tolerances will sometimes split phases a human would call the same.
- Disagreement between two calculations is not evidence that either is wrong. Only the comparison against measurement speaks to accuracy.
No language model computes, estimates or adjusts any number here. A deterministic core produces every value, and the explanation layer can only describe what that core emitted.