Do materials databases agree with each other? I measured it
I compared 344 structure-matched materials across the Materials Project and OQMD. Only 26 percent of formation energies agree within the conventional threshold.
Machine learning for materials runs on database values. We train on formation energies and band gaps pulled from the Materials Project, OQMD, AFLOW and others, and we treat those numbers the way we treat measurements: as the thing itself.
So I wanted to check something simple. If two of the largest open materials databases describe the same material, how closely do they agree?
I built a benchmark to find out. The short version: less closely than I expected, and the reason turned out not to be what I thought.
Getting the comparison right first
There is one mistake that makes this entire exercise worthless, and most of the work went into avoiding it.
A chemical formula does not identify a material. Diamond and graphite are both carbon. TiO2 is rutile, anatase or brookite depending on how the atoms are arranged, and those have genuinely different properties. If you join two databases on formula, you can end up comparing a calculation of one crystal structure to a calculation of a completely different one, then reporting the mismatch as a disagreement between the databases.
That is not a small effect. In an earlier pass over Materials Project data, I found the calculated band gap for a single formula could vary by more than 4 eV depending on which polymorph you picked, which is several times larger than the disagreement I was trying to measure. A careless join does not add noise. It manufactures a result.
So nothing in this benchmark compares two values until their crystal structures have been matched with pymatgen’s structure matcher, at default tolerances, deliberately not loosened. Records without a structure are never grouped with anything, because an unstructured record cannot be shown to be the same material as anything else. This is the same principle I wrote about in properties belong to structures.
That reduced a few hundred candidate compositions to 344 pairs where both databases genuinely describe the same material. Those are the only comparisons that mean anything.
Formation energies: 26 percent agreement
The conventional threshold for saying two DFT formation energies agree is 0.05 eV per atom.
Of the 344 structure-matched pairs, 26 percent met it.
The mean absolute difference was 0.119 eV per atom, more than double the threshold. The worst case reached 0.67 eV per atom. For scale, 0.05 eV per atom is roughly the accuracy people rely on when deciding whether a hypothetical compound is stable enough to be worth synthesising. Three quarters of these pairs disagree by more than that.
Band gaps: better, with a long tail
Band gaps held up more comfortably. 85 percent agreed within 0.5 eV, with a mean absolute difference of 0.28 eV.
But the tail is ugly. The largest disagreement was 3.98 eV, which is the difference between predicting a transparent insulator and predicting a semiconductor. If your screening pipeline hits one of those, no downstream statistics will save you.
The part where I was wrong
Here is the finding I did not expect, and it is the most useful thing in the benchmark.
My hypothesis going in was that Hubbard U would explain the disagreement. The two databases document different +U policies, applying the correction to different elements under different circumstances, and +U shifts energies substantially. It seemed obvious.
For band gaps, the hypothesis held cleanly. Where the two +U policies differ, mean disagreement is 0.483 eV. Where they agree, 0.212 eV. More than a factor of two, exactly as expected.
For formation energies, it came out backwards. Disagreement was larger where the +U policies agreed: 0.130 eV per atom against 0.090 where they differed.
That kills the hypothesis, so the interesting question becomes what is actually going on. The signed differences give it away. Where the policies agree, the Materials Project sits 0.117 eV per atom below OQMD on average, a consistent one-directional offset. Where policies differ, the mean signed difference is close to zero (0.016) but the spread is wider.
That pattern points somewhere else entirely: the elemental reference energies and anion corrections that each database fits independently. Every formation energy is defined relative to reference states for the constituent elements, and each project fits its own. That produces a systematic offset which applies to perfectly ordinary main-group compounds and never cancels out. Where +U policies also differ, the two effects partly offset each other, which is why that group looks better on average while actually being less predictable.
So the dominant term is not exotic physics. It is bookkeeping, and bookkeeping is much harder to notice.
What this means if you train models on this data
Three practical consequences.
Do not pool databases naively. Combining Materials Project and OQMD formation energies into one training set injects a systematic offset of roughly 0.1 eV per atom between subsets. Your model will spend capacity learning which database a value came from.
A model trained on one database predicts that database. Not “formation energy”, and certainly not experiment. This compounds a point I made about DFT band gaps underestimating measured ones: the same materials in this benchmark showed the Materials Project underestimating measured band gaps in 84.5 percent of cases, with 13.5 percent of materials computed as metals turning out to be measured semiconductors.
Disagreement is not noise. It is structured, attributable, and mostly explainable once you look at correction schemes and reference states. Which means it can be corrected for, if you know it is there.
Honest limitations
The benchmark covers 287 compositions and two databases. That is enough to establish the effect and attribute it, not enough to characterise it across all of chemistry. The material set was chosen to over-sample compositions where +U policies differ, so the raw agreement fraction is not a random sample of either database. Structure matching at default tolerances will occasionally split materials that a human would call the same phase. Every one of those choices is documented in the repository rather than buried.
The code
Everything is open, including the structure matching, the physics-consistency checks, and the full results:
github.com/ibtisamkhan96/materials-trust-benchmark
The design rule throughout is that no language model may compute, estimate or adjust a number. The deterministic core produces every value; the explanation layer can only describe what the core emitted, and a guard mechanically traces each number in its output back to a tool result.
Why I am building this
Every serious AI-for-materials effort right now, including GNoME, MatterGen and a growing pile of startups, is racing to generate candidate materials faster. Almost nobody is working on whether the underlying data can carry that weight.
Generation is scaling. Validation is not. That gap is where I think the interesting work is, and it is what I plan to keep building. If you want the practical version of this at the level of a single entry, I wrote how to audit a database entry.
Found a mistake? Good, tell me. This publication flags its own suspect values. Reach me on LinkedIn.