ML on the Materials Project, part 6: the statistical auditor
An Isolation Forest given zero rules flags the dataset's strangest entries on geometry alone, including a formation energy twenty times past any sane threshold.
A rule-based auditor works from rules a human wrote in advance: densities cannot be negative, hull distances above 0.5 eV per atom are suspect. Every rule encodes something a person already knew to check for. Part 6 asks the inverse question: can a model flag suspicious entries knowing no rules at all, purely by learning what “normal” looks like and noticing what isn’t?
The tool: Isolation Forest
Play this game with a single point in feature space: make random cuts, a random feature, a random threshold, until the point sits alone, isolated from everything else. A point buried deep in a dense crowd needs many cuts before it is alone. A point sitting out on its own gets isolated in two or three. Anomalies, by this definition, are simply the points that are easy to isolate. The Isolation Forest builds hundreds of these random cutting-trees and scores every material by its average isolation depth. No rules, no labels, just geometry.
The setup
The inputs are the 12 composition features plus four real properties, band gap, density, formation energy, and hull distance, all standardised. Including the actual properties (not just composition) lets the forest flag entries whose properties do not fit their composition, which is exactly an auditor’s territory.
One caveat frames everything that follows: statistically weird is not the same as wrong. Flags split three ways, data errors (the genuine treasure), exotic-but-real chemistry (fascinating, not broken), and boring artifacts. A human still has to judge which is which. This tool does triage, not verdicts.
What it caught, given zero rules
- Ta11O2, the single most impossible entry in the dataset. Formation energy +8.99 eV per atom, hull distance 9.71 eV per atom. A hand-written rule flagging anything above 0.5 would already catch this, and this entry is twenty times past that threshold, computational debris sitting in a production database. The forest ranked it number one without ever being told what formation energy means.
- Exotic metal-rich suboxides, W3O, Os3O, Th3O, Pu3O2, even Au3O (gold oxide). Mostly theoretical high-hull constructs, statistically lonely simply because real oxide chemistry rarely looks like this.
- The molecular gases, NO2, N2O, OF. Densities of 1.2 to 2.4, sitting at the bottom edge of what counts as a “solid” because they are essentially frozen gas molecules. Not errors, genuinely unusual citizens of an oxide dataset, the exotic-but-real category made concrete.
The cross-check: statistics versus rules
Simple hand-written rules (extreme thresholds on density, gap, or formation energy) flag 2,497 of the dataset’s 26,566 entries. Of the statistical forest’s top 100 anomalies, 58 percent also break one of those hand rules, and 42 percent break none of them at all, their individual values are each perfectly fine, but the combination is jointly unusual, exactly the class of anomaly no list of rules can enumerate in advance.
That 58/42 split is the honest division of labour: rules catch known failure modes and come with a built-in explanation attached, but they cannot catch combinations nobody thought to write a rule for. The statistical auditor catches the unusual without needing any foresight, but it cannot explain itself, and it cannot tell wrong from merely exotic on its own. They are complements, not rivals. Triage with statistics, judge with rules and a person who knows the chemistry.
The chart shows the top 100 anomalies living in two colonies at opposite edges of the map, the heavy suboxides on one side, the light molecular gases on the other, while ordinary chemistry sits untouched in the middle.
The takeaway
It is possible to find the weird entries in a dataset without knowing in advance what weird looks like. It still takes chemistry knowledge to decide which weird ones actually matter.
Part 7: what happens when a model is asked to report its own uncertainty, and does it tell the truth?
Found a mistake? Good, tell me. This publication flags its own suspect values. Reach me on LinkedIn.