ML on the Materials Project, part 7: knowing what you don't know
A Gaussian Process predicts a value and an honest error bar. We audited whether the bars can be trusted, and found the model measured its own noise floor.
Every model in this series so far has output a bare number: 2.1 eV, with no confidence attached, treating its best guesses and its desperate ones identically. For a discovery pipeline that is a real weakness: “2.1 plus or minus 0.1” and “2.1 plus or minus 1.5” describe very different situations wearing the exact same number.
The Gaussian Process idea
A Gaussian Process does not learn a single function the way earlier models did. It effectively keeps every function consistent with the training data, under one governing assumption: similar inputs should give similar outputs, smoothly. For a new material, the prediction is the average across all of those functions, and the uncertainty is how much they disagree with each other. In territory the model has seen plenty of, the functions mostly agree, a tight error bar. Far from anything in the training data, they diverge, a wide one. Distance from experience becomes an actual number, which is the same property behind active learning and self-driving laboratories: measure next wherever the bars are widest.
The price for this honesty is computational: a Gaussian Process compares every point against every other point, so cost grows with the cube of the training size. That means training on 2,000 materials here, not the full 26,000-plus. Accuracy was traded for honesty on purpose. And following part 4’s law, the split stays grouped by formula throughout, so there is no polymorph leakage hiding in any of these numbers.
The results
Accuracy came in at R2 0.273, MAE 1.011 eV, weaker than the Random Forest’s honest 0.446, exactly as expected from a tenth of the training data and a smoothness assumption that real chemistry only partly obeys. That was the trade being made, and it was made in the open.
Calibration is where this model earns its keep. 71 percent of true values fell within one standard deviation of the model’s own claimed uncertainty (an honest bar should promise roughly 68 percent), and 94 percent fell within two standard deviations (honest promise: roughly 95 percent). The bars tell the truth.
The money test. The 20 percent of predictions the model was most confident about missed by an average of 0.697 eV. The least confident 20 percent missed by 1.127 eV. The model is measurably more accurate exactly where it claims to be confident, which means it genuinely knows when it is guessing, not just producing a number that looks like a confidence score.
The quiet star of the whole episode: the model’s kernel learned a noise floor of 0.72 eV entirely on its own, its estimate of the irreducible scatter that no model could ever explain away. We already know exactly what lives in that number: polymorph twins with identical features and different true gaps. The representation ceiling that has followed this series since part 1 shows up again here, for a fifth time, this time self-reported by the model measuring its own limits from the inside.
A small bonus: the kernel pushed several features’ length-scales toward infinity, its own way of saying these features barely matter, a preview of the question part 8 asks properly.
The takeaway
An honest “I don’t know” is worth more than a confident guess, and it turns out to be purchasable: the Gaussian Process paid away some accuracy for calibrated uncertainty. In any discovery pipeline where the next experiment costs real money, that is usually the right trade to make.
Part 8, the finale: which features does the model actually use, and did it learn real chemistry, or something else entirely?
Found a mistake? Good, tell me. This publication flags its own suspect values. Reach me on LinkedIn.