ML on the Materials Project, part 4: were we lying to ourselves?
Grouped cross-validation measures how much a model's score was flattered by polymorph twins leaking between train and test. The answer: thirty percent.
Every part so far has carried the same footnote: the scores are still slightly flattered, because polymorph twins are scattered across the train and test sets. Part 4 stops footnoting it and measures it.
The crime, named
Since part 1 we have known the suspects. 4,463 formulas in this dataset appear as multiple polymorph entries, identical composition features, different true properties. SiO2 alone appears 322 times. A random train/test split scatters those twins across both sides, so the model effectively “predicts” a test material while holding a near-duplicate of the same question, answer included, in its training memory.
This is data leakage, specifically group leakage: information about the test set reaching the model through groups of related rows. It is one of the most common ways published machine learning results flatter themselves, not through fraud, just through a split that quietly cheats.
The tools
Cross-validation. Instead of one 80/20 split, cut the data into five folds. Train on four, test on the fifth, rotate five times. The result is a mean score and a spread, so one lucky split can no longer make a model look better than it is.
GroupKFold, the leakage fix. Declare every row sharing a formula to be one group, and require each group to stay entirely on one side of every split. After this change, the exam contains no formula the model has ever studied. A correct answer can only come from generalising chemistry, not from recognising a question.
Running both schemes on the same models turns the difference between them into a direct measurement of the flattery.
The prediction, made before running anything
Linear regression should barely move: it is too simple to memorise its training data, so it has nothing to lose when memorisation becomes impossible. The Random Forest, which memorises plenty, should drop.
The results (5-fold cross-validation, mean R2 and spread)
| model | naive KFold (leaky) | GroupKFold (honest) | loss |
|---|---|---|---|
| linear regression | 0.091 +/- 0.009 | 0.068 +/- 0.033 | 0.023 |
| random forest (300) | 0.638 +/- 0.010 | 0.446 +/- 0.047 | 0.192 |
Three findings:
The forest lost 0.19 of its R2: thirty percent of its apparent skill was leakage. The 0.638 celebrated in part 2 becomes an honest 0.446. Nothing about the model changed; the only thing that changed is that it was no longer allowed to recognise questions it had already seen.
The prediction held. Linear regression barely moved. This is not grouped cross-validation being generically “harsher” on everyone: only the model that had been using the memorisation channel paid a price when that channel closed. The contrast between the two models isolates the mechanism cleanly.
Honest evaluation is also wobblier. The honest spread nearly tripled, from 0.010 to 0.047. Grouped folds contain genuinely different chemistry from one fold to the next, so scores vary more between them. The leaky scheme had been hiding that uncertainty as well as the inflated score.
The habit this installs
Whenever rows in a dataset arrive in related groups, polymorphs of one formula, repeated measurements of one sample, multiple patients from one hospital, the split must respect the groups, or the resulting score is a flattered one. When a paper reports a high R2 using random cross-validation on a dataset that plainly has such groups, the first question is now automatic: what happens to that number under a grouped split?
0.446 is this series’ official Random Forest number from this point forward.
The takeaway
The most valuable number this series has produced so far was not a score. It was the difference between two scores.
Part 5: stop predicting, start seeing. What does the shape of 26,566 materials look like with no labels at all?
Found a mistake? Good, tell me. This publication flags its own suspect values. Reach me on LinkedIn.