ML on the Materials Project, part 3: is it even a metal?
Same features, a different question. Classification exposes the accuracy trap live, and asks why metallicity is hard to predict from composition.
Parts 1 and 2 asked “how big is the band gap?”, a regression question. Part 3 asks something simpler-sounding and harder-hitting: is this material a metal at all? Same 12 composition features, a different kind of answer, and a lesson about a very misleading number.
The setup
The target is is_metal, true or false. 22 unlabeled entries get dropped and reported (never silently guessed), leaving 26,544 materials: 65 percent non-metals, 35 percent metals. The split stays 80/20, stratified so the class balance is identical in both halves.
Three contestants sit the same exam:
- Majority baseline: always say “non-metal”, the score you get for free by refusing to think.
- Logistic regression: classification’s linear model, this time with feature scaling, which linear models need and trees do not.
- Random Forest classifier: part 2’s regression winner, in classifier form.
Four numbers, and why one of them lies
- Accuracy: the fraction of predictions that are right. With imbalanced classes this rewards cowardice: a model that always calls the common class scores the common class’s share, without having learned anything.
- Precision: of everything the model called a metal, what fraction really was one? How much you can trust its alarms.
- Recall: of all the actual metals, what fraction did it catch? How much slips past unnoticed.
- F1: the harmonic-mean compromise between the two.
Which one matters most is a cost question, not a maths question. In a discovery pipeline, a miss (low recall) loses a candidate material forever; a false alarm (low precision) only wastes one verification step.
The results
| model | accuracy | precision | recall | F1 |
|---|---|---|---|---|
| always say non-metal | 0.650 | 0.000 | 0.000 | 0.000 |
| logistic regression | 0.657 | 0.609 | 0.051 | 0.094 |
| random forest (300) | 0.766 | 0.704 | 0.572 | 0.631 |
Here is the accuracy trap, caught live rather than described hypothetically: logistic regression’s accuracy is 0.657, less than one percentage point above the do-nothing baseline. Judged by accuracy, that looks almost respectable. Judged by recall, 0.051, it found five percent of the real metals, it is nearly useless. No constructed example needed. The real data supplied the trap on its own.
The forest has real skill and honest limits: 77 percent accuracy, 70 percent trustworthy alarms, but it still misses 43 percent of real metals.
The confusion matrix (5,309 test materials)
| says METAL | says NON-METAL | |
|---|---|---|
| actually metal | TP 1,061 | FN 795 (missed) |
| actually non-metal | FP 447 (false alarm) | TN 3,006 |
The confusion matrix draws the two different ways of being wrong as two different numbers, rather than blending them into one score that hides which kind of mistake is happening.
Why metallicity is hard from composition alone
Whether a material conducts electricity is often decided by structure, not just what atoms are present. The extreme case is carbon: graphite conducts, diamond does not, same element, different arrangement. Our 12 features describe composition only; they cannot see structure. We are asking a structure-heavy question using structure-blind features, the same representation ceiling from parts 1 and 2, showing up for a third time in a new disguise.
The takeaway
Never report a classifier by accuracy alone, especially on imbalanced data. Report the confusion matrix, or at minimum precision and recall, and say plainly which error direction costs more for the job the model is actually doing.
Part 4: were we lying to ourselves this whole time? Measuring exactly how much every score so far has been flattered.
Found a mistake? Good, tell me. This publication flags its own suspect values. Reach me on LinkedIn.