There really is a distinction. Barometric altimeters do make errors, even when properly adjusted, at nonstandard temperatures. They are systematics and are highly reproducible. The point is that everyone makes the same errors, so it doesn't matter, except for terrain avoidance. And terrain avoidance is done either visually or with large margins (though there are correction tables, and it can get significant when really, really cold).
The distinction is much more obvious in the flight levels, where the pressure is always presumed to be standard. You can make enough of a difference there using the GPS altitude to put you in an opposite-direction RVSM airway. Not good at all.