Frost is a buildup which I think would be more susceptible to continuing to build up even after engine start and taxiing? Or even as you fly.
Paint is normally smooth and polished, and even when cracked isn’t as ragged as frost, which often covers a large expanse of the wing and leading edge. And should actually be repainted. Shouldn’t it? It degrades performance too doesn’t it, but it isn’t as irregular as frost buildup. Also frost has weight associated with it. The original paint was factored in, and the overall affect of it flossing off is it gets lighter not heavier.