That's most of it. No one would want to sit in the grandstands to watch. The racetrack would develop a single groove where the least water had collected, and pulling out of it to pass would be nearly impossible. Every time the pack came by the fans would be doused with whatever water and grit was on the track surface.
Indycar experimented with this a number of years ago, they came to the conclusion that no one would want watch cars lap an oval in the rain. If
@Half Fast wants to call them names, I suppose he can, but he might want to try going down the front straight at 235 mph and making a hard left into turn 1 before doing so.
I have watched sports car races in the rain many times, and they can be interesting, but I wouldn't watch a wet oval race.