Tecnically, SSDs do fragment. a file once written has a fixed size, and comes with an EOF or, end of file mark. The next file can begin just after the EOF and may or not be contiguous depending on the amount of linear space in the addressable range. Also, the first file may be modified such that it grows and one of the main jobs of any file system is to keep a table of pointers to the entire contents of the file. This table is kept in several areas for security.
As has already been said, SSDs have no rotational position latency, and suffer from no seek, move, settle time before reading. The only latency involved would be a look up in the file allocation table for the next pointer to the fragmented file, which is very minimal.
All modern FATs, no matter the type take into account a media device that is an SSD and provides a kind of rolling write mechanism that moves blocks written around to get away from the max rewrite limitation. When the disk is specified, the OS will know based on the architecture that the 'disk' is an SSD and will act accordingly. I think there are some versions of Linux which doesn't have this, but there are aftermarket utilities, and even the SSD itself may have a relocating algorithm in some cases.
The only benefit to defragging an SSD is to shrink the file allocation table, and reduce the number of linked pointers to a file, and maybe many files. Every modern FAT also has a utility that will warn the OS when the FAT is becoming saturated, and the OS will call a utility that will automatically defrag to shrink the entries in the FAT. You can see this sometimes when you are doing a lot of file manipulation like moving music or video around. These two file types do not like to have many links in them, because they are streaming file formats, and don't tolerate a lot of seeking. It's immaterial with an SSD.
Defragmenting becomes important when a computer has a limited amount of RAM/cache memory. In the case of Win7 or Vista, although I think they will run with 4Gb of RAM, it is not optimal. In this case, if you don't have an SSD, you would do well to defrag at night so that you won't do a lot of thrashing of the disk when you have to page memory in and out.
