Madison, Indiana is a really cool town with over 100 blocks on the national historic register. It's a beautiful old all brick town on the Ohio River. Lots of shops, lots of restaurants and lots of B&Bs.
The downside:
The airport is a few miles out of town and they have no courtesy cars so you have to rent a car just to have lunch or to spend the afternoon.
It's a great town in which to spend an entire afternoon...or more.
Also, for future planning, Lee Bottom is pretty close to you too and just south of Madison. I think they've adequately recovered from their tornado and will probably have sinful Sundays again this year. Sinful Sunday is the second Sunday of each month in June, July & August. Basically, it's a big ice cream social and party.
SSSS = Sinful Sunday is Second Sunday.
I've been to Madison a few times and the last time we went, we stayed at a B&B for two nights and then dropped by Lee Bottom for Sinful Sunday on the way home. It was a great weekend. Highly recommended.
As an aside:
I just got off the phone with the mayor's office and the chamber of commerce both informing them that they're really missing the boat. I recommended that they establish a shuttle service to/from the town & airport at least from noon Friday through the end of the day on Sunday. I don't think a courtesy car or two would work well for them because of the typical time limit.