I've flown with and without the autopilot in my airplane, both in VMC and IMC and I find myself more in touch with what's going on when I'm hand flying. As a result, I switch on the autopilot once every 3-4 flights for a few minutes to make sure it works and that's about it.
About the only time hand flying gets really taxing is when the ride is really rough, but at that point, the A/P may well throw in the towel in any case.
I recently took a 3.2 hour trip of which ~ 2.7 hours was in IMC. It was a thick stratus deck, almost continuous precip, mostly smooth but occasional light to moderate turb. It was nice when it was all over, but it certainly wasn't death-defying and didn't require particularly good flying skills other than unwavering confidence in what the instruments are telling ya.
The stick and rudder part of it is pretty much just like flying VFR quite honestly. If the plane is well trimmed, there isn't much to do. And I say all this about an airplane that has a reputation for being "hard to fly" and a "terrible IFR platform because it's so touchy." It isn't...and, it isn't.
Go for it without the a/p and build your confidence in the your ability to efficiently fly/manage your airplane. There are few things more satisfying.