I do tend to put my fake flying into two different categories. One is when I load something up and mess around, ie take the F18 and somehow go Mach 3 or the BF109 and fly that around. That's more of the video gaming thing and I bet a VR headset would be pretty cool assuming the graphics, frame rates ect are improved.
The other is more of the simulator thing where I'll load up an IFR flight and fly it complete with using my EFB, ect. I find that extremely useful to real work flying since out of 400 hours I only have about 20 real IMC. Not sure how you would do that with a headset unless you somehow write clearances or can pull readable charts up in the game and not take the headset off.
What I don't do typically is use checklists, taxi, preflights or any W&B stuff. When I think immersion I typically think of people doing those things.
I also replaced my yoke and throttle controls with a HOTAS joystick for the reasons you said above. It's not 'realistic' for me but it's much easier to map things to the joystick then try and mess with the keyboard or turn dials with the mouse as you're flying.
So, after a few weeks of messing around in VR (roughly 10 hours or so), I've come to realize your point. X-plane still hasn't come out with native VR support (yet) and the 3rd party plugin (FlyInside) performs too poorly for me to be acceptable, (and that's on my high-end PC, overclocked to 5ghz)!
Even when XP11 becomes VR complete, I doubt I will be able to achieve complete sim flights, without using my hardware switches, panels, efb, etc. that I've setup over the years. Probably at some point, but not for awhile.
That said, I'm having a blast in DCS World! The A-10C and Huey are incredible and I'm getting very smooth, good performance overall. The ability to put the helicopter down exactly where you want to, is so much easier in VR. Hovering, landing on rooftops, flying into open hangars and making a 180, etc. Firing rockets/machine guns and blowing **** up with the Huey gunship, while your door-gunners engage, is incredibly fun

. I need to fly this thing a little better so they survive longer under fire. Door gunners keep getting shot by ground fire!
Another bonus of the rift (which I was unaware of at first) is the built-in mic. There's a nice little program called Voice Attack, which allows you to give verbal commands to your wingmen, contact ATC, control aircraft systems, etc.
So, overall, I am very happy with my purchase of the Oculus Rift CV1. And I haven't even tried any other VR titles yet.
Oh yeah, VR sickness is a real phenomenon, lol.
First flight, no problem. Second time, took on too much and started feeling a little dizzy and weird

. Took a break and came back a couple of days later. Played short sessions (no more than 20 mins at a time). After a few flights, I had my "VR legs".
Lesson: Take it easy at first!