When I visit my folks (8500') from where I live (1000') it always takes me a few days to get over the headaches. The dry skin and sinuses also take a while to get used to, but when I go from 60%-70% RH to about 10%, it's going to take it's toll.
Every year someone has to be carried off Pike's Peak (14k'). As I posted earlier, my nephew grew up, lived, and worked almost all his life at >8000', included a few years in Leadville (10k'). Then one day while snowboarding, something he's done since he could walk, he got hit with pulmonary edema bad enough to be transported down to Denver. But even where my folks live, there are people that need to be taken down the hill, and some of those folks are long time residents. Sometimes it just happens.