I dunno if this is universal but from the small sample of acro pilots I've met there seems to be a disproportionate amount of jerks.
I haven't noticed that and I'd bet I know just a few more acro pilots than you. You can remove "acro" and insert warbird, RV, Cirrus, jet, etc. and find others who would think the same thing. You'll find a small percentage of jerks no matter what activity in life you're into. Pilots are all pretty much the same in my experience. Anyone who thinks one group is special is probably basing this on isolated anecdotes, hearsay, or other biases...who knows. Humans are an emotional and judgmental bunch.
Regarding why more pilots aren't into acro? Lots of reasons. Fear and lack of practicality pretty much eliminates 80-90% of the pilot population right away. Look at the average GA ramp and what's in the hangars. It's obvious most pilots place a high value on practicality, even if it's only perceived for lots of folks. Plenty of pilots can't afford two airplanes. And when a significant portion of the pilot population is uncomfortable with stalls and slips, and terrified of spins, fear is a real obstacle.
There are lots of misconceptions about safety, discomfort, and just generally what's it's all about. Unless you're a newbie, it's not even about adrenaline. Heck most non-acro folks seem to think all competition aerobatic sequences look like Sean Tucker's airshow routine. People seem to have this image of aerobatics as some sort of extreme sport that beats up your body, and requires the skills and fitness of a 20-something superhuman. You should see the gray hairs and beer bellies at aerobatic contests.
Acro doesn't "hurt" unless you are pushing a lot of negative G without building a tolerance for it. 99% of recreational aerobatic pilots don't even push any negative G. I've never gotten bruises - even at +9/-6G. It's all about acclimation and tolerance - even with light positive G, which many find uncomfortable at first. At first, lots of folks feel mild discomfort and wide-ranging nausea, but don't seem to pursue it long enough to get over it through exposure - which (getting over) WILL happen for most folks.
Regarding safety, I'd guess greater than 95% of aerobatic fatalities are attributable to training and judgment issues - not too much different from flying in general. Good equipment, judgment, and training is extremely likely to cause you to lead a long life and ultimately die of non-acro related causes.
Lack of exposure and insurance issues have already been mentioned. There are very few flight schools that even have tailwheel airplanes available, much less aerobatic airplanes. In the old days the modest Citabria was a common primary trainer, which also provided some acro intro for lots of student pilots.
There is cost, considering training, rental, or additional aircraft purchase prices. But it all comes down to priorities. Those who truly have the interest will find a way to do it.