Bottom Line -->
What ever you decide to do is up to you
Do NOT, however, decide that YOUR definition of safety must be imposed on everyone else just to make YOU feel good about your ideas
Truer words were never spoken. If only we could extend this attitude throughout our society! Perhaps then we could go back to the days when manufacturers weren't forced to put goofy warning labels on everything from plastic bags to ladders.
As to whether "risk" in aviation can be quantified, I think there are simply too many variables. With literally hundreds of parts, dozens of actions, and weather conditions all coming into play on every flight, "risk" varies every time we fly.
IMHO, this is one of the main reasons so few people choose to fly light planes. Even if you don't consciously think these thoughts, there are always things that can keep you awake at night: Did your A&P reassemble the trim assembly correctly at the last annual, or is the retaining nut loose? Did that battery you dropped work it's way into the flight controls, and they're about to jam?
There are a jillion of these thoughts that can give you analysis paralysis, virtually preventing flight. So, in that regard, every flight is a leap of faith.
The best we can do is to eliminate stupid stuff. I would list these as:
1. Running out of gas
2. Flying with known mechanical problems
3. Flying into bad weather.
4. Flying when physically unable.
After those Top Four problems, I would rank
5. Exceeding design limitations of your aircraft
6. Low level aerobatics, including buzzing.
And, after that I would rate
7. Not following readily accepted procedures in the airport vicinity.
8. Flying too seldom to remain proficient.
Beyond eliminating these eight pretty obvious things, really, what can we do to enhance "safety"? Can you add any others?