I think there are two issues here. For a potential passenger who doesn't know anything about flying it's all about perception. Passengers may have a visceral fear of flying, they may be afraid of the unknown or they may be the type who are afraid of new things. Something else that may come into play is that they look at you, the pilot, their friend and peer, as having generally the same abilities as they do. Since they can't imagine themselves being able to fly an airplane they don't quite understand how you are able to do it successfully. Besides, they may have seen you drive.
As a pilot I don't think you can ignore the statistics. As someone else said, they are only a reflection of how we are doing as a group. Of course there are ways you can make your own odds better, but there is a danger in thinking that you are any better than average... because everyone thinks that! There is also a danger in thinking that you can control all factors and that you are sure you will react correctly in any situation because that has been proven time and again to be false, even with experienced pilots.