Questionnaire on the effect of experience on risk perception

C

C_Mer

Filing Flight Plan
Joined
Jun 12, 2015
Messages
6
Display Name
Display name:
C_Mer
Dear Pilots of America members,

As part of my undergraduate studies in psychology at the University of Malta, I am conducting research on the link between experience and risk perception in pilots.

If at any point in your life you have held a student’s pilot license or higher, I would appreciate your contribution. The questionnaire may be accessed here: Closed, thank you :D

It should not take you more than 10 minutes to complete, and all responses will remain strictly confidential. Additionally, data will not register until you click submit at the very end, so feel free to withdraw at any time up to that point.

I look forward to your participation. Please do feel free to contact me if you have any questions, comments either through a reply below, or by sending me an email on cmer0018[at]um.edu.mt

Additionally, I plan to update this post with a summary of the findings a few months down the line.

Thank you for your time,
Charles Mercieca
 
Last edited:
Done. Better thought out than most the student surveys put up, but as I mentioned in the notations several of them had unlisted factors to consider that could swing a 0 to 100 or vice versa.
 
Henning said:
Done. Better thought out than most the student surveys put up, but as I mentioned in the notations several of them had unlisted factors to consider that could swing a 0 to 100 or vice versa.
Click to expand...

I feel the same, the best written one I've seen on here yet, but certainly you were lacking adequate information to really judge the risk on many of them.

Pretty sure it's the first one I finished. Usually I end up quitting about a third of the way through because of frustrations.
 
Henning said:
Done. Better thought out than most the student surveys put up, but as I mentioned in the notations several of them had unlisted factors to consider that could swing a 0 to 100 or vice versa.
Click to expand...



Indeed, well written survey.

I'd love to see the results.
 
Thank you for your participation. Unfortunately I can't take credit for the vignettes on page two, they are from David Hunter, an aviation psychologist who used to work for the FAA. If you are into human factors, you might find his website of interest: http://www.avhf.com/

I had planned to write everything myself, but alas it would have of been too great of an undertaking for an undergraduate, especially since you have to standardize them and so on.

Additionally, if I seem to abandon this thread until next week it's so as not to 'contaminate' any newcomer's responses. However I look forward to joining the discussion next week.
 
No offense, but responses are already "contaminated" by self selection.

You surely must have studied selection bias as a psychology student.
 
Done, would also like to see the results!
 
"Contamination" is not the problem introduced by self-selection. Rather, the problem is that it is not clear what population of pilots the results can be generalized to.
 
Last edited:
What I'm not sure I understand is that on top it says:

"As an anchor, you should assume that driving a car on the freeway during the day in good weather has a risk score of 50"

Given that statistically speaking, flying an airplane is more risky than driving a car on the freeway in good weather, then none of the answers can really be below 50.
 
Johann said:
What I'm not sure I understand is that on top it says:

"As an anchor, you should assume that driving a car on the freeway during the day in good weather has a risk score of 50"

Given that statistically speaking, flying an airplane is more risky than driving a car on the freeway in good weather, then none of the answers can really be below 50.
Click to expand...

True, but that's kind tempered by the statistics being compromised of all conditions. Personally I think that flying on a nice day SEL counts as 35 and MEL as a 20 in comparison to being on the highway on the basis of proximity to other similar moving masses that are beyond my control. But again, there are many other factors that can change that number significantly, especially SEL like the terrain beneath me where 35 is over moderately hostile terrain with periods that don't allow for safe glide to landing.
 
Glad you put (in years) after age. I have been telling it in metric (millimonths) for so long :)
 
I do not like that most of the questions are worded from the standpoint of a low time private pilot. The one where the guy is landing in a 15 knot crosswind - to me that wouldn't even merit a second thought, but to some people that could be a very challenging exercise. I think it would be a better survey if you were supposed to gauge the questions based on your own ability and experience level instead of a low time fair weather flyer.

But I also don't understand data collection, or psychology. That may be a no-no in the survey world?
 
Last edited:
DavidWhite said:
I do not like that most of the questions are worded from the standpoint of a low time private pilot. The one where the guy is landing in a 15 knot crosswind - to me that wouldn't even merit a second thought, but to some people that could be a very challenging exercise. I think it would be a better survey if you were supposed to gauge the questions based on your own ability and experience level instead of a low time fair weather flyer.

But I also don't understand data collection, or psychology. That may be a no-no in the survey world?
Click to expand...

I suspect the point of the excerise is to determine the difference in risk perception of the risk an inexperienced pilot faces from both the inexperienced persons perception versus those that have experience that look at an inexperienced person in those scenarios.

Its not about the risk those scenarios are for someone experienced its about the risk of someone inexperienced in those scenarios and whether or not experienced pilots consider the risk different for those people then an inexperienced.

For example, how would I perceive the risk a fresh 40 hour pilot is in versus how that 40 hour pilot feels. I suspect those experienced may feel the risk is greater than those without experience think which leads to inexperienced people getting into accidents - they don't know how dangerous their situation was.
 
Since the OP was wise and backed out while the survey is active, did anybody put down 100 for answers? I did on two of them. Also, what is a COM certificate?
 
Hi all, I went ahead and closed the survey.

Firstly, a big thank you to everyone who took the time to answer, total response was 167, which means I'm quite happy.

MAKG1 said:
No offense, but responses are already "contaminated" by self selection.

You surely must have studied selection bias as a psychology student.
Click to expand...

No offence taken :wink2:

While I agree that the best theoretical way to collect data would have of been to obtain a registry of all active airmen and randomly select from that, 1) data protection safe guards would make that take ages 2) Very few countries except the USA actually have such a centralized list. Even if I get permission, I'd need to mail large numbers of questionnaires overseas with return mail and so on.

That being said, the FAA's 2014 Active Civil Airmen Statistics puts the average pilot at 44.8 years, with 94% being men and 6% being women.

For my survey, the average age was 48.6, and 95% of the participants were men, while 5% were women.

While it's true that this forum might capture a subset of the pilot population that's perhaps more vocal and inclined to discuss aviation matters online, it's still a pretty representative snapshot I'd say.

pilot1956 said:
"Contamination" is not the problem introduced by self-selection. Rather, the problem is that it is not clear what population of pilots the results can be generalized to.
Click to expand...

The plan was to classify them by certificate level.

Johann said:
What I'm not sure I understand is that on top it says:

"As an anchor, you should assume that driving a car on the freeway during the day in good weather has a risk score of 50"

Given that statistically speaking, flying an airplane is more risky than driving a car on the freeway in good weather, then none of the answers can really be below 50.
Click to expand...

Interestingly, a couple are lower than 50, and quite a few are close.

Henning said:
True, but that's kind tempered by the statistics being compromised of all conditions. Personally I think that flying on a nice day SEL counts as 35 and MEL as a 20 in comparison to being on the highway on the basis of proximity to other similar moving masses that are beyond my control. But again, there are many other factors that can change that number significantly, especially SEL like the terrain beneath me where 35 is over moderately hostile terrain with periods that don't allow for safe glide to landing.
Click to expand...

The average of Q17, Cruising in VFR weather to a destination 1 hour away with 3 hours of fuel had an average rating of 27.

DavidWhite said:
I do not like that most of the questions are worded from the standpoint of a low time private pilot. The one where the guy is landing in a 15 knot crosswind - to me that wouldn't even merit a second thought, but to some people that could be a very challenging exercise. I think it would be a better survey if you were supposed to gauge the questions based on your own ability and experience level instead of a low time fair weather flyer.
Click to expand...

Interesting. The reason I opted for 3rd person stories was so one wouldn't feel the exercise was 'personal' and perhaps be defensive etc.

woodchucker said:
did anybody put down 100 for answers? I did on two of them. Also, what is a COM certificate?
Click to expand...

There were quite a few 100's. The highest average for all 167 responses on a question was 95, and unsurprisingly that's for the VFR into IMC while flying through a mountain pass.

Again, thank you all so much.

I'll begin more thorough analysis of everything shortly. I'll keep you informed, but I'll have to check how much of the results I can share online before I submit the dissertation.

Although if this is an issue, you have my word that I'll upload the whole thing here afterwards.

Charles
 
pilot1956 said:
"Contamination" is not the problem introduced by self-selection. Rather, the problem is that it is not clear what population of pilots the results can be generalized to.
Click to expand...

It could be generalized to pilots who self select to participate in surveys of this type.
 
Doggtyred said:
It could be generalized to pilots who self select to participate in surveys of this type.
Click to expand...

You are correct, of course. Problem is, contrary to OP's statement that the sample is "pretty representative", neither he nor anyone else knows the extent to which pilots who self-select for participation in surveys provide answers similar to those who do not participate in surveys. Nor does he know what proportion of eligible pilots actually chose to complete the survey.

This kind of convenience sample may be representative of all GA pilots, unfortunately, it is very hard to confirm that claim.
 
Last edited:
pilot1956 said:
You are correct, of course. Problem is, contrary to OP's statement that the sample is "pretty representative", neither he nor anyone else knows the extent to which pilots who self-select for participation in surveys provide answers similar to those who do not participate in surveys.

This kind of convenience sample may be representative of all GA pilots, unfortunately, it is very hard to confirm that claim.
Click to expand...

This is fair criticism. I'll be sure to discuss it as part of the limitations. Thanks.
Charles
 
Charles, if the demographics of your sample (age, gender, license type, total hours) is similar to that of the US pilot population, that observation adds support to the inference that your sample is representative of that population (it cannot confirm the inference, however).
 
Since no one can force you to participate in a survey, even if the OP had mailed it to a random selection of people off the FAA list there still would have been self-selection.
 
It's not really that important of a survey is it? The results aren't going to affect anyone if they are inaccurate, regulations aren't being rewritten due to the results. I think this method of executing a survey is going to give a result about as accurate as any other.
 
Everskyward: "Since no one can force you to participate in a survey, even if the OP had mailed it to a random selection of people off the FAA list there still would have been self-selection."

Yes, but if the OP had mailed it to a random selection of people off the FAA list, he would have known what proportion of those pilot had responded. The "response rate" is an indicator of degradation of sample fidelity. Specifically, a high response rate tends to assure representativeness of the study sample. A low response rate raises the possibility of un-representativeness (with no way of knowing the extent to which it was not representative).

Henning: "I think this method of executing a survey is going to give a result about as accurate as any other." There is a large body of evidence on this topic, it is well-established, and is says that your opinion is incorrect. You are entitled to your own opinion, however, you are not entitled to your own facts about survey research methods.
 
pilot1956 said:
Everskyward: "Since no one can force you to participate in a survey, even if the OP had mailed it to a random selection of people off the FAA list there still would have been self-selection."

Yes, but if the OP had mailed it to a random selection of people off the FAA list, he would have known what proportion of those pilot had responded. The "response rate" is an indicator of degradation of sample fidelity. Specifically, a high response rate tends to assure representativeness of the study sample. A low response rate raises the possibility of un-representativeness (with no way of knowing the extent to which it was not representative).

Henning: "I think this method of executing a survey is going to give a result about as accurate as any other." There is a large body of evidence on this topic, it is well-established, and is says that your opinion is incorrect. You are entitled to your own opinion, however, you are not entitled to your own facts about survey research methods.
Click to expand...

Surveys of this order are in and of themselves meaningless, as are the results. The accuracy of any test that requires a response based on perception is already prone to inaccuracy due to requiring a considered response of perception. I recently observed an eye exam where the doctor used a light and lenses and never once asked the patient the typical "Is this better, or this?". I asked him about it and he said that the non responding method produced a better result than relying on perceived result answers. Just introducing perception introduces inaccuracy. Since this survey is all about perception, the factual accuracy of the survey regardless of how it is formed will always be inaccurate.
 
Henning, not worth discussing this further with you. Self-report is valid in many settings. Your personal interaction with one doctor doing one eye examination is not a substitute for actual knowledge or experience.

Further, you have changed the topic in order to avoid the error of your previous point. The issue is the validity of the sampling strategy, nothing more, nothing less. The OP gets it and you do not.
 
Hello everyone!

At last I’m able to post a summary of my research/findings. I tried using an inverted pyramid format and include simple explanations for those that don’t want all the details as well as some numbers for those who love statistics, however if you want a more/less technical post to clarify some things, just let me know.

Main Hypotheses

The main purpose of the study was to test whether experience (measured in flight time) had any effect on risk perception. To do this a correlation test was used (Spearman's).

Total flight time’s effect on the mean risk perception score of all 17 items had a correlation score of -.258. It’s a relatively weak relationship as far as correlations go, but simply put the more hours a pilot has, the lower he rates risk. The result was statistically significant at the p<0.01 level, which means the odds of this result occurring purely do to chance are very remote.

Recency was also explored by using the flight time in the last 30 and 90 days. For the last 30 days none of the results were statistically significant, I suspect partially due to a smaller sample compared to the other two (quite a number of pilots had not flown at all in the past 30 days).

The relationship between flight time in the last 90 days and mean risk perception was however statistically significant (p=0.033), and the correlation was similarly negative but weaker than for total flight time (-0.165).
All of that means that risk perception isn’t something that varies very remarkably with experience, and, from my research at least, it tends to go down even if ever so slightly as a pilot flies more.

However, when one examines FAA/NTSB data for fatal accidents, there does seem to be proof more experienced pilots crash less. The most ‘popular’ book along these lines is Paul Craig’s ‘Killing Zone’, but recently William Knecht modified a bit the approach by using accident rates rather than frequencies. His paper is here: https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/2010s/media/201503.pdf but more significantly, he has an updated ‘camel hump’ curve on page 16 which illustrates that the ‘killing zone’ is larger than previously assumed.

Age, Main Type of Flying activity, and Personality traits

A number of other collected demographic factors were also tested. Age did not have any significant effect, and neither did main type of flying activity. Since there was some data linking automobile accidents to Extraversion and Openness to Experience, (the more extraverted and open to experience a driver is, the more he is involved in fatal accidents), I thought it would be interesting to also include a short personality test to measure these two scales, however the results also turned out to be statistically insignificant (more extraverted individuals didn’t have significantly higher risk perception scores for instance).

Highest Licence held

One group where there was some statistical significance was by highest licence/rating held. The One-Way ANOVA score for the effect of certification level on the mean risk perception score was F(6,160)= 3.032, p=.008.

cert_level.png


The major limitation of this graph is that some groups have rather small numbers, so generalizations have to be made with caution.

Additionally, I noted all your comments about the scale’s difficulty (the anchor set too high, hard time in rating on a 1-100 scale). One of this study’s recommendations is a new scale since this would help validate this scale and the whole risk perception construct in general. The tricky bit is it’s very hard to measure perception using means other than a self-report inventory.

Super Summary

Regardless, if I can sum it up in one sentence, it would be this: Experience does seem to be a protective factor against accidents, but why this is so, and the exact mechanism through which it works to do so is: (i) very unclear and (ii) probably not straightforward.

If you have any questions/comments please let me know, and additionally, I plan to post the whole PDF here when it becomes publically available on my University’s website.

Additional Info:
The questions ranked from most dangerous to least: https://www.dropbox.com/s/8uyst8kf6ul4umk/Questions%20ranked.docx?dl=0
Craig's book: http://www.amazon.com/Killing-Zone-...ng+Zone,+Second+Edition:+How+&+Why+Pilots+Die

-Charles
 
Henning said:
Interesting that SP has the highest perception of risk.
Click to expand...

That is interesting. I imagine the majority of SPs flying generally well equipped LSAs in good weather on usually short flights...

Anyway thanks for posting Charles.
 
oregonboy109 said:
That is interesting. I imagine the majority of SPs flying generally well equipped LSAs in good weather on usually short flights...

Anyway thanks for posting Charles.
Click to expand...

Fear, and perception of risk, are typically inverse to knowledge. Many of the given situations are not part of the Sport Pilot curriculum beyond "Avoid" and "Don't do this".
 
Which, IMO, is a problem since many, potentially dangerous, situations are possible for an SP to get into and then have to get out of with little to no training on the subject like you mention.
 
oregonboy109 said:
Which, IMO, is a problem since many, potentially dangerous, situations are possible for an SP to get into and then have to get out of with little to no training on the subject like you mention.
Click to expand...

Yep, this is why I say "If you don't have a medical issue which prevents you from getting a medical, SP is a poor choice." The people who say "It's easier to get" are the people who are costing themselves that knowledge.
 
You must log in or register to reply here.
Back
Top