FAA Exam Software

P

Pilot99B

Pre-Flight
Joined
Jul 2, 2022
Messages
42
Display Name
Pilot99B
Honest question here.

How is FAA written exam software such as Sheppard Air not considered cheating by the FAA, and enforcement action taken?
They literally solicit pilots to remember exam questions, and provide them to them.

If this occurred in Academia it would be considered cheating and punished accordingly.
 
Pilot99B said:
Honest question here.

How is FAA written exam software such as Sheppard Air not considered cheating by the FAA, and enforcement action taken?
They literally solicit pilots to remember exam questions, and provide them to them.

If this occurred in Academia it would be considered cheating and punished accordingly.
Click to expand...
Welcome to “not academia”
 
Been pretty standard in a lot of areas for a long time. I've been in I.T. for 30 years and pretty much every certification test in I.T. works the same way. I just hope it's not the same for continuing education for surgeons.
 
strikes2k said:
I just hope it's not the same for continuing education for surgeons.
Click to expand...
Well..... In the way back time... later 1980's... a group of surgeons decided to memorize a certain numerical range of questions for our national specialty board exam (ie- Joe takes question 1-5, Jim, 6-10, Sally, 11-30 note- women never seem to forget anything so she gets more. LOL) This compilation of questions had been circulating amongst residents in my particular specialty as it was widely disseminated. The powers to be at the board were a bit miffed but little they could do about it. I had a copy which I used in study prep to get a better idea of the layout of expected material that was emphasized. It might have helped a little but nothing like Sheppard for IR written. In my case, passing the written just opened the doorway for a two day oral exam. If you limped your way through the written by studying old test questions, your @zz was to be handed to you during the oral exam. Good thing the bar in the Drake Hotel in Chicago, where the exam was held, had a great blues group appearing nightly and a great bourbon and scotch selection.....it was needed after day two!
 
It’s been going on for years,it’s a standardized test,with a few different models. The safety is on the oral if you just memorized the questions the examiner can pick it up on the oral.
 
  • Like
Reactions: SLB
frfly172 said:
It’s been going on for years,it’s a standardized test,with a few different models. The safety is on the oral if you just memorized the questions the examiner can pick it up on the oral.
Click to expand...
If you use Sheppard to memorize the answers, and actually take the time to work out why those answers are correct, it's harder to get surprised on the oral.
 
The FAA has been working on this problem of test question/answer memorization for several years. They now update the questions databases every quarter, and will soon be including dynamically-generated questions (they call this AIG - Automatic Item Generation). Their stated goal is to thwart question harvesting by test-prep providers and minimizing question/answer memorization as a useful study strategy.

1748015128497.png
 
write-stuff said:
The FAA has been working on this problem of test question/answer memorization for several years. They now update the questions databases every quarter, and will soon be including dynamically-generated questions (they call this AIG - Automatic Item Generation). Their stated goal is to thwart question harvesting by test-prep providers and minimizing question/answer memorization as a useful study strategy.
Click to expand...

I’m all for AIG; part of my team is responsible for our product and process knowledge question banks.

Rote memorization can be hindered by dynamic question generation and routine updates to question wording and answer choices. What we find difficult is there’s only so many ways to ask a question, especially if the learning or performance objective(s) for a topic are at the bottom of Bloom’s taxonomy.

We’re experimenting with some dynamic scenario based activity assessments and are finding our question banks need to be about 4x larger than they are now to account for the (fixed) audience size moving thru our training programs on an annual basis.

We don’t have the problem of our trainees having relatively unconstrained study time though and that, to me, is the biggest challenge the FAA has to overcome with their written exams.
 
There's nothing wrong with rote memorization, under the right circumstances. Think of things like regulations. About the only way to learn them is rote memorization. The problem is that some people expand the concept of rote memorization directly to memorizing the test questions/answers.
 
The fundamental problem with the FAA’s written exams is that someone thinks they need to be hard questions or that it should be difficult for the test taker to discern what is being asked. These are all straightforward subjects where it’s easy to to pick out what is important and what is not. I don’t care if test takers are memorizing the written test questions. I care if they can apply their academic knowledge to the practical tasks of piloting. You could eliminate the written test and just have a standardized oral exam as part of the checkride instead.

The real question is what purpose is the test attempting to serve and is it succeeding or failing?
 
YakDrvr said:
The fundamental problem with the FAA’s written exams is that someone thinks they need to be hard questions or that it should be difficult for the test taker to discern what is being asked. These are all straightforward subjects where it’s easy to to pick out what is important and what is not. I don’t care if test takers are memorizing the written test questions. I care if they can apply their academic knowledge to the practical tasks of piloting. You could eliminate the written test and just have a standardized oral exam as part of the checkride instead.

The real question is what purpose is the test attempting to serve and is it succeeding or failing?
Click to expand...
exactly the Knowledge test is just a stepping stone to the oral exam. No point in even starting the or even prepping for the Oral exam if you can't pass the Knowledge test. So use the knowledge test to get you up to a minimum standard and then start building on that rote knowledge.

like mentioned it is possible to study in such a way this doesn't help but few people can/will actually do that and it will be identified during or prior to the Oral exam.

Brian
CFIIG/ASEL
 
frfly172 said:
It’s been going on for years,it’s a standardized test,with a few different models. The safety is on the oral if you just memorized the questions the examiner can pick it up on the oral.
Click to expand...
I don't see how this is realistic. The questions a DPE asks are from the ACS and also computer selected from your test score/areas missed. So you already have the questions to basically an open book test. If you missed all the 'systems' category questions then you can 100% expect to get some systems questions. I scored a 97, so basically my oral was standard ACS topics.

The PPL test is basically a 'scenario' where the same questions are asked (as per the ACS) of everyone. The DPE has discretion, but unless you bugger up the basics, any minutia can easily be cleared up with 'let me check the FAR/AIM to be sure'.

I get that a lot of people buckle under pressure... and you are paying a lot of money... but its an open book test and you are given the questions in advance.
 
brcase said:
So use the knowledge test to get you up to a minimum standard and then start building on that rote knowledge.
Click to expand...
I don't agree. The knowledge test is basically a shotgun approach. Most of the performance chart questions are in a format I've never seen (the draw a line left to right hitting the values on column). What GA AC is that from??

Also there are VOR questons but nothing about GPS. The written doesn't cover important details like logbook entries (both pilot and maintenance). Most CFIs never even show the MX logbook until right before the practical test. Way to make safe pilots.

The knowledge test IS rote knowlegde. Study using an app until you get consistant 95's then take the test. Prep for the oral. The oral will be all about the AC you fly, and a trip YOU plan.

THe FAA should get on the AI bandwagon and feed the LLM FAR 61 & 91 and have it ask and grade questions.
 
write-stuff said:
Without the knowledge test first, can you imagine how long an oral exam would take to cover everything in the ACS? It would be an all-day event.
Click to expand...
The written is laughable. Most of the information 'learned' from it is done automatically in an EFB when you make a flight plan.

AI should 100% be used for the knowledge exam.

I've taken 2-4 hours tests for job interviews.... its not like its unheard of.
 
It wasn't all that long ago the FAA published the questions and answers, it was that way when I got my PPL.

Someone at the FAA finally realized people were memorizing the Q&A, so they decided to mix things up, but randomizing the order of the answers.

Honestly I don't feel bad about it one bit. The FAA written test are very poorly written. It doesn't matter how well you study the material you are going to mess up a question because many have two "correct" answers, but only one gets you the point. Many of the questions are severely out of date and out of touch with modern flying as well.
 
MauleSkinner said:
For example?
Click to expand...
Any question from the ATP written. Most worthless test ever written, never mind the ridiculous ATP/CTP course I had to fork over $5,000 to pay for the privilege of being eligible to take the worthless test. I’ve never had to prep for a test with so many poorly written or irrelevant questions before (the commercial was just as bad, but fewer total questions). You can’t get your ATP without taking a check ride. Maybe ATP checks in a piston twin are a joke (I doubt it), but a lack of knowledge will rapidly make itself evident during the oral and flight phases of a type ride.
 
YakDrvr said:
Any question from the ATP written. Most worthless test ever written, never mind the ridiculous ATP/CTP course I had to fork over $5,000 to pay for the privilege of being eligible to take the worthless test. I’ve never had to prep for a test with so many poorly written or irrelevant questions before (the commercial was just as bad, but fewer total questions). You can’t get your ATP without taking a check ride. Maybe ATP checks in a piston twin are a joke (I doubt it), but a lack of knowledge will rapidly make itself evident during the oral and flight phases of a type ride.
Click to expand...
Ok…so pick a question that indicates “The fundamental problem with the FAA’s written exams is that someone thinks they need to be hard questions or that it should be difficult for the test taker to discern what is being asked.” I’m curious how you discern that thought process.
 
Pilot99B said:
Honest question here.

How is FAA written exam software such as Sheppard Air not considered cheating by the FAA, and enforcement action taken?
They literally solicit pilots to remember exam questions, and provide them to them.

If this occurred in Academia it would be considered cheating and punished accordingly.
Click to expand...
You think that's wild, up until a few years ago, the FAA published all the questions and answers.
 
write-stuff said:
There's nothing wrong with rote memorization, under the right circumstances. Think of things like regulations. About the only way to learn them is rote memorization. The problem is that some people expand the concept of rote memorization directly to memorizing the test questions/answers.
Click to expand...

I’m not entirely convinced. I think regulations are best understood when you reason through why they exist. For example, why an airplane must give way to a balloon, but not the other way around, or why VFR requires 1,000 feet above and 500 feet below clouds in controlled airspace, but just "clear of clouds" in Class B. However, you’re right—there are some things you just have to memorize because they don’t always seem to follow a clear logic, at least not to me.
 
Multiple choice tests are not a very good way to incentivize or ensure deep understanding, or that the student remembers the material for very long. They are relatively easy to administer though. Obviously a thorough interview with a skilled examiner and unlimited time would be preferable, but not practical. The once-and-done testing model also encourages cramming and poor learning habits.

I guess the question for the grey-hairs here is "if you pass your oral/checkride with flying colors panache and exemplary competence, does it matter if you scored zero on the written?"

As it stands I think the written is designed to make the checkride shorter and more manageable, not to ensure a true "if you don't know this stuff you do not belong in the air/clouds/for hire worlds"
 
Hengelo said:
Multiple choice tests are not a very good way to incentivize or ensure deep understanding, or that the student remembers the material for very long. They are relatively easy to administer though. Obviously a thorough interview with a skilled examiner and unlimited time would be preferable, but not practical. The once-and-done testing model also encourages cramming and poor learning habits.

I guess the question for the grey-hairs here is "if you pass your oral/checkride with flying colors panache and exemplary competence, does it matter if you scored zero on the written?"

As it stands I think the written is designed to make the checkride shorter and more manageable, not to ensure a true "if you don't know this stuff you do not belong in the air/clouds/for hire worlds"
Click to expand...
Apparently a practical test is not a very good way to incentivize or ensure deep understanding, or that the student remembers the material for very long.
 
I'll reply to the honest question with an honest question - why should anyone care any? It's just a requirement in the process to prove minimal knowledge so that you don't waste time with an actual examiner, who's going to quickly sort out if you: a) know enough to safely fly and b) have the skills to safely fly.

Making the process more complicated than it already is, in my view, is just wasting people's time. There's no harm in people memorizing answers to cloud clearance, signage types, airspace types, etc, where you're really talking about memorizing things anyway.

As far as using AI to write test questions, whomever came up with that idea shouldn't be working with technology. To have a remote chance of making that fair someone is going to have to record every BS question that it comes up with, for every student, and provide an actual human being to score them if they fail. And if they fail, they absolutely should push for that, because it's AI is just a random text generator for all intents and purposed.
 
I think the process sucks but it’s related to how the tests are conducted. I hated the questions that had an answer that was less than 5 degrees off between two answers, considering you were using a transparent over lay on the testing supplement and a wide line from a grease pen or dry erase marker to try to be precise on. Seemed like an uphill battle on those questions to get the right one and frustrates the hell out of the person taking the test.

Using a study guide, or using prep software, isn’t cheating.
 
Sierra1 said:
I think the process sucks but it’s related to how the tests are conducted. I hated the questions that had an answer that was less than 5 degrees off between two answers, considering you were using a transparent over lay on the testing supplement and a wide line from a grease pen or dry erase marker to try to be precise on. Seemed like an uphill battle on those questions to get the right one and frustrates the hell out of the person taking the test.

Using a study guide, or using prep software, isn’t cheating.
Click to expand...
But is it perpetuating the problem?
 
I'll reply to the honest question with an honest question - why should anyone care? It's just a requirement in the process to prove minimal knowledge so that you don't waste the time of actual examiner, who's going to quickly sort out if you: a) know enough to safely fly and b) have the skills to safely fly.

Making the process more complicated than it already is, in my view, is just wasting more people's time. There's no harm in people memorizing answers to cloud clearance, signage types, airspace types, etc, because you're really talking about memorizing things anyway.

As far as using AI to write test questions, that's an example of a solution looking for a problem, or a vendor looking for a handout. To have a remote chance of making that fair someone is going to have to record every BS question that it comes up with, for every student, and provide an actual human being to score them if they fail.

Or to put it another way, I'm pretty sure that nobody ever crashed an airliner because they didn't know the answer to a question on the knowledge test, and instead memorized it as "answer C". If someone makes a mistake at that level, it's an issue with judgement/attitude, skill, or knowledge specific to the systems they're working with.
 
MauleSkinner said:
But is it perpetuating the problem?
Click to expand...

I think it depends on your viewpoint. It’s hard to learn it the “right” way when people feel the FAA stacks cruddy testing processes against you like above. Learning the methodology and material is always the best option and makes you better. But when there is a feeling of the FAA putting really close precise answers on charts trying get ya, while giving you imprecise methods to utilize the charts in the testing process I think people go for the sure thing of rote memorization to get by and get the 70%. If you don’t understand the material, the score doesn’t matter really as you just regurgitated things you don’t understand.

I liken it to HAM radio. Straight rote memorization is the most used option with known test questions from a pool. The saying is “get the license then learn the stuff you need to know with where you go in HAM”. The stakes of not knowing the material are low in reality. I don’t think such is a good thing with pilot licenses at all with higher stakes that matter far more.
 
Sierra1 said:
I think it depends on your viewpoint.
Click to expand...
I think it depends on whether your viewpoint is on the side of fixing the problem or perpetuating it.
Sierra1 said:
It’s hard to learn it the “right” way when people feel the FAA stacks cruddy testing processes against you like above. Learning the methodology and material is always the best option and makes you better. But when there is a feeling of the FAA putting really close precise answers on charts trying get ya, while giving you imprecise methods to utilize the charts in the testing process I think people go for the sure thing of rote memorization to get by and get the 70%. If you don’t understand the material, the score doesn’t matter really as you just regurgitated things you don’t understand.
Click to expand...
If you got all of the “imprecise” questions wrong, you’d probably still be well above a 90% score on the written, so there’s really no need to get those questions correct. The problem is that one of the primary ways to identify and resolve bad questions is by identifying questions that a lot of people get wrong. If 50% of testers get a question wrong, it’s likely a bad question. If less that 10% of testers get a question wrong, it’s within the realm of possibility that it’s a good question that should remain, as there’s a knowledge deficiency.

Much like scheduling a DPE before the 8710 is signed, it’s a short term solution that perpetuates the problem.
 
The knowledge test is just a screen and the FAA has never had a particularly good one. It's full of junk that isn't the least bit relevant, some of which is just flat wrong or a figment of the FAA's imagination, so you just have to learn what they want to hear. Shepherd or Irwin Gleim or any of the ones who specialize in greasing you through not only is legitimate, it's often essential.
 
Standardized multiple choice tests have always(?) had prep courses which do their best to mimic the questions. I used to see the books when I worked in a public library when I was in high school in the 1960's and I'm pretty sure I saw people using them as soon as I was old enough to use the adult reference section. The FAA has done a few things to limit the value of memorization, but there's really just so much that can be done.

Besides, as @flyingron indicates, it's at best a gatekeeper. Basically (and those with relevant standard test learning issues aside), if you can't get at least a 70 on a test you study for by reading the questions and answers in advance, you should be kept away from anything with moving parts, let alone an airplane. The real "knowledge test" is the practical test.
 
MauleSkinner said:
Ok…so pick a question that indicates “The fundamental problem with the FAA’s written exams is that someone thinks they need to be hard questions or that it should be difficult for the test taker to discern what is being asked.” I’m curious how you discern that thought process.
Click to expand...
It’s been a long time since I took that test so I’m left with my impressions of the questions and how I studied. Long story short I felt like the ATP/CTP course was an exercise in enriching the sim facilities that offered it. The test itself was an exercise in FAR trivia and badly designed performance questions. Sure, you don’t comply with the FARs you will have a short career and if you blow off performance data you will eventually die, but the test questions as written seem optimized to drive business to test prep services. Why would I learn how to use charts different from any plane I’d ever or will ever fly? I came into the process with thousands of hours flying heavy jets all over the world and a fair amount of experience administering both checkrides and training for a large government agency. The FAA’s process is abysmal.
 
Hengelo said:
I guess the question for the grey-hairs here is "if you pass your oral/checkride with flying colors panache and exemplary competence, does it matter if you scored zero on the written?"
Click to expand...
Well, you won't get to the checkride if you score zero on the knowledge test. But while the likelihood is very good you won't score zero on the knowledge test if you can pass the oral on the checkride, I think there are enough questions on the knowledge test that have little practical relevance that it is possible to score low on the knowledge test and do very well on the practical unless you meet up with a DPE more interested in theory than practical application. It might be nice to know that at least 4 satellites are required for 3D GPS position, but I don't see the practical use. Your GPS annunciates "No GPS Position." First of all, it might be something else like a loose antenna, but assuming you actually go to the page that shows it and find that it not picking up one, exactly what are you going to do about it?
1748278373013.png

It's been ages since I've looked at sample private knowledge test questions, but you can see multiple examples on the instrument side. My all-time favorite, which I think is no longer there, as the VOR distance questions which come down to. "if you violate your IFR clearance by turning off course, you will be able to calculate that if you didn't, you'd be home by now." A current long-time favorite are the VOR position questions which assume you were beamed off the Enterprise into the cockpit rather than having navigated for the past 2 hours.
 
Last edited:
midwestpa24 said:
The FAA written test are very poorly written
Click to expand...
I am not entirely convinced that this isn't intentional.
if you look at the knowledge test as a learning tool vs a test it makes more sense.

When they ask a question where the answer isn't obviously correct it should force the student to learn why one answer is better than the other i.e. actually research and understand the topic better. I am know this doesn't always happen as often these kinds of questions the correct answer just gets memorized. But it at least gives the student the opportunity to learn the subject a bit better.

Brian
CFIIG/ASEL
 
brcase said:
I am not entirely convinced that this isn't intentional.
if you look at the knowledge test as a learning tool vs a test it makes more sense.

When they ask a question where the answer isn't obviously correct it should force the student to learn why one answer is better than the other i.e. actually research and understand the topic better. I am know this doesn't always happen as often these kinds of questions the correct answer just gets memorized. But it at least gives the student the opportunity to learn the subject a bit better.

Brian
CFIIG/ASEL
Click to expand...
But in some cases applicants make it more difficult…I some of the examples we’ve seen in various threads here, the verbiage is straight out of an FAA publication, but the question MUST be about something else.
 
You must log in or register to reply here.
Back
Top