Knowledge-test revisions surprise, scores suffer

AuntPeggy

AuntPeggy

Final Approach
PoA Supporter
Joined
May 23, 2006
Messages
8,479
Location
Oklahoma
Display Name
Display name:
Namaste
http://www.aopa.org/advocacy/articl...ge_test_revisions_surprise_scores_suffer.html
AOPA has engaged the FAA in response to reports that unannounced, significant changes in the question banks of at least three airman knowledge tests have dramatically increased failure rates.

Question banks affected by the changes include the Fundamentals of Instruction (FOI), Airline Transport Pilot (ATP), and Flight Engineer (FE) knowledge test. Test providers, universities and flight schools have reported significant increases in the number of failures since the changes were implemented. More than half of those taking the FOI knowledge test since the changes were implemented failed the exam.

There are also reports from flight training programs that previously correlated scores between practice knowledge tests and actual tests for the instrument rating have begun to diverge.
Click to expand...
 
If the test questions (and answers) were well written, then a candidate who truly understood the material would be relatively unaffected by a new bank of questions.

But I think the problem here is that various "users" are concerned about the validity of the exams - that it's actually measuring what it's supposed to measure.

I know that I got marked wrong on a question on my FIA airplane exam that I answered correctly. On other wrong answers I could see my mistake or where I "fell" for an "almost right" answer, but this one was clearly not programmed properly into the exam.
 
When I was at the Academy last Spring they came in during one class and discussed the knowledge base of test questions. We were told they would begin a major rewrite of the questions and stop providing them to the public.

Looks like simply memorizing the answers won't work anymore.

post from May 21, 2010:

Rotor&Wing said:
The FAA has stopped making the knowledge test questions available to the public with the exception of a few "sample questions" on the faa.gov website http://www.faa.gov/training_testing/testing/

The test guides out now have a few of the actual questions but that's it. Eventually they will have none of the actual questions.

The days of memorizing the answers are dwindling away. :rolleyes:
Click to expand...
 
Last edited:
IIRC FAA was originally forced to make the test questions public by the courts (in the late 70's?). While I applaud this new policy, I wonder what has changed, legally, to allow its implementation.

I took most of my writtens using ACME (remember them?) sample questions as an outline for good old-fashioned study. The technique works fine. :)
 
Last edited by a moderator:
rotorhead1026 said:
IIRC FAA was originally forced to make the test questions public by the courts (in the late 70's?). While I applaud this new policy, I wonder what has changed, legally, to allow its implementation.
Click to expand...

I seem to recall the same thing. I may be wrong, but I believe other agencies like the FCC that provide radio license tests must also publish their test questions and possible answer pool.

IF their tests were rid of the trick questions and ones irrelevant to safe flight, I would not mind not having them not available. But they have a lot of things that distract, such as:
The definition of nighttime is:

A. sunset to sunrise
B. 1 hour after sunset to 1 hour before sunrise.
C. the time between the end of evening civil twilight and the beginning of morning civil twilight.
​
(Each possible answer actually is meaningful in the regulations!)

Am I safety hazard if I forget these differences?

What is the point of asking questions that basically involve nothing of import other than how well you remember the meanings of V speed subscripts? That's why glossaries were invented.

Another favorite of mine:
FAA advisory circulars (some free, others at cost) are available to all pilots and are obtained by

A. distribution from the nearest FAA district office.
B. ordering those desired from the Government Printing Office.
C. subscribing to the Federal Register.
​
The NTSB accident archives are overflowing with cases where failure to know the correct answer to the above was found to be the probable cause! So the finite time all students have is well spent learning the above. :crazy:
 
Welcome to my world. No data base of questions or answers was published, just show up and take the test and see how it goes. I'm a hell of a lot more proud of my unassisted 78 on the PPL than the 98 on some of the later tests when we knew all the answers.

I think everybody who took one of the brain-dead writtens should have to take it over.
 
Rotor&Wing said:
Didn't ACME make most of those products Wile E. Coyote would order to try to catch the Roadrunner?? :D
Click to expand...

acme.jpg
 
CISSP exam was confidential, but it was a good exam. When the quality of the FAA writtens matches the quality of the the CISSP and others like it, then I'd be fine with keeping it confidential.

Guess I better finish the II before my written expires in the fall.
 
When I started pursuing my certificate, the exam seemed like an enormous obstacle. Then I found out ALL the questions AND answers were publicly available. Then, my only question was "How could ANYONE fail????"

Asfar as I'm concerned, this is a good move, aslong as the FAA gets rid of dopey anachronistic questions about ADF and distance from [VOR.

Do you need to know those things? Sure, maybe, if you even have an ADF. The odds of your new, non-canadian, non-Alaskan plot needing to know anything about ADF are virtually ZERO. Spare me the "My home droneonly has an ADF approach sobstories. One, we're talking PPL. Two, your ADF has the expected lifespan of a Thanksgivin turkey. I have a (actually, two) GPS receivers. I really don't give a bowel movement how many satellites make up the gps constellation. The FAA has absolutely positively no friggin clue how to assess one's knowledge of gps (hntf - follow the magenta line of death)
 
I took the IR written in December and there were many questions on subject du-jure that were NOT in the database of study questions nor hilighted in the study prep materials. Yes, the old ADF nav questions and the famous which plane is represented by which indication on the RMI crap was still there (as were HSI questions) but virtually nothing on GPS work (I am told that the FAA has not caught up with that yet) I did have 7 intense questions about icing, the impact of such on airframe and flight characteristics, and proper pilot inputs to deal with it (the Colgan effect?)

Glad its over....now if only we could schedule the checkride...
 
Last edited:
DutchessFlier said:
I took the IR written in December and there were many questions on subject du-jure that were NOT in the database of study questions nor hilighted in the study prep materials. Yes, the old ADF nav questions and the famous which plane is represented by which indication on the RMI crap was still there (as were HSI questions) but virtually nothing on GPS work (I am told that the FAA has not caught up with that yet) I did have 7 intense questions about icing, the impact of such on airframe and flight characteristics, and proper pilot inputs to deal with it (the Colgan effect?)

Glad its over....now if only we could schedule the checkride...
Click to expand...

I passed the IR written last October and I only recall a couple of questions (related to icing) that I hadn't seen during my prep. Like you, I'd like to get this finished, too. Weather and travel (I'm writing this from the RCC at NRT) keep getting in the way. Clean up the last mile on the ILS and I'll be about ready for the check ride.
 
Consider the FOI, which is based on FAA-H-8083-9, Instructor's Handbook. Is the knowledge test still based on this document, dated 2003 but really from 1999? Or is there new material and the document hasn't been updated yet to match?

I've used Dauntless' Ground Instruction software and this evening's update brought the number of possible FOI questions to 252.

What's really annoying is $150 for a 50 question exam. But we've discussed that in another thread, no need to rehash here.
 
TMetzinger said:
If the test questions (and answers) were well written, then a candidate who truly understood the material would be relatively unaffected by a new bank of questions.
Click to expand...

That's just the problem. The FAA written tests are CRAP. Poorly worded, "trick" questions etc... I've gotten better than 90 on every one I've taken, but while I do know my stuff, I attribute those scores largely to studying the question bank and finding the "tricks" that way.

Only when the questions and answers are completely unambiguous will it be fair to not publish them. When I get an E6B question and the calculated result is EXACTLY HALFWAY between two of the answers... What's the test really testing?

I would say that about 60% of the questions are reasonable and easy to answer, while the other 40% are either written poorly or as trick questions. They're pretty worthless at that point.
 
I have one saving grace here: after retiring from 26 years in the world of public education and on career number 2 (or is it 3?) I want to work on my ground instructors cert-and thank goodness don't have to put up with the FOI exam, and yes, I would have gotten tarred and feathered if I ever put an exam out that was crafted this poorly.
 
flyingcheesehead said:
When I get an E6B question and the calculated result is EXACTLY HALFWAY between two of the answers... What's the test really testing?
Click to expand...
Or how about the one about MSA sectors where on the provided chart, the sector the question refers to has the number smeared to the point of being illegible (though the other one is clear as day)? I've found that one in several study guides and AFAIC it raises "trick question" to a whole new level. I agree with Bruce: as a lecturer who has to devise and edit online test bank questions as part of my job, I'd be mobbed by my students if I let a scanned image like that appear on an actual test! Hopefully they'll fix the image before too long, or maybe they already have.

Another data point for comparison: I passed the instrument written at the end of June and there were something like 2, maybe 3 questions I hadn't seen in the ASA study materials. One I remember, that I could only guess on, was about, IIRC, the specific effect of popping boots on airflow over the wing. Not only hadn't I seen that question, but the material hadn't even been covered in any of the prep material I used (though to be fair, my King course was several years old, so if that material was recently added to the exam, it's not their fault I hadn't studied it).
 
azure said:
Or how about the one about MSA sectors where on the provided chart, the sector the question refers to has the number smeared to the point of being illegible (though the other one is clear as day)? I've found that one in several study guides and AFAIC it raises "trick question" to a whole new level. I agree with Bruce: as a lecturer who has to devise and edit online test bank questions as part of my job, I'd be mobbed by my students if I let a scanned image like that appear on an actual test! Hopefully they'll fix the image before too long, or maybe they already have.
Click to expand...

I had a long chat with ASA a couple weeks ago about the iPhone version of the Prep series. My #1 complaint was that the images did not "zoom" nor did the text. (Would be fine on the iPad, but that's not the issue). According to them, the FAA does not allow any alteration of the graphics, hence there's no zoom. Yet the entire collection of graphics published by the FAA (FAA-CT-8080-5E) is letter size PDF format. Which means I can display at any size with no degradation of the quality.

And the ASA version only has 192 questions as of today.
 
azure said:
Or how about the one about MSA sectors where on the provided chart, the sector the question refers to has the number smeared to the point of being illegible (though the other one is clear as day)? I've found that one in several study guides and AFAIC it raises "trick question" to a whole new level. I agree with Bruce: as a lecturer who has to devise and edit online test bank questions as part of my job, I'd be mobbed by my students if I let a scanned image like that appear on an actual test! Hopefully they'll fix the image before too long, or maybe they already have.

Another data point for comparison: I passed the instrument written at the end of June and there were something like 2, maybe 3 questions I hadn't seen in the ASA study materials. One I remember, that I could only guess on, was about, IIRC, the specific effect of popping boots on airflow over the wing. Not only hadn't I seen that question, but the material hadn't even been covered in any of the prep material I used (though to be fair, my King course was several years old, so if that material was recently added to the exam, it's not their fault I hadn't studied it).
Click to expand...

Are you seriously suggesting that someone involved in writing test questions in Oklahoma City conspired with someone at the Government Printing Office in DC to somehow perform some trickery during the printing process? I just imagine what the print order might say...."Print 50,000 copies of the text provided; on page 32, question 1234, make the print unintelligible."

Have you ever seen a commercial press in action?

Bob Gardner
 
bobmrg said:
Are you seriously suggesting that someone involved in writing test questions in Oklahoma City conspired with someone at the Government Printing Office in DC to somehow perform some trickery during the printing process? I just imagine what the print order might say...."Print 50,000 copies of the text provided; on page 32, question 1234, make the print unintelligible."

Have you ever seen a commercial press in action?

Bob Gardner
Click to expand...

How come it's screwed up on the digital version too?
 
Okay I misspoke... it's not smeared, it's blotted out. The figure is Figure 49, the LOC/DME 21 at PDX and the MSA number is behind a river. On the actual chart it's quite legible, but in the King test prep booklet from 2004 and a couple other places I've seen it more recently, it's not.

I haven't suggested anything about HOW it was done, only THAT somehow, it was, and the FAA question asks about the MSA sector that's not legible, which, even if the illegibility was accidental, they had plenty of time to either correct the image quality, or change the question.
 
wabower said:
Welcome to my world. No data base of questions or answers was published, just show up and take the test and see how it goes. I'm a hell of a lot more proud of my unassisted 78 on the PPL than the 98 on some of the later tests when we knew all the answers.

I think everybody who took one of the brain-dead writtens should have to take it over.
Click to expand...

I had to do it at the GADO too!
 
wabower said:
I'm a hell of a lot more proud of my unassisted 78 on the PPL than the 98 on some of the later tests when we knew all the answers.
Click to expand...
You remember what you got on the PPL written? :hairraise:
 
I guess that confirms that my nearly failing the FOI written (78%) with the excuse that the questions were different then what I studied was not an excuse.

More than half of those taking the FOI knowledge test since the changes were implemented failed the exam.
Click to expand...
That makes me feel even better about it. The questions were very-much-different in my opinion and written to be intentionally confusing. On such a dry already-confusing subject that you pretty much need to memorize anyways that's a bad combination.
 
TMetzinger said:
How come it's screwed up on the digital version too?
Click to expand...

Because the digital version is just a scan of the print version. No one at the FAA bothered to scan then take it into Illustrator to clean it up, then save it as EPS or PDF so it could be used at various sizes.

It's a crappy scan.
 
I definitely got hit with some of these. Some quite odd-ball questions about tailplane stalls and structural icing that read like they came directly out of the Colgan accident report. It was that blatant. This was about two-three months ago.

The problem was that they were also "tweaked" in such a way as to almost be trick questions. Very strange wordings. Whoever wrote them was definitely trying to be "clever". Fine.

I marked them, and came back to them at the end, read and re-read them, and decided my score would mathematically still be well within the passing range if I just guessed and that they were wasting my time.

I clicked my best guesses (more than 50% of those specific questions wrong) and my score suffered a not-that-little indignity, and I passed. But not with a "great" score. Don't care... either I know what they want me to know on the day I take my checkride, or I don't. The penalty for me isn't so big a deal... more flying.

For an aspiring professional though, I could see why some folks would be ticked... if it were my job or going to be my job I might be Type-A enough to have complained.

In my case... not much to worry about... just moving on, ignoring whoever decided it was time to be ultra-annoying. I mean, they still have a pile of frakkin' RMI and other dying breed "How do you read this instrument" questions in the pool, so the pool has to somehow change into something relevant to what most people are flying these days.

Of course, it'd be odd to see specific questions like "in what menu on X instrument can you find Y information on a Garmin Z?", too.

My pet peeve is the slaved compasses and crap... "Which way, clockwise or counter-clockwise?" Who cares? Push a button... if it starts slewing the wrong way, push the other button. After three attempts with your particular instrument, you'll have it figured out.

At least it looks like they're at least moving to a question pool made of things that could actually kill you, versus trivia about old instruments virtually no one has on board anymore.
 
Yep, and not likely to forget. Fail rate was high back then, and I had no help from CFI or anybody else. Local guy wasn't interested in teaching ground school, said passing the written was the student's responsibility to prove they reallly wanted the ticket.

So I just bought some books and studied until I thought I knew something about the material and took the test. Based on other scores I heard, I think 78 would have been at least a B on the curve.

Everskyward said:
You remember what you got on the PPL written? :hairraise:
Click to expand...
 
Last edited:
Is this true for the comm test questions, too? I'm using the Gleim (for the 3rd time).
 
I took the commercial test in December and I'd say that most of the questions are in the bank. There may have been two or three new ones but it didn't matter since more than 70% were the canned questions.
 
You must log in or register to reply here.
Back
Top