by Dan Limmer
If you’re reading this post, you’ve either written a multiple choice exam item or are about to begin writing one. I’ve put together my six favorite tips on how to write a killer question—seven, if you count this basic premise:
Always begin with the intent to write a tight, accurate, well-constructed question. Create a concise stem and include 4 choices for each question.
Then…
1. Get rid of the fluff.
Tell me you’ve never written a question stem like this (I have):
It is a dark and stormy night. You and your partner have just finished dinner when you are dispatched to a 68-year-old female who “passed out.” You arrive to her well-kept home and find her sitting in a chair. She appears pale. Her vital signs are: pulse 92, respirations 22, blood pressure 102/78, SaO2 96% on room air. When she stands up to move to the stretcher, she feels faint. You believe this is because: …
(Yawn)
The following concise stem essentially asks for the same conclusion:
Your patient experienced a syncopal episode. She stands up from a chair and feels faint. You suspect this is because: …
It is a common—and false—belief that a longer question is more complex or at a higher cognitive level. The opposite is often the case. Long stems and choices test reading skills as much as, or more than, the actual content. NREMT questions are rarely more than three sentences long. Keep your stems and choices tight.
2. Use proper grammar and spelling.
Consider this question:
A person who has a greenstick fracture has a
a. partial break.
b. angulated injury.
c. open injury.
d. closed but angulated injury.
Incorrect grammar (a angulated injury, a open injury) lets a student rule out two of these choices immediately. By using a/an at the end of the stem, instead of only a, you could have avoided this problem. Other ways to avoid it would be to include choices that all begin with consonants, or to include a or an appropriately at the beginning of each of the four choices.
Poor sentence structure, incorrect spelling and grammatical errors detract from the question itself. You might not believe students notice or care—but they do. These errors not only make you look bad but also can make it easier for students to guess the correct answer.
3. Avoid jokes and “tells.”
Occasionally we put a bit of humor into our questions, often through a silly or humorous distractor. Although this might be okay occasionally, it inflates success on multiple choice questions by 25%. If you have a stem that asks about the carina, avoid including a distractor such as, “A new model Toyota.”
Humor in the stem can detract from the question’s meaning as well as increase anxiety in a nervous test taker.
There are several “tells” that astute students use to guess the answer easily when they don’t know it. Here are two of the most common:
- Disproportionately often, the choices All of the above or None of the above are the correct answer. We may fall back on these answers when we’re struggling to come up with that last choice in the multiple choice question. These answers also may become the default when the student sees a couple of likely choices.
- The longest choice is usually correct. Avoid this trap by making all the choices in a multiple choice question a similar length. Or, at the very least, vary your choices so that the longest answer isn’t always the correct one.
4. Distractors should distract.
A solid multiple choice item should have distractors that are both homogenous and plausible. These two concepts are the hallmarks of the “best-answer” exam question.
Best-answer questions are the most challenging to write. They are also the most beneficial to your students who will be taking the NREMT exam, because they require the most thought and understanding to answer.
The NREMT instructs item writers to give each distractor some plausibility so “…that the correct answer is the one and only correct answer; [so] that each distractor option has some plausibility…”. (Source)
Homogeneity means that all the items in a series share a sameness. For example, if you write a question that asks, Which abdominal organ resides in a particular quadrant?, all of the choices should name an abdominal organ. Failure to do this essentially gives the student a 25% or greater chance of getting the question correct.
Plausibility means that all the answer choices appear to be believable. Plausibility is challenging when creating an exam item, because the item writer must create plausibility without introducing ambiguity into the question—that is, one answer must still stand out as correct. Here is an example of an item that incorporates plausibility (taken from our EMT PASS app):
A 26-year-old male complains of increased breathing difficulty over the past two days. He tells you he has been coughing frequently. His vital signs are B/P 124/88, P 108, R 18. His SpO2 is 91% on room air. You suspect his SpO2 reading is low because he has
a. many collapsed alveoli in his lungs.
b. bacteria in his lungs.
c. lower airway constriction.
d. inadequate tidal volume.
All four choices name a condition that can cause a decreased oxygen saturation—thus each choice is plausible. This question also includes distractors that require the student to analyze what each choice means. Collapsed alveoli in the lungs is atelectasis—unlikely in a 26-year-old. Lower airway constriction would cause wheezing, which isn’t mentioned in the stem. The patient says he has been coughing and his vitals/saturation aren’t that bad, so inadequate tidal volume (respiratory failure) is also unlikely. The correct answer is bacteria in his lungs, also known as pneumonia.
5. Match your questions with your objectives and lesson plan.
Exams do several things. They measure the effectiveness of both learning and teaching. They also hold students accountable and motivate them to study. In a class on item writing, Joe Mistovich tells educators, “If you want your students to learn superficially, test superficially. If you want your students to learn comprehensively, test comprehensively.” Students will complain about their first few comprehensive exams, but stick with it. If you don’t back down, their only option is to study—and to learn more.
Your exams should be prepared with care so that your test items match your lesson objectives. The increased demands on educators can create a time squeeze, resulting in exams put together in haste after a quick look through a publisher’s exam bank. It takes a lot of time to create objectives and lesson plans for each topic in your class—and this must happen before you can create those amazing exam items.
Much is discussed about “valid” exams. Frequently the conversation centers on the validity of individual items within an exam. Remember that the most basic definition of a “valid” exam is one that accurately measures the desired learning outcomes.
6. Have other educators review your questions (and spend time reviewing other educators’ questions yourself).
If all your questions are so clear-cut that you never need to have a few fellow educators take a look at them, then your questions are likely too easy.
The NREMT uses an item writing panel in which 8 to 10 educators submit questions in advance of an in-person meeting. Over a period of days, each item is projected on a screen and dissected by the group. It is unreasonable to believe that a single educator working in a vacuum will be able to create test items of the highest quality.
A mutual review of test items not only benefits your current exam but also helps you create better items for future exams by having read items written by others.
Better exams are within your reach. If you have comments—or additional tips for better exam item construction—please share them in the comments below.
A previous post on item writing can be found here.
