Limmer Education’s item writing panel convened in Las Vegas during EMS World Expo was an overwhelming success in every way we hoped it would be. We have a new batch of amazing questions for our products, the attendees worked together and learned the whole time, and it was fun.
After taking a week or so to digest it, we came up with some takeaways for writing high-quality multiple choice exam items. Some you may have seen before in this blog while others were quite revealing.
Too much “fluff” in stem of question
It is time to do away with the “It was a dark and stormy night…” intros and the unnecessary information. Long stems are difficult to read. They also tend to include too much information which makes the question easier.
Grammar matters
Poor grammar and awkward sentence structure detract from an otherwise solid question. Bad grammar makes items more difficult to read and reflects poorly on you and your exams. Improper transition into the distractors (a vs. a/an) can provide clues to the correct answer.
Distractors affect difficulty level significantly
The single biggest thing you can do to up your difficulty level is to provide less wiggle room between the distractors. Using a “best answer” vs. a “correct answer” format helps prepare students for the NREMT (and the street).
Consider “choose-ability” when creating distractors
When we write basic multiple choice questions, we choose distractors that seem to fit based on the stem. The distractors should be similar, and in the “best answer” format we go a step farther. They should all have a plausibility. We went even farther and discussed “choose-ability.” How likely is a weak, average or strong student to choose a particular distractor? One way to identify good distractors is to include some short answer or fill-in questions on your exams. Use student’s frequently incorrect answers as solid distractors in subsequent exams.
A great question needs review from your peers
All of the item writers submitted solid questions. It was amazing to see how other educators interpreted the questions—and were able to make them better. To get the high-quality question (stem with just enough information and tight distractors) you need review by other educators to avoid ambiguity.
Don’t let the need for consensus prevent you from creating difficult questions
One thing we noticed as we got into the morning is that consensus often led us to questions becoming less challenging because the group could agree on the more straightforward solution. Great questions take work, revision, and more revision. During the session, we realized this and spent a little extra time to make sure we got it right.
Not all questions make the cut
There were a few times when we decided we couldn’t fix a question in the group—or that it would take too much time and we “tabled” it, or cut it altogether. Recognize the situations where you are better starting from the drawing board.
Do you have experiences in item writing you would like to share? We’d love to hear them.
Our panel in Las Vegas won’t be our last item writing panel. We are hooked on the experience and the quality it adds to our exam items here at Limmer Education. More to come! Contact us if you are interested in joining us for a panel or want to host a session.
