by Carrie Wiser, Senior Instructional Designer, The Game Agency
Multiple choice questions are not only used for knowledge checks and quizzes—they are also the basis of many eLearning games. For example, Trivia and Jeopardy!® gamify the learning experience, but at their root, they require designers to write a series of questions with answer choices. At least one of the choices is the correct answer. The other choices are wrong answers, also known as “distractors.”
What makes a good distractor? According to Butler (2018), effective distractors should be “simple in format (e.g., avoid the use of complex item types), challenge students but allow them to succeed often, and target specific cognitive processes that correspond to learning objectives.”
Although it can be challenging to write an instructionally sound question, it can be even more difficult to write plausible distractors that are not too obviously wrong. Let’s look at some guidelines that will help you write and validate multiple choice distractors for both eLearning courses and games.
Start with an effective question
First, consider the context of the question. Although multiple choice questions are commonly used for assessment, they can also be part of a pre-teaching or inquiry-based learning approach. The context of the question can impact the tone and variety of distractors. For example, a pre-teaching question might be lighter in tone with more familiar terminology and even “fun” distractors.
Next, consider the purpose of the question. What learning or performance outcome does it address—what do you want people to think and do differently? In any question-writing document, it’s a good idea to have a field where you specifically note the related objective. This documentation is not only important for initial goal-setting. It will also help you write distractors, validate the quiz or game, and manage change over time.
Once you’ve identified the context and the performance outcome, clearly state the question. There should be no confusion about what the learner is expected to do. If it is too long or complicated, people might perceive it as a “trick question,” which can have a negative impact on learning and motivation.
Consider three answer options
Have you ever struggled to write “just one more” distractor? Reducing the number of answers can save design time and improve the quality of the distractors. Many quizzes and tests include four options (A, B, C, D) but the conventional wisdom is shifting towards three. Haladyna and Rodriguez (2013) recommend using “only options that are plausible and discriminating; three options are usually sufficient.” Kilgour and Tayyaba (2015) found that many learners perceived questions with two distractors to be more challenging than questions with three or four distractors. That said, the number of options is ultimately driven by the content, and it is okay to vary the number of options within the same course or game.
Think of distractors as non-examples
When you are learning something new, your brain organizes that information into categories or schema. For example, lettuce is green and leafy and it is a vegetable, so green and leafy kale must be a vegetable too. Over time, your schema adapts to fit newly learned information; distinguishing between examples and non-examples is an important part of that process. For example, an orange carrot with a green and leafy top is a vegetable, but a round orange that grows on a tree is not.
When you are writing distractors, look for items that you can frame as non-examples. Review prerequisite knowledge or information covered in other parts of the course. Or, research topics or skills that use similar terminology or tools. Do not try to trick learners or overcomplicate the answer! Instead, look for similar or related concepts that will lend authenticity to the distractors and help learners begin to form a mental framework for new knowledge. Remember to close the loop with good feedback that explains why a distractor is wrong, rather than just telling the learner “that’s incorrect.”
Talk to subject matter experts
Distractors are often the most difficult part of the question to write, but they are a great way to address key misunderstandings and misconceptions. So if you are fortunate to have access to a subject matter expert (SME), focus on the distractors—not just the questions. Ask for their input on realistic alternatives and common phrases and language. As an added bonus, their suggestions may improve the questions in important ways that you might otherwise miss. You can guide SME input with questions like:
- Are there common misconceptions or misunderstandings about this concept?
- What behaviors do you value vs. what do you discourage?
- What are some realistic and relatable mistakes?
- What happens when you do not know this / do this correctly?
- Do you use specific jargon or phrases when you talk about this on the job?
Follow semantic rules
Sometimes a grammatical issue can give a clue to the “right” or “wrong” answer. For example, a question might ask what lions “are” followed by the options “dog, cats, reptile.” In this case, the mixture of plural and singular nouns gives a hint to the correct answer (cats). To avoid errors like these, make sure all the answers and distractors:
- Match the question stem
- Are grammatically parallel
- Are similar lengths (This is more difficult than it sounds! But our attention is drawn to answers that look shorter or longer than the rest.)
Limit “all” and “none”
Rather than gauging what people actually know, “All of the above” or “None of the above” often give an unintended clue or shortcut to the correct answer. Try to limit or remove them. It is better to have a smaller number of quality distractors than a longer list that ends with “all/none of the above.”
On a related note, it is also a good idea to avoid negative phrasing (e.g., “which one is NOT true” or “all of these are true EXCEPT”). Unless the learning objective requires that type of distinction, this phrasing puts undue emphasis on the non-examples and things that you do not want people to do. Try to adjust the distractors and reword it as a positive question instead. Or, shift from a single answer (Which color is NOT in the rainbow?) to a multiple select question (Select ALL the colors in the rainbow). Keep in mind that multiple select questions are often more challenging. So use them purposefully and not as a catch-all replacement for negative phrasing.
Validate your work
Before you complete the design process, review and validate the questions and distractors. Confirm that every question ties back to a specific learning objective. As mentioned earlier, this is easier if you track it as a note in your document. Make sure the distractors are straightforward, easy to interpret and follow semantic rules. Whenever possible, find a colleague or other “fresh set of eyes” who can read over the questions and distractors for you. One common mistake to look for: distractors that are not mutually exclusive (for example, if the answer is “10”, “more than 5” and “more than 7” are both correct.)
After you deploy the quiz or game, improve your questions (and your quiz-writing skills) by conducting a distractor analysis. Even if you have limited time, a distractor analysis on one previously written quiz or game can provide valuable insight that will help you improve future questions and distractors. For example, if your questions are in a game on The Training Arcade you can drill down on a question-by-question level to see which distractors people selected. Some things to look for include:
- Frequently selected distractors may indicate a misunderstanding or other training needs that you need to fill.
- Distractors that are rarely or never chosen might need to be removed or revised.
Effective distractors are key to a challenging but effective quiz or game. Reducing the number of distractors, looking for non-examples, and talking to subject matter experts can make it easier to write more plausible distractors. Following semantic rules, limiting “all/none,” and validating your work help ensure that your thoughtfully generated distractors are framed in the best way possible. Before you write your next knowledge check, try conducting a distractor analysis on an existing multiple choice question set. Then take those lessons learned with you to the next quiz or game you write!
Schedule a personal demo here to discuss how we can help you build games with The Training Arcade game-authoring software. It’s quick and easy but if you don’t have time, we get it, we can provide game-building services to get you started!
Resources:
- Brame, C. (2013) Writing good multiple choice test questions. Retrieved July 25, 2022 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
- Butler, A. C. (2018). Multiple-choice testing in education: Are the best practices for assessment also good for learning? Journal of Applied Research in Memory and Cognition, 7(3), 323–331. https://doi.org/10.1016/j.jarmac.2018.07.002.
- Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529
- Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Routledge.
- Kilgour JM, Tayyaba S. An investigation into the optimal number of distractors in single-best answer exams. Adv Health Sci Educ Theory Pract. 2016 Aug;21(3):571-85. Retrieved July 25, 2022 from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4923093/
- Malamed, C. The Importance of Writing Effective Distractors (2020). Retrieved July 25, 2022 from https://theelearningcoach.com/elearning_design/tests/the-importance-of-writing-effective-distractors/
- “Schemas and Memory.” Psychologist World. https://www.psychologistworld.com/memory/schema-memory
