Co-Writing Community (Fridays 8:15 – 9:15 a.m., Spring 2024)

Tara DaPra, one of CATL’s Instructional Development Consultants, will lead a “Co-Writing Community” this spring.  A Co-Writing Community is a zero-obligation, zero-preparation, zero-outside work activity. Use the time to work on creative or scholarly projects that might otherwise get pushed aside by the demands of teaching. All faculty and staff are welcome!

The Co-Writing Community will meet throughout the spring semester on Fridays from 8:15 – 9:15 a.m. Feel free to join early or late, weekly, or when your schedule allows! All meetings will be held virtually. Simply drop in with this Zoom link which will be reused for each session.

Please email daprat@uwgb.edu with any questions.

Writing Effective Multiple-Choice Questions

Writing good multiple-choice questions is challenging. Tricky or verbose questions can reduce the test item’s reliability and validity, while a poor selection of answer choices can make a question either far too easy or incredibly difficult. A question that suffers several common pitfalls might even work against the learning outcomes it is trying to measure. Fortunately, researchers and assessment experts have identified some common guidelines for creating more equitable and reliable multiple-choice assessments. In this guide, we’ll walk through seven tips for writing more effective multiple-choice test items.

The scope of this guide is focused specifically on authoring multiple-choice questions. If you’d like to dig into when to use multiple-choice assessments, as well as recommendations for scaffolding, testing in online environments, and providing feedback, check out this other CATL blog post on general considerations for creating impactful multiple-choice assessments.

Before getting into the tips on writing questions, we’ll review the anatomy of a multiple-choice question and outline some common language that we use throughout this guide.

Table of Contents

The Anatomy of a Multiple-Choice Item

Throughout this article, we will use the following terms and definitions when referring to the parts of a multiple-choice question:

  • Item: A question and its answer choices as a unit
  • Stem: The posited question that respondents are asked to answer; often phrased as a question, but can also be a statement (e.g., fill in the blank)
  • Alternatives: A list of suggested answers that appear after the question stem; comprised of several incorrect answer options and one (or more) correct or best answer(s)
  • Distractor: An incorrect alternative

An example multiple-choice question in which the top portion is labelled as the “stem” and the answer choices A through F are labelled “alternatives,” with “A” serving as the answer and “B-F” serving as distractors

Example of a multiple-choice item, its stem, and the alternatives

Source: Vanderbilt Center for Teaching and Learning

Tips for Writing Effective Multiple-Choice Questions

Most of the recommendations in this guide have been adapted from How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty Simple (Burton et al, 1991) and Developing and Validating Multiple-choice Test Items (Haladyna, 2004). We’ve distilled down these long-form documents into a few simple guidelines that align with current recommendations from experts at other centers for teaching and learning (see “Additional Resources”). If you are interested in learning more about the research behind these suggestions, we encourage you to check out one or both of the resources linked above.

Tip #1: Tie each item to a learning outcome

In order to maximize an assessment’s validity and reliability, each multiple-choice item should be clearly aligned with one of the assessment’s learning outcomes (and, by extension, the course learning outcomes). Generally, it is recommended to have each item tied to only one outcome each. However, in the case of items that assess higher order thinking and present complex problems or scenarios, it is possible that a multiple-choice item may assess more than one outcome.

Tip #2: Create a specific, clear, and succinct stem

A straightforward, clear, and concise stem free from extraneous information increases a multiple-choice item’s reliability. When writing and revising your question stems, it is a good practice to ask yourself if there is a simpler or more direct way to rephrase a question. Overly wordy stems rely on students’ reading comprehension, which is usually not one of the intended outcomes of the assessment. Likewise, confusing or ambiguous stems can be accidentally misleading. Ideally, a student who has mastered the target outcome should be able to answer the question posited even without the alternatives present.

For millennia, humanity has been entranced by the ebb and flow of the tides. Many past civilizations believed the ocean's waters were controlled by monsters, spirits, or gods, but today know the scientific laws and theories that explain the tides. These movements are influenced by, in part, the gravitational force from the sun, the earth’s rotation, shoreline geography, and weather patterns, but all of these pale in comparison to the effects of:

  • a)  El Niño
  • b)  The gravitational force of the moon
  • c)  The ozone layer
  • d)  Deep-sea trenches

(Answer: B)

Why it doesn’t work: The extra information in the question stem makes it difficult for the test-taker to discern the question that is being posed. The question itself is also worded ambiguously.

Earth’s tides are influenced primarily by:

  • a)  El Niño
  • b)  The gravitational force of the moon
  • c)  The ozone layer
  • d)  Deep-sea trenches

(Answer: B)

Why it works: The question stem has been revised to remove all unnecessary information and it now poses a simple, straightforward question.

Tip #3: Avoid using negatives in question phrasing

It is usually best to avoid negative phrasing in question stems, such as asking students to identify which alternative does not belong. Negatively phrased stems tend to be less reliable in assessing students’ learning than stems that ask students to identify the correct answer. The exception to this guideline is in cases when knowing what not to do is key, such as questions related to safety protocols. If you do choose to include a negative qualifier, use bold or italics to emphasize the negative word and make sure that you don’t create a double negative with any of the alternatives.

Which of the following is not a quality of an active listener? 

  • a)  Not talking over others 
  • b)  Making eye contact with the speaker 
  • c)  Asking clarifying questions 
  • d)  Mentally planning a rebuttal while the other person is speaking 

(Answer: D)

Why it doesn’t work: The question stem is phrased in the negative and the negative qualifier is not emphasized, making the question less reliable. Additionally, one of the alternatives also contains the word “not,” creating a double negative with the question stem. 

True or false? An active listener…

  • Refrains from talking over others (T/F)
  • Makes eye contact with the speaker (T/F)
  • Asks clarifying questions (T/F)
  • Mentally plans a rebuttal while the other person is speaking (T/F)

(Answers: T, T, T, F)

Why it works: This question stem has been rephrased to avoid using the word “not.” The answer choices have been turned into four separate true/false statements so each item can be assessed separately and have been revised to remove the word “not.”

Which of the following is not a recommended action to protect yourself during an earthquake if you are inside a building?

  • a) Drop to your hands and knees
  • b) Take shelter under a sturdy nearby desk or table
  • c) Crawl to the nearest exit
  • d) Cover your head and neck with your arms

(Answer: C)

Why it works: In this scenario, knowing what not to do during an earthquake is one of the learning outcomes, so it is appropriate to use a negative qualifier. The negative qualifier in the stem, “not,” has also been emphasized with bold and italics to draw attention to it.

Tip #4: Use plausible distractors

Good distractors need to appear plausible to students that have not met the target learning outcome, but not so tricky that they could be argued as correct answers by a test-taker that has met the learning outcome. When you are writing a multiple-choice question it is often useful to write the stem first, then the correct answer first. Once you have decided on these two pieces, formulate 2-4 distractors based on common student misconceptions. If you can’t think of another “good” distractor for a set of alternatives, it is usually better to have fewer alternatives than to include extra alternatives just for the sake of consistency.

George Washington Carver is best known for his work as a(n) ______.

  • a)  Agricultural scientist
  • b)  Extraterrestrial expert
  • c)  Basket-weaver
  • d)  Juggler

(Answer: A)

Why it doesn’t work: The distractors are so absurd and far-removed from the topic of the question that even a student who knows nothing about George Washington Carver could discern the correct answer, making the test item neither reliable nor valid.

George Washington Carver is best known for his work as a(n) ______.

  • a)  Agricultural scientist
  • b)  Electrical engineer
  • c)  Microbiologist
  • d)  Politician

(Answer: A)

Why it works: The distractors seem plausible, creating a question that will more accurately assess students’ knowledge of George Washington Carver.

Tip #5: Use homogeneous phrasing and formatting for alternatives

Small typos, inconsistencies in tenses or phrasing, or changes in text formatting can accidentally provide clues about which alternatives are the distractors and which are correct answers. Savvy test-takers can pick up on these inconsistencies and use this information to deduce the correct answer even if they have not achieved mastery for the desired outcome, so keep an eye out for these things as you proofread your exam. If you notice formatting inconsistencies in your Canvas quizzes, you can use the Rich Content Editor to remove all formatting and set the selected text to Canvas’s defaults.

What three parts of speech can an adverb modify?

  • a)  Verbs, adjectives, and other adverbs
  • b)  Noun, adjective and preposition
  • c)  Verb, noun and conjunction
  • d)  Adjective, adverb and exclamation

(Answer: A)

Why it doesn’t work: Answer choice “A,” the correct answer, is in a different font from the other alternatives. Additionally, the distractors use the singular version of each part of speech, rather than the plural, and omit the Oxford comma before “and.” These inconsistencies hint to students that “A” is the odd one out.

What three parts of speech can an adverb modify?

  • a)  Verbs, adjectives, and other adverbs
  • b)  Nouns, adjectives, and prepositions
  • c)  Verbs, nouns, and conjunctions
  • d)  Adjectives, adverbs, and exclamations

(Answer: A)

Why it works: The distractors have been revised to look consistent with the correct answer, creating a question that assesses students’ knowledge of parts of speech, rather than their eye for detail.

Tip #6: Avoid using none-of-the-above or all-of-the-above as alternatives

Questions that provide “all of the above” or “none of the above” as alternatives are generally less reliable for assessing outcomes than a multiple-choice question with mutually exclusive alternatives. The table below outlines the use cases for “all of the above” and “none of the above” along with why they are flawed for reliable assessment in each instance. If you can’t think of another distractor while drafting a question, remember that it is okay for some questions to have fewer alternatives.

Use of “all of the above” and “none of the above” 

Alternative

Weakness

“All of the above” as the answer Can be identified by noting that two of the other alternatives are correct
“All of the above” as a distractor Can be eliminated by noting that one of the other alternatives is incorrect
“None of the above” as the answer Measures the ability to recognize incorrect answers rather than correct answers
“None of the above” as a distractor Does not appear plausible to some students

(Adapted from How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty, Brigham Young University)

Tip #7: Create questions with only one correct alternative

Like none- or all-of-the-above alternatives, asking students to identify multiple correct alternatives is a less reliable form of assessment than an item with only one correct answer. Multiple-response questions are also reliant on confusing grading calculations, since selecting an incorrect alternative “cancels out” a correct selection (this Canvas guide goes into more detail about how Multiple Answer questions are auto-graded). And, in questions with more incorrect than correct answers, students can still score points by selecting no answers at all!

A straightforward multiple-choice item with only one correct answer and mutually exclusive alternatives is a more reliable way of discerning whether a student truly knows a concept or is guessing. Another option is to turn a multiple-answer question into a series of true/false questions, which will provide a more reliable picture of students’ understanding and a more valid grade for their efforts.

Check all that apply. COVID-19:

  • a)  Is an infectious disease
  • b)  Is spread primarily through fungal spores
  • c)  Can be treated with antibiotics
  • d)  Can infect people of all ages

(Answer: A and D)

Why it doesn’t work: Because of the way multiple-response questions are graded, they are less reliable than individual multiple-choice or true/false questions.

True or false? COVID-19:

  • Is an infectious disease (T/F)
  • Is spread primarily through fungal spores (T/F)
  • Can be treated with antibiotics (T/F)
  • Can infect people of all ages (T/F)

(Answers: T, F, F, T)

Why it works: Each statement is assessed individually, allowing for more granular and accurate scoring.

Questions?

Want more tips for writing multiple-choice questions? Looking for someone to help brainstorm outcome-aligned questions with? CATL is here for you! Reach out any time to set up a meeting or send us your questions at CATL@uwgb.edu.

Additional Resources

Revising—and Reframing—Your Teaching Philosophy

Article by Tara DaPra, Assistant Teaching Professor & 2022-23 Instructional Development Consultant

Why should you write a teaching philosophy? Chances are, you already have, even if it was way back in graduate school or when you applied for the job you now hold. But if you are going up for promotion, as many of us in the teaching professor category may now do, or if—happy days—someone nominates you for a teaching award, your teaching philosophy may need updating. You may be dreading this. You may continually move it to the end of a long list of more pressing tasks. You may ask yourself if anyone will really read this. Leonard Cassuto says what many of us are thinking when he writes, “Teaching philosophies account for some of the most tiresome reading that academe has to offer (and that’s saying something).” But must they be? Rather than a chore or a high-stakes assessment, why not re-frame what a teaching philosophy can—or perhaps should—be? What if you instead treated your teaching philosophy as a celebration of your time in the classroom and a vision for the future?

In an article for The Chronicle of Higher Education, James Lang argues that teaching philosophies “fall under the genre of creative nonfiction,” a genre of writing that privileges techniques like voice, narrative arc, and compelling details while insisting on a non-negotiable commitment to the truth. Lang warns writers of teaching philosophies not to fall into the default mode we so often see in student writing—telling rather than showing. So instead of regurgitating your course learning objectives or points from your CV, Lang advises that we zoom in and describe a day when those objectives were lived in a particularly meaningful way. He writes, “Readers remember and respond to your stories, not your explanations.”

Another hallmark of creative nonfiction is to distinguish between what the writer knows and does not know—and to lean in to the latter. In her essay “Memory and Imagination,” Patricia Hampl writes, “It still comes as a shock to realize that I don’t write about what I know: I write in order to find out what I know.” Teaching philosophies are, essentially, a personal essay, a space for writers to puzzle over a complicated question and attempt to answer it from many angles. The word essay itself means “trial.” What, then, is the question you most want to discover, as it relates to your teaching? What parts of that question have you answered and what parts remain a mystery?

Writing a teaching philosophy can help us to reflect upon and articulate our ideas about what makes for effective teaching. And doing this can help to ensure that what we do in our classes is consistent with those beliefs—but it can also acknowledge pieces of the teaching puzzle that we have yet to fit together. And so, while teaching philosophies should certainly highlight a teacher’s strengths and successes, good teachers might also acknowledge what they hope to learn next.

If you’d like to read more about writing effective and reflective teaching philosophies, CATL has gathered some resources.

Turnitin: Beyond Plagiarism Review

A feature highlight for Canvas this week is Turnitin. Although most instructors may be familiar with this tool as a plagiarism checker, it has additional impactful uses within the classroom. While checking for plagiarism is important to ensure academic integrity in student work, Turnitin can also function as a powerful feedback tool and as a self-assessment tool.  

According to several studies, feedback helps to further develop a learner’s cognitive abilities. Wisniewski, Zierer, and Hattie (2020) discuss how various forms of feedback have become a focus in teaching and the practice of teaching in recent years. The most impactful forms of feedback are task- and process-oriented feedback or formative feedback. Both provide students with information not only on how well they’ve met specific goals of an assignment or assessment, but also on how to improve their strategies for achieving those goals in the future.  

While feedback may not have a large impact on behavioral outcomes for learners, self-assessment and reflection activities do. Combining both feedback and self-assessment activities within an ongoing assignment opens up communication pathways within the classroom and may also help increase motivation in the learning environment. Feedback and self-assessment can also turn mistakes and errors into teaching moments for synchronous and asynchronous classes. As Terada (2020) points out both within and outside of the classroom, making and learning from errors is an integral part of the learning process. And Turnitin is one tool which can help provide those teaching moments. While we are familiar with the similarity reports produced by Turnitin, and often have used these reports for plagiarism review, studies show that Turnitin similarity reports can also be used for self-assessment. Chew, Ding, and Rowell (2013) in particular focus on how the similarity reports generated by Turnitin can be used by students to review and assess drafts of their own work or their peers’ work. With its integration in Canvas, Turnitin can be used both synchronously and asynchronously for all course modalities.  

Within the UWGB Canvas instance, Turnitin is paired with the Assignment feature and can be used in conjunction with peer review so that students can receive both a similarity report for self-assessment as well as receive formative feedback from both their peers and instructors. The Canvas SpeedGrader works with Turnitin to allow for suggestions, edits, and general comments to be provided in written, audio, or video format. For best results, a Turnitin Assignments can be paired with a Rubric, allowing students to both see the criteria for the assignment and review their drafts and feedback based on how well they met those criteria. Best practice would be to incorporate course outcomes within the rubric. This will provide transparency between instructor and students in setting and achieving overall course goals as well as expectations of the student.

To build out a Turnitin Assignment in Canvas, follow these directions.

  1. First, in your course site, navigate to the Assignments tab in the course navigation menu on the left side of the screen and click on Assignments.
  2. Next, click the + Assignment button in the upper right corner to create a new Assignment and then give the assignment a name.
  3. In the assignment Editing window, scroll down under settings and select the Online submission type, and then check the File Uploads option. This will cause a new setting option to appear called Plagiarism Review.
  4. The Plagiarism Review setting is default set to “None.” Change it to Turnitin. Turnitin is now enabled on the Assignment.
  5. To ensure draft submissions are not stored in a repository, change the setting under "Store submissions in" to Do not store the submitted papers. 
    • This setting is important for assignments that allow for multiple draft submissions. Not storing drafts into a repository means that subsequent drafts of the same assignment will not be flagged in the similarity report.
  6. Next, you can toggle on and off the different content types you want draft submissions to be compared to (student repository, website content, or periodicals, journals, and publications). 
  7. Below that, you can select what to exclude from the similarity reports generated by Turnitin. 
  8. You can also set the number of submissions to be "unlimited.”
    • This setting will allow students to resubmit drafts several times to the same assignment. For self-assessment it may be good to allow students to submit multiple drafts to review their similarity reports. Just remember to select “Do Not store the submitted papers” under the repository settings so students do not get flagged for work done on a previous draft of the same assignment.
  9. Lastly, click the button in the bottom right corner to Save and Publish your Turnitin Assignment.

Here is a general guide from Canvas discussing how to add or edit details in an assignment.