Learning Outcomes that Lead to Student Success 

What are learning outcomes and why do you need them?

There’s a famous misquote from Lewis Carroll, “If you don’t know where you’re going, any road will get you there.” The same is true in our courses: if you don’t know what you want your students to learn, it doesn’t really matter how or what you teach them. Every instructor wants to ensure student success, but if we as instructors don’t have accurate and well-thought-out learning outcomes, what does success mean in our classes? Creating learning outcomes should be a collaborative process where instructors responsible for teaching a course come together to craft these statements based on the most important learning in a course, taking care to maintain a balance between critical thinking and base knowledge while keeping an eye toward what makes a learning outcome an achievable learning goal.

Learning outcome creation

Before you create course learning outcomes

  • If your course is part of a program, you should ensure that the learning outcomes mesh with the rest of the program to meet all program learning outcomes.
  • Plan collaboratively with colleagues teaching the same course. All learning outcomes for sections taught of the same course should have the same learning outcomes according to the HLC (Higher Learning Commission) criteria 3a.
  • With colleagues, determine and list the most important learning or skills that will take place in this course.
  • Whittle down the list if it is too large. Consider what you and your colleagues can reasonably accomplish during the semester.
  • Pay attention to the conversation around Generative AI. What your students need to know and do may change because of the rapid development of AI.

Considerations as you create your learning outcomes

  1. Keep assessment and, therefore, your verb choices in the forefront of your mind. As you write learning outcomes, you want to ensure that the learning outcomes contain actions that can be demonstrated. When you ask students to “understand” something, this is difficult to demonstrate. If they “explain” it instead, that is an action that can be done and measured in various ways.
  2. Keep Bloom’s Taxonomy next to you as you create. It makes sense to use a taxonomy when writing outcomes. In Bloom’s model, skills and verbs on the bottom of the pyramid are less complex or intellectually demanding than those at the top of the pyramid; keep in mind they may still be totally appropriate, especially for lower-level courses. More critical thinking skills are required for those skills at the top of the pyramid, but it is useful and acceptable to use verbs and abilities from all levels of the pyramid. If you are teaching an upper-level course, you don’t want to draw all your verbs and skills from Bloom’s Taxonomy’s knowledge level. You should be using some higher levels in Bloom’s system.  The chart below can be a guide as you create those learning outcomes and note that generative AI developments may make the original chart problematic in different ways. There are alternatives to Blooms, as well.

    Alternatives to Blooms Taxonomy levels and verbs.
    Newtonsneurosci, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, v Wikimedia Commons
  3. Use SMART Goals also. In addition to including Bloom’s Taxonomy as part of your learning outcomes, we encourage you to make sure that your learning outcomes are created using the SMART goals model.   SMART goals were developed in 1981 by George Duran, who noticed that most business goals were not created in a way that could be implemented effectively.

SMART is an acronym we can use to describe the attributes of effective learning outcomes for your students. Please note that you will find different versions of the acronyms in the SMART goal model, but these are the ones CATL uses to discuss learning outcomes:

    • Specific – target a specific area, skill, or knowledge
    • Measurable – progress is quantifiable
    • Attainable – able to be achieved or realistic
    • Relevant – applicable to the students in the class
    • Time-based – achieved in a specific timeframe, such as a semester

Example: By the end of the semester (T), students will be able to diagram (M) the process of photosynthesis (S, A) in this biology class (R).

Learning outcomes are more likely to be meaningful if they can meet all of the qualifiers in the SMART acronym. Think specifics as you create your learning outcome. If you can’t tell if your learning outcome meets one of the qualifiers, you should rework it until it does.

Review your learning outcomes

Your next step as a team should be to review your learning outcomes. Compare them to the SMART model and Bloom’s Taxonomy or any other relevant model you might be using. If it helps, consider these examples. First, “Students will improve their understanding of passive voice.” On the surface, it might look like a reasonable goal, but then as you ask, “What does it mean to improve? Where did the student start from? When does this need to be done by?” This goal offers no answers to those questions.

How about this one? “By the end of the semester, all students will receive a 100% score on their math notation quiz.” For context, this is a Writing Foundations course. That begs the question, is this outcome relevant to this group of students? Is 100% a reasonable and attainable goal?

Consider these questions as a guide when creating SMART goals. A more reasonable goal for this group of writing students is that by the end of the semester, students will be able to identify and accurately and effectively use scholarly research in their writing projects 80% of the time. One part of the review process is ensuring your outcomes are SMART, but there are additional elements to consider, including the questions below.

  • Can you identify the verb in your learning outcome?
  • If your students master the skills in your learning outcomes, will they be satisfactorily prepared to go to another course that teaches the next level of this material?
  • If this is a course in a series, have you checked to be sure that your outcomes make sense with the previous and next courses?
  • Has your unit done curriculum mapping for its goals, and do your course outcomes align with that mapping?

Put it all together

Creating learning outcomes that reflect the learning necessary to achieve mastery in a course can be an arduous process. It should be a collaborative process as well. We encourage you to reach out to the CATL team if you would like guidance or help walking through Bloom’s Taxonomy and the SMART goal model. We are always available to help!

Resources on creating learning outcomes

Implementing Open Educational Resources (OER) into Your Course

This article is the third part in our series on OER. You can read more about Open Educational Resources and Affordable Educational Resources in part one and two alternative models for textbooks, Inclusive Access and Equitable Access, in part two.

I’m ready to adopt an Open Educational Resource (OER) – how do I find a text?

First, realize that OER don’t have to be a formal textbook, although often they are. OER can be pieces of textbooks that you use in conjunction with each other. They can be a Canvas course, a module, or a series of resources that meet your learning outcomes and the topics you need to ensure your students are meeting the learning outcomes. They could even be podcasts, films, and websites. This broadens the field of where to find OER. So where do you start?

  • Start with the librarians on your campus. Librarians are experts in locating materials and can make the search easier.
  • Use the libguide created by the library about materials you already have access to that can be used as part of an OER course.
  • Use one of the many repositories available online that offer the distinct types of resources mentioned above. College of the Canyons maintains an up-to-date OER/Zero Textbook Cost (ZTC) Repository which is a good jumping-off point.

Can I create my own OER?

An option for integrating OER into your course is creating your own materials. Creating your own materials doesn’t necessarily mean writing a textbook. Open Educational Resources can be any of the items below or some combination thereof.

  1. Written textbook
  2. Videos
  3. Curated articles that are openly licensed
  4. Podcasts
  5. Curated textbook chapters taken from openly licensed books
  6. Media you create yourself

As you begin searching, you may decide you want help creating and licensing your materials. As mentioned earlier, the library is a great place to start. The library may be able to offer significant resources to help you create your OER, so be sure to reach out and see what support is available.

Ready to get started?

If you’re interested in getting started on creating low or no-cost resources for your class or just want to get some more information, you can reach out to Carli Reinecke, the OER Librarian to get started with your project.

Resources

Inclusive and Equitable Access Models for Course Materials: Comparisons with OER and AER

This article is the second part to our post about Open Educational Resources and Affordable Educational Resources.

It is important to acknowledge a few other contenders in the push to lower textbook costs for students: Equitable Access and Inclusive Access. Equitable access replaces the costs of textbooks with a fee added to students’ tuition at the beginning of a term that covers the cost of all course materials for that semester, no matter the discipline. Students have the option to opt out and can apply their financial aid. The cost is the same for every student, which creates some concerns when you consider the cost difference between a low-material-cost humanities course and a science course with books that may cost hundreds of dollars. There is an expected course savings with this, as deals have been negotiated between the publishers, the bookstore, and the university. This is a textbook system that is becoming more popular with universities.

Inclusive Access is more common, as it focuses on just one or a few courses instead of all the courses a student is taking. Like Equitable Access, it is a service provided by publishers and college bookstores, marketed as a tool to lower costs for students. Inclusive Access involves a plan to provide eBooks to students for an entire course section, course, or department, depending on the agreements entered into by the publisher and a bookstore. The selected text is provided to all students by the first day of class and is typically paid for as a registration fee instead of a separate textbook cost. This can provide significant savings to students who would be likely to buy a new copy of the textbook, but savings are debatable for those who would acquire their texts by other means. Students can normally opt out, but penalties could arise for departments that sign on and don’t end up with enough student participation.

Please note that there are certain concerns with both the EA and IA models. The cost savings suggested for both models are often based on the difference between students participating in the program or students buying full-price textbooks, which is the only option available to students. There have also been concerns about the issue of these programs being opt-out instead of opt-in for students. The Department of Education is presently reviewing whether financial aid will cover these programs if they are opt-out only.

Similarities and differences between OER, AER, IA, and EA

Conditions OER AER IA EA
Free to students yes possibly no no
Free to university yes possibly yes** yes
Low cost yes yes possibly possibly
Copyright applies no possibly yes yes
Reduces equity gaps yes yes possibly possibly
Open to share with others yes no no no
Able to be remixed yes no no no
Available on the first day of class yes yes yes no
Potential for hidden fees no no yes yes
Students can get a hard copy of the book possibly possibly possibly possibly
Students can get a digital copy of the book yes yes yes yes
Students get their books at the bookstore possibly possibly yes yes
Ability to make modifications to the materials yes no no no
Might include scholarly articles found in the library databases no yes no no
Who benefits from the use students students publishers publishers

** Free as long as certain conditions are met.

The table above presents in tabular format the distinctions between OER, AER, EA, and IA throughout this toolbox article. It highlights how materials can be shared, how costs are passed on, and who benefits from the specific textbook arrangement.

Resources

Intro to Open Educational Resources and Affordable Educational Resources

What are Open Educational Resources (OER)?

You will find multiple definitions of OER, but for our use, we will focus on the 2017 definition from UNESCO, Open Educational Resources (OER) are teaching, learning, and research materials in any medium – digital or otherwise – that reside in the public domain or have been released under an open license that permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions.” While OER used to be difficult to find and were only available for the most common general education courses, more and more professors are creating the materials they would like for their classes and then offering those resources to other educators at no cost.

For materials to be considered Open Educational Resources, there is no cost to use the materials — either for the student, the faculty member, or the university involved. There is an alternative to this model, Affordable Educational Resources, which will be discussed below. Ideally, Open Educational Resources will have an open license. An open license has a 5Rs framework, as proposed by David Wiley. A completely open license allows the user to do all of the following:

  • Reuse: Share content as-is
  • Revise: Content can be adapted, revised, or modified
  • Remix: Content can be combined with other material to create new content
  • Redistribute: Share the original, revised, or remixed versions with others
  • Retain: the right to make, own, and control copies of the content

While open licenses are ideal, many faculty creators choose a Creative Commons license which offers some rights to those who choose to adopt the OER. Creative Commons licenses work alongside copyright, where it makes transparent to the user how they can use, modify, or distribute the work.

Why choose OER?

The most obvious reason for making the choice to switch to OER is the cost savings to students. As of Spring 2024, UWGB instructors have saved 1957 students $240,442. This represents 34 instructors who have participated in the formal OER program. In addition to this obvious benefit to students, the reasons instructors choose to use OER can be any or all of the following:

  • Increase equity
  • Allow for customization
  • Improve access to information
  • Avoid copyright issues
  • Increase representation and diversity in course resources
  • Meet changing learning outcomes

What’s the difference between OER and AER?

Open Educational Resources are explained in detail in the previous section. What we have not explored yet are Affordable Educational Resources (AER). Many of the textbooks our students are asked to purchase for their classes cost in the hundreds of dollars. Because of limited resources or lack of financial aid, students may attempt to take the course without purchasing the text or might not be able to get the text until weeks into the course. While affordable educational resources might have a cost associated, it is a much more manageable cost. AER means something different at every university. At UWGB, the cost is $50 or less for the total cost of resources purchased by the student for one class. There are other distinctions between OER and AER. This video explains them succinctly.

Want more information? Contact the UWGB OER librarian, Carli Reinecke.

Writing Effective Multiple-Choice Questions

Writing good multiple-choice questions is challenging. Tricky or verbose questions can reduce the test item’s reliability and validity, while a poor selection of answer choices can make a question either far too easy or incredibly difficult. A question that suffers several common pitfalls might even work against the learning outcomes it is trying to measure. Fortunately, researchers and assessment experts have identified some common guidelines for creating more equitable and reliable multiple-choice assessments. In this guide, we’ll walk through seven tips for writing more effective multiple-choice test items.

The scope of this guide is focused specifically on authoring multiple-choice questions. If you’d like to dig into when to use multiple-choice assessments, as well as recommendations for scaffolding, testing in online environments, and providing feedback, check out this other CATL blog post on general considerations for creating impactful multiple-choice assessments.

Before getting into the tips on writing questions, we’ll review the anatomy of a multiple-choice question and outline some common language that we use throughout this guide.

Table of Contents

The Anatomy of a Multiple-Choice Item

Throughout this article, we will use the following terms and definitions when referring to the parts of a multiple-choice question:

  • Item: A question and its answer choices as a unit
  • Stem: The posited question that respondents are asked to answer; often phrased as a question, but can also be a statement (e.g., fill in the blank)
  • Alternatives: A list of suggested answers that appear after the question stem; comprised of several incorrect answer options and one (or more) correct or best answer(s)
  • Distractor: An incorrect alternative

An example multiple-choice question in which the top portion is labelled as the “stem” and the answer choices A through F are labelled “alternatives,” with “A” serving as the answer and “B-F” serving as distractors

Example of a multiple-choice item, its stem, and the alternatives

Source: Vanderbilt Center for Teaching and Learning

Tips for Writing Effective Multiple-Choice Questions

Most of the recommendations in this guide have been adapted from How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty Simple (Burton et al, 1991) and Developing and Validating Multiple-choice Test Items (Haladyna, 2004). We’ve distilled down these long-form documents into a few simple guidelines that align with current recommendations from experts at other centers for teaching and learning (see “Additional Resources”). If you are interested in learning more about the research behind these suggestions, we encourage you to check out one or both of the resources linked above.

Tip #1: Tie each item to a learning outcome

In order to maximize an assessment’s validity and reliability, each multiple-choice item should be clearly aligned with one of the assessment’s learning outcomes (and, by extension, the course learning outcomes). Generally, it is recommended to have each item tied to only one outcome each. However, in the case of items that assess higher order thinking and present complex problems or scenarios, it is possible that a multiple-choice item may assess more than one outcome.

Tip #2: Create a specific, clear, and succinct stem

A straightforward, clear, and concise stem free from extraneous information increases a multiple-choice item’s reliability. When writing and revising your question stems, it is a good practice to ask yourself if there is a simpler or more direct way to rephrase a question. Overly wordy stems rely on students’ reading comprehension, which is usually not one of the intended outcomes of the assessment. Likewise, confusing or ambiguous stems can be accidentally misleading. Ideally, a student who has mastered the target outcome should be able to answer the question posited even without the alternatives present.

For millennia, humanity has been entranced by the ebb and flow of the tides. Many past civilizations believed the ocean's waters were controlled by monsters, spirits, or gods, but today know the scientific laws and theories that explain the tides. These movements are influenced by, in part, the gravitational force from the sun, the earth’s rotation, shoreline geography, and weather patterns, but all of these pale in comparison to the effects of:

  • a)  El Niño
  • b)  The gravitational force of the moon
  • c)  The ozone layer
  • d)  Deep-sea trenches

(Answer: B)

Why it doesn’t work: The extra information in the question stem makes it difficult for the test-taker to discern the question that is being posed. The question itself is also worded ambiguously.

Earth’s tides are influenced primarily by:

  • a)  El Niño
  • b)  The gravitational force of the moon
  • c)  The ozone layer
  • d)  Deep-sea trenches

(Answer: B)

Why it works: The question stem has been revised to remove all unnecessary information and it now poses a simple, straightforward question.

Tip #3: Avoid using negatives in question phrasing

It is usually best to avoid negative phrasing in question stems, such as asking students to identify which alternative does not belong. Negatively phrased stems tend to be less reliable in assessing students’ learning than stems that ask students to identify the correct answer. The exception to this guideline is in cases when knowing what not to do is key, such as questions related to safety protocols. If you do choose to include a negative qualifier, use bold or italics to emphasize the negative word and make sure that you don’t create a double negative with any of the alternatives.

Which of the following is not a quality of an active listener? 

  • a)  Not talking over others 
  • b)  Making eye contact with the speaker 
  • c)  Asking clarifying questions 
  • d)  Mentally planning a rebuttal while the other person is speaking 

(Answer: D)

Why it doesn’t work: The question stem is phrased in the negative and the negative qualifier is not emphasized, making the question less reliable. Additionally, one of the alternatives also contains the word “not,” creating a double negative with the question stem. 

True or false? An active listener…

  • Refrains from talking over others (T/F)
  • Makes eye contact with the speaker (T/F)
  • Asks clarifying questions (T/F)
  • Mentally plans a rebuttal while the other person is speaking (T/F)

(Answers: T, T, T, F)

Why it works: This question stem has been rephrased to avoid using the word “not.” The answer choices have been turned into four separate true/false statements so each item can be assessed separately and have been revised to remove the word “not.”

Which of the following is not a recommended action to protect yourself during an earthquake if you are inside a building?

  • a) Drop to your hands and knees
  • b) Take shelter under a sturdy nearby desk or table
  • c) Crawl to the nearest exit
  • d) Cover your head and neck with your arms

(Answer: C)

Why it works: In this scenario, knowing what not to do during an earthquake is one of the learning outcomes, so it is appropriate to use a negative qualifier. The negative qualifier in the stem, “not,” has also been emphasized with bold and italics to draw attention to it.

Tip #4: Use plausible distractors

Good distractors need to appear plausible to students that have not met the target learning outcome, but not so tricky that they could be argued as correct answers by a test-taker that has met the learning outcome. When you are writing a multiple-choice question it is often useful to write the stem first, then the correct answer first. Once you have decided on these two pieces, formulate 2-4 distractors based on common student misconceptions. If you can’t think of another “good” distractor for a set of alternatives, it is usually better to have fewer alternatives than to include extra alternatives just for the sake of consistency.

George Washington Carver is best known for his work as a(n) ______.

  • a)  Agricultural scientist
  • b)  Extraterrestrial expert
  • c)  Basket-weaver
  • d)  Juggler

(Answer: A)

Why it doesn’t work: The distractors are so absurd and far-removed from the topic of the question that even a student who knows nothing about George Washington Carver could discern the correct answer, making the test item neither reliable nor valid.

George Washington Carver is best known for his work as a(n) ______.

  • a)  Agricultural scientist
  • b)  Electrical engineer
  • c)  Microbiologist
  • d)  Politician

(Answer: A)

Why it works: The distractors seem plausible, creating a question that will more accurately assess students’ knowledge of George Washington Carver.

Tip #5: Use homogeneous phrasing and formatting for alternatives

Small typos, inconsistencies in tenses or phrasing, or changes in text formatting can accidentally provide clues about which alternatives are the distractors and which are correct answers. Savvy test-takers can pick up on these inconsistencies and use this information to deduce the correct answer even if they have not achieved mastery for the desired outcome, so keep an eye out for these things as you proofread your exam. If you notice formatting inconsistencies in your Canvas quizzes, you can use the Rich Content Editor to remove all formatting and set the selected text to Canvas’s defaults.

What three parts of speech can an adverb modify?

  • a)  Verbs, adjectives, and other adverbs
  • b)  Noun, adjective and preposition
  • c)  Verb, noun and conjunction
  • d)  Adjective, adverb and exclamation

(Answer: A)

Why it doesn’t work: Answer choice “A,” the correct answer, is in a different font from the other alternatives. Additionally, the distractors use the singular version of each part of speech, rather than the plural, and omit the Oxford comma before “and.” These inconsistencies hint to students that “A” is the odd one out.

What three parts of speech can an adverb modify?

  • a)  Verbs, adjectives, and other adverbs
  • b)  Nouns, adjectives, and prepositions
  • c)  Verbs, nouns, and conjunctions
  • d)  Adjectives, adverbs, and exclamations

(Answer: A)

Why it works: The distractors have been revised to look consistent with the correct answer, creating a question that assesses students’ knowledge of parts of speech, rather than their eye for detail.

Tip #6: Avoid using none-of-the-above or all-of-the-above as alternatives

Questions that provide “all of the above” or “none of the above” as alternatives are generally less reliable for assessing outcomes than a multiple-choice question with mutually exclusive alternatives. The table below outlines the use cases for “all of the above” and “none of the above” along with why they are flawed for reliable assessment in each instance. If you can’t think of another distractor while drafting a question, remember that it is okay for some questions to have fewer alternatives.

Use of “all of the above” and “none of the above” 

Alternative

Weakness

“All of the above” as the answer Can be identified by noting that two of the other alternatives are correct
“All of the above” as a distractor Can be eliminated by noting that one of the other alternatives is incorrect
“None of the above” as the answer Measures the ability to recognize incorrect answers rather than correct answers
“None of the above” as a distractor Does not appear plausible to some students

(Adapted from How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty, Brigham Young University)

Tip #7: Create questions with only one correct alternative

Like none- or all-of-the-above alternatives, asking students to identify multiple correct alternatives is a less reliable form of assessment than an item with only one correct answer. Multiple-response questions are also reliant on confusing grading calculations, since selecting an incorrect alternative “cancels out” a correct selection (this Canvas guide goes into more detail about how Multiple Answer questions are auto-graded). And, in questions with more incorrect than correct answers, students can still score points by selecting no answers at all!

A straightforward multiple-choice item with only one correct answer and mutually exclusive alternatives is a more reliable way of discerning whether a student truly knows a concept or is guessing. Another option is to turn a multiple-answer question into a series of true/false questions, which will provide a more reliable picture of students’ understanding and a more valid grade for their efforts.

Check all that apply. COVID-19:

  • a)  Is an infectious disease
  • b)  Is spread primarily through fungal spores
  • c)  Can be treated with antibiotics
  • d)  Can infect people of all ages

(Answer: A and D)

Why it doesn’t work: Because of the way multiple-response questions are graded, they are less reliable than individual multiple-choice or true/false questions.

True or false? COVID-19:

  • Is an infectious disease (T/F)
  • Is spread primarily through fungal spores (T/F)
  • Can be treated with antibiotics (T/F)
  • Can infect people of all ages (T/F)

(Answers: T, F, F, T)

Why it works: Each statement is assessed individually, allowing for more granular and accurate scoring.

Questions?

Want more tips for writing multiple-choice questions? Looking for someone to help brainstorm outcome-aligned questions with? CATL is here for you! Reach out any time to set up a meeting or send us your questions at CATL@uwgb.edu.

Additional Resources