Article - Using Scenario-Based Questions as an Assessment Tool
NOVEMBER 12, 2013
In my first article in this series, I described the research-based support for the power of scenario-based questions. You can access that article by clicking here. In the second article, I shared tips on how to write great scenarios. You can access the second article by clicking here.
Why Scenario-Based Questions for Assessment?
Sharon Shrock and Bill Coscarelli, in their classic book, Criterion-Referenced Test Development, offered the clearest model for the potency of scenario-based questions. They essentially argued that there were only three acceptable ways to measure workplace learning using a testing methodology.
- Measure real-world on-the-job performance.
- Measure simulated real-world performance.
- Measure scenario-based decision making.
Note that real-world performance is still the best performance indicator—and well-designed simulations are likely to be better than well-designed scenarios. Still, scenario-based questions are a relatively good proxy for performance, especially if we are careful in choosing our background situations and our key decisions. They are also significantly less costly in terms of time, money, and effort.
Start with Learning and Situation Objectives
The first step is to get a clear list of the instructional objectives you want to test. Ideally, these should be compiled based on a comprehensive training needs analysis. Similarly, you’ll want to develop a list of situation objectives—the workplace situations for which your scenario-based decisions will be set. Make sure you prioritize your list. I once led a team that was building a simulation to teach basic business concepts. We crafted over 100 learning objectives. We couldn’t possibly ask the learners to go through that many scenarios, so we had to prioritize the learning objectives we did tackle. We ended up focusing on about 30 learning objectives over two simulations. Depending on your subject matter, you’re going to have to prioritize too!
Dealing with Your Subject-Matter Experts
Creating scenario-based questions is a ton of fun, but there can be some rough spots. Dealing with subject-matter experts (SMEs) is critically important—especially if you’re going to use your questions for some sort of assessment. Unfortunately, many SMEs focus on concepts and principles and are relatively weak in thinking about the Situation-Action contingencies required for successful decision making. In developing simulations for managers, the teams I worked with often had to spend several days with SMEs to reorient their thinking to Situations and Actions. You may need to do the same.
In using scenario-based questions for testing, you should seriously consider getting somebody—besides yourself—to validate that the information you’re testing is the most important stuff and that the concepts tested are accurately tested. The following are things you should consider having your SMEs help you validate:
- Your list of learning objectives.
- Your list of situation objectives.
- Your scenario-based questions.
The Question-Validation Process
First, you’ll want your SME to validate your list of learning objectives—and your prioritization of those objectives. Second, you’ll want your SMEs to approve the list of key situations your learners will face on the job. Finally, you’ll want to get your SMEs to validate your scenario-based questions.
The best way to validate your questions is to use a two-phase process with two separate sets of SMEs. After you develop your first set of questions—fulfilling all your targeted learning objectives—ask your first set of SMEs what they think of the questions and the answer choices. There are several ways to do this (I’ll discuss these in my upcoming workshop—see below). Once your SMEs are happy with your questions, you might consider cloning your questions before you validate them further. In the next section, I’ll talk about how to clone your questions. You might want to clone your questions to develop an item pool of questions covering the same learning objectives. You might want to use one set of cloned questions on a pretest, one set in a simulation or in training, and one set several weeks after the training ends as a refresher.
Once you’ve written all your questions, it’s time to REALLY validate them. You can do this by getting a fresh set of SMEs and having them ANSWER the questions (with no help, no discussion, etc.) just as if they were learners. By doing this, you’ll know whether your questions and their answer choices are really consistent with what experts think. In evaluating a blended learning program, I once utilized this procedure and found six SMEs to answer my questions. I only kept the questions where at least five of the SMEs chose the answers I had designated as “correct.”
Cloning humans may not be such a great idea. But cloning scenario-based questions is divine! First, it’s relatively easy to create a question by cloning it. You’ve already gone through the hard work of crafting a good question and getting it pre-validated. Second, having extra questions on the same learning point gives you a ton of great options. Here are some of the things you can do with cloned questions:
- Create item pools of questions (to prevent cheating, to provide variety, etc.)
- Create questions for pretests and posttests (to measure learning)
- Create questions for repetitions (to reinforce learning)
- Create questions for spaced repetitions (to support remembering)
- Create questions for pre-learning diagnostics (to diagnose where learners should focus their efforts)
- Create questions for learning (for prequestions before topic presentation, for post-learning questions to enable learners to get feedback on whether they clearly understand the concepts and how to apply them in decision making, for questions to generate discussions, etc.).
Cloning is relatively simple. All you have to do is change the “surface characteristics” of the scenario. So, for example, if your protagonist is John in the original scenario, John becomes Sally in the cloned scenario. If your protagonist works for Ajax Computers in the marketing department in the original scenario, she’ll work for Greeley Insurance in the finance department in the cloned scenario. Similarly, maybe the issue John wrangled with in the original scenario has to be tweaked slightly when Sally deals with it in the finance department. The key is to keep the base issue the same. So, for example, if John is dealing with a team member who has been underperforming, then Sally will also have to deal with a team member who is underperforming—but her team member will have to underperform in a different way than John’s team member.
The trickiest part about cloning a scenario-based question is cloning the answer choices. If you change them too much, you’re changing the likelihood that people will answer the question in the same way as the original. If you don’t change them enough, you might be providing hints (if another clone of the question has been seen before). In general, it’s best to have answer choices that have the same meaning as the originals, but are re-written using different wording.
Validating your questions using SMEs is generally a good idea. Sometimes, it’s even the law! Be especially careful if you decide to use scenario-based questions to determine hiring, promotion, or job placement. I’m no lawyer, but I’ve heard that it’s critical to have validation where job access is concerned.
This is the third article in this series regarding scenario-based questions. We’ve seen how research demonstrates the power of scenario-based questions. We’ve learned some secrets of writing great scenarios. And now we’ve been introduced to how to use scenario-based questions for assessment purposes.
Obviously, there’s a ton more to learn. I’ve been using scenario-based questions for all types of applications since 1986—and I’m still learning more each time I create another set of scenario-based questions. You’ll need to keep a long-time horizon in your heart if you want to become a great scenario-based question writer. I can attest that the rewards are worth it.
Scenario-based questions are one of the most potent tools in an instructional designer’s toolkit. With much less investment than high-production-value approaches like video or high-fidelity simulations, scenario-based questions can help learners gain deep understandings and see practical realities. By using them, we can support full comprehension, generate long-term remembering, and motivate our learners. We can also assess decision-making competence. And here’s a final secret. Writing great scenario-based questions is a way to create something beautiful and to feel that you are contributing to a rich artistic product. The best scenario-based questions resonate with aesthetic energy and sage truths about the human condition. You won’t always feel in this kind of flow—but it’s possible. With scenario-based questions, many things are possible!
To Learn More
Earlier Articles in this Series
The first article in this series provided background information on how and why scenario-based questions are so potent. You can access that article by clicking here. The second article in this series provided hints about how to write scenario-based questions. You can access that article by clicking here.
Comprehensive Workshop at the Immersive Learning University Conference
I will be teaching a one-day workshop as part of the Immersive Learning University Conference this January. Click here to sign up for this fantastic learning experience. This hands-on workshop is the best way to get started in writing great scenarios.
Contact Me to Schedule a Workshop for Your Company
If you can’t make it to the Immersive Learning University Conference in January or you want to schedule a workshop for your company, feel free to contact me at: firstname.lastname@example.org.
« Return to all articles