Do Actions Really Speak Louder than Words or Just Differently?

By Paul Horwitz

Say you’re interviewing an applicant for a job as an electronics technician and you want to know whether she knows how to use a multimeter—a common piece of test equipment that can measure voltages, currents, and resistances—but you don’t happen to have a multimeter on hand to test her. What do you do?

Figure 1. Simulated circuit and multimeter set up to measure the current through the resistor.

You could ask the person to explain to you how to use a multimeter to make some specific measurement. If you wanted a permanent record for your files, you could ask her to explain it in writing. But either method might discriminate against someone who knows perfectly well how to do the job, but is not very good at communicating how to do it to someone else. To get around that problem, you could show your applicant a few different setups and ask her which one she would use to make the measurement—in effect, a multiple-choice test for circuit measurement.

Would that work? Can you get the same information by asking multiple-choice questions that you get from observing someone do something?

To find out, we did an experiment. In February 2008, Tidewater Community College, located in Virginia Beach, VA, and one of our partner schools on the CAPA (Computer-Assisted Performance Assessment) project, was host to hundreds of students from local technical high schools as part of a nationwide observance of Engineering Week. We took advantage of the opportunity to work with 89 of these students. Using software developed on the CAPA project, we presented 46 of the students with the simulated circuit and multimeter shown in figure 1 and challenged them to measure the voltage across the resistor, the current through the resistor, and the resistance itself. The other 43 students were given a multiple-choice test that dealt with how to use a multimeter to make measurements. One of the items from that test is presented in figure 2. It deals with measuring the current through the resistor—which turned out to be the hardest of the three measurements, as we shall see.

How do you measure current?

A multimeter can measure voltage, current, or resistance 1 so one of the requirements for this task is that it be set to measure current—that is, function as an ammeter. The multiple-choice item presupposed that the meter would be set appropriately, but the performance assessment gave no guidance as to how to do this. Still, most of the students, it turned out, did accomplish this sub-task correctly. What they found more difficult was the correct placement of the probes in the circuit when the multimeter was serving as an ammeter. To see why, we must briefly delve into the electronics.

To measure the current through the resistor the multimeter must be placed in series so that the current passes through it. This means that it must become an integral part of the circuit that supplies the current. Normally, to measure current one “opens” the circuit somehow in order to place the multimeter directly in the flow. For our test, the students couldn’t do this (the simulation wouldn’t let them), but to make the measurement all they had to do to was open the switch and then bypass it with the multimeter (as shown in figure 1). And this setup is answer “C” on the multiple-choice question.

The current measurement is counter-intuitive because the leads of the multimeter have to be placed across the switch, rather than (as with the voltage and resistance measurements) across the resistor. Also, the switch must be open in order to force the current to pass through the ammeter. For both these reasons, we predicted from the start that the current measurement would be the hardest of the three, both for the multiple-choice test and for the performance assessment.

And we were right, but with a twist…

A surprise in the data

Of the 43 students who took our multiple-choice test, only 28% correctly answered the question about measuring current (the question depicted in figure 2). This is a score only slightly better than chance: if the students had just thrown darts at the test to determine their choices, they would have received a score of 25% since there were only four possible answers.

Figure 2. One item on multiple-choice test asking students how to use a multimeter to make measurements.

But they didn’t throw darts and they didn’t just guess randomly. How can we tell? Because their answers were by no means evenly distributed among the four possibilities. On the contrary, two of the four answers received 86% of the students’ “votes” and of these the most popular—chosen by more than half the students—was the one in which the multimeter’s leads were placed on either side of the resistance and the switch was closed—the appropriate setting for measuring voltage, but not current. This incorrect choice can confidently be attributed to a widely held misconception, not a guess.

So, what about the performance assessment? Before I let the cat out of the bag, what would you predict? Were the students who took the performance assessment, and actually measured the current (albeit using a simulation), more or less likely to succeed than were the students who answered that multiple-choice question? You may argue that the comparison isn’t a fair one, because the measurement calls for more initiative than the question. After all, in order to measure the current the student has to (a) set the multimeter to measure current, (b) open the switch, (c) place the leads on either side of the switch, (d) read off the measurement and write it in the answer box provided, and (e) pick the appropriate units from a pulldown menu. This certainly sounds harder than simply picking the correct answer out of a set of four.

But wait a minute! What if we were to score the performance assessment solely in terms of lead placement and switch position, the two steps called out in the multiple-choice item? Wouldn’t that “tilt” the contest toward the performance assessment? Remember, the student gets immediate feedback from the simulation, so if something doesn’t “feel right” about a measurement, he is free to repeat it until he thinks he’s got it right. 2 And what about the students who aren’t very verbal, but are “good with their hands”? Shouldn’t they do better on a hands-on assignment than one that involves reading words and parsing static diagrams?

When we put it to the test, here’s what happened: of the 46 students who took the performance assessment, only 4, or 9%, got both the leads and the switch position correct! Most of the rest set the circuit up for a voltage measurement—exactly the same mistake made by the students who answered the question on the multiple-choice test.

What does this all mean?

First of all, clearly these students weren’t very good at using a multimeter to measure current. If they had been applying for a “multimeter technician” job, most of them probably wouldn’t have been hired. And we see the same pattern whether they’re measuring the current with a simulation or merely answering a question about it. But it’s striking that only one-third as many students performed correctly on the simulation as were able to answer the question correctly. Since the two groups were chosen at random from all the students who were visiting the college that day, it’s reasonable to assume that if the conditions had been reversed, the results would have been the same.

If this finding is replicated in future trials, it could have important implications for the way we certify technicians. The majority of certification tests, used both as criteria for graduation and by employers evaluating new hires, are in multiple-choice format. Our research indicates that many students who receive a passing score on a multiple-choice test may not be able to do the job for which they’re being tested. Assuming that simulated assessments are a reasonable substitute for the “real thing,” we may be greatly overestimating the skills of the people we are graduating from technical schools. The data is still preliminary and we plan to repeat this “micro-experiment” soon, using more students and more complex assessment tasks. In the meantime, though, if someone tells you he knows how to use a multimeter, maybe you’d better have one handy. Or at least a computer.

Paul Horwitz (phorwitz@concord.org) directs the Computer-Assisted Performance Assessment project.