If you picked the first choice, consider the way we decide whether our students have learned science. We give them several hundred completely unrelated questions and ask them to pick the "right" answer from four confusingly similar ones. And we require that they do this in a time period carefully calibrated to be too short for useful reflection, much less serious problem solving.

We profess to value inquiry skills, but we test for recall of knowledge and mastery of superficial heuristics. This at a time when the National Science Education Standards assert that "rather than checking whether students have memorized certain items of information, assessments need to probe for students' understanding, reasoning, and the utilization of knowledge."

Sidney Harris

Why do we do this? Leaving aside whatever social forces may be at work to retain the status quo, it is undeniably difficult to assess reasoning and inquiry skills on a passive test. How do you gauge a student's ability to think when your only input is the answers to an endless list of questions? How does one reward curiosity on a multiple choice test? What we need is a way to create "active" assessments that engage students in authentic problem solving activities but can be easily scored. To do this, we must be able to monitor, record, and react to student actions, provide them with help when necessary, and confront them with probing questions in the context of a sustained, constructive task.

Here at The Concord Consortium we hope to change that. Well, not all at once, of course, but we think we may be able to take a first step.

For years now, the folks in our Modeling Center have been creating fun, computer-based environments that help students learn to "think like scientists" by giving them manipulable models of real objects. They can build rocket ships that travel at close to the speed of light, or fiddle with genes and watch an organism change. They can wonder how the speeding knight is going to squeeze his foreshortened lance between the doors of the barn, or try to guess the color of an invisible dragon by observing its offspring. These puzzles are hard, but not too hard. They take a long time to do, but not so long that kids get frustrated. They are fun, but not mindless fun - they reliably provide the "Aha!" experience that is the basis of all good teaching. They make marvelous assessment items.

GenScope™ is a manipulable model of genetics. It offers students representations of genetic information at multiple levels, from DNA to populations. It lets them manipulate this information and observe the effects of their interventions. It has been used successfully from middle school to college to help students reason about observable phenomena in terms of underlying causes that are not directly perceivable. (See @CONCORD Winter 1998 for a description of GenScope.)

Imagine using GenScope to determine whether a student has learned genetics. What would you do? One possibility would be to give the student a problem to solve (the one with the invisible dragon, say), set up GenScope, and then sit back and look over the kid's shoulder as she tries to solve the problem. You would be careful to note critical events (e.g., which particular dragons does she choose to cross with the invisible one?). You might offer hints (e.g., "You can look at the genes of the offspring, you know") and react to specific actions by posing questions ("Why did you choose to do that?"). When the student solves the problem you congratulate her and give her a related one that builds on the first. It might take a while, but at the end of a class period spent this way you have a pretty good idea of how much the student learned.

A group of us on the GenScope project have spent countless hours doing exactly this kind of evaluation. We have learned a lot from the experience. We have learned how to formulate the right sequence of problems, we have learned when to give hints, we have learned what questions to ask. But this approach to assessment is time consuming and much too labor intensive ever to constitute a viable alternative to, say, the SAT test. Can the process be automated in any way?

Assessment Conference An Invitational Conference on "Assessment in the Information Age" will be held June 21-22 at Shelburne Farms in Vermont, a lovely rural location convenient to the Burlington airport. The conference will bring together researchers in education, social psychology, computer science, psychometrics, and other disciplines to discuss the implications for educational assessment of advances in learning theory and information technology. Participants will focus on the questions "What changes do we expect in the next century in what we teach and how we teach it?" and "What do these changes imply for what we assess and how we assess it?" One of the desired outcomes of the conference is the creation of innovative proposals. For more information please contact: Paul Horwitz 978-371-5856 paul@concord.org

Enter BioLogica™

We are currently developing a new program, based on GenScope, that solves this problem. BioLogica™ is not a standalone application like GenScope. Rather, it consists of a collection of applets that run under the control of an executive program, which itself is controlled by scripts that can be written by researchers, curriculum developers, or teachers. (See @CONCORD Spring 1998 for a description of BioLogica™.)

A typical BioLogica™ script will pose a challenge to the students and then set up the BioLogica™ interface to match the challenge. It will make available particular applets, for example, set up the screen layout for them, and add menu items and tools that match the investigation. It will then monitor the students as they work using "listeners" - software agents embedded in the applets. This enables the script writer to record what the students are doing and to intervene at propitious moments. Thus, at various points in the investigation - for instance, when the students make the first cross - the script may offer hints, or provide feedback to the students' actions. It may alter the functioning of BioLogica™, for instance, prohibiting additional crosses until the students have answered a question. It can prompt the students to enter notes or screen shots into their personal portfolios, and it can record significant events automatically into the same portfolios. At particular "teachable moments" it can suggest that the students call their teacher over for a discussion.

Scripts are a powerful tool for producing active assessments. The assessment items can be as linear or as open ended as one might wish. They can be embedded in teaching activities, or presented by themselves. They can be programmed to record appropriate data and can then use that data to calculate an overall numeric score.

BioLogica™ is just now coming online and the process of writing scripts has only just begun. So we can't say, at this juncture, whether we will succeed in creating active assessments that really reflect a student's understanding and ability to reason about biology. We are like kids ourselves, kids who have been given a new, more powerful Lego set. We don't know what it can do yet, but we're eager to find out.

Paul Horwitz is the principal investigator for the GenScope and BioLogica™ projects. Paul@concord.org

Copyright © 1999 The Concord Consortium, All rights reserved. Last updated: 29-May-99
Questions and comments regarding this site can be sent to webmaster@concord.org