PRH Consulting Inc. Blog

One of the biggest complaints in schools is that the “No Child Left Behind” act has set up standard test hurdles to be cleared by all schools and students. The idea seems sound — set a standard and then expect everyone to meet it. It allows teachers the freedom to vary their methods but, ultimately, they would be held accountable for results.

In practice, it doesn’t work that way. In the informal conversations I have had with people in the education business, this is seen as a wrong and short-sighted idea. If there is a test, “teachers will not worry about the kids learning…they will just teach to the test.” At its most absurd, an example of this would be, instead of teaching how to add (the core capability) teachers would teach how to add “5 + 7″ because that is on the test.

I would h0pe that the actual test content is not available to allow teachers to directly teach the test answers. But is there any harm in defining in advance the subjects and type of test questions (i.e., the capabilities) to be tested? In fact, that would seem to be the best way to get standardization.

One thing that muddies the waters though, is that teachers are accustomed to having almost no oversight. They are pretty much allowed to do whatever they want in their classroom. In a business, managers who run their organization well, meeting their goals and getting good employee feedback, are often left alone as well. But someone is looking at the results. The standardized tests are the results. So, if they are well-designed tests, they would show which teachers are getting the job done and which are not. Many get concerned because results have never been tracked and reported so publicly before..and anytime you introduce testing it is perceived as threatening because the performers are being asked to 1) trust that the tests are fair and 2) that the results will be used constructively. Often, that is a big leap to expect people to take.

As it turns out, it can be argued that the tests are not completely effective. Of course, to my knowledge, we haven’t taken the step to figure out exactly what we want students to be able to do when they get out of school. If you don’t know what you are shooting for, any attempt to measure whether you have hit the mark is a futile effort.

Another problem is more practical. Tests for large numbers of students are built for electronic grading — so every question has to be multiple choice or some other easy-to-grade format. If life were only multiple choice…it would be so much easier. But figuring out which of four (intentionally unambiguous) options is the right answer isn’t the same as have the capability to do something. It doesn’t prove that you know something other than how to use the process of elimination.

Another problem is that the tests are often administered poorly. In some cases, students with learning disabilities and IEPs (individual learning plans…which means they are intentionally NOT following the same sequence and pace as the standard) are still tested. In one case, students that could not read were forced to take the test but their aide was not allowed to read them the questions. So, they looked at the tests and randomly filled in circles…

Finally, it seems that the tests are intended to measure a minimum standard. But, due to the emphasis placed on them, they are in danger of becoming the actual goal. An effective teacher who is focusing on getting real and important learning to happen should produce students that blow through the simple standardized tests like a trained athlete would ace a basic physical. Administrators and parents wouldn’t need to fret so much about hitting the numbers. (Anecdote: We probably all know at least one teacher who has had an irate parent complain about a poor grade on an elementary school test impairing their child’s chance to get into Harvard. Any society that doesn’t see that as absurd should go slap itself.)

The fix though, is not to discard the tests and go back to the good ole days. The first step should be an analysis of the performance, followed by the design of tests that test capability (not the ability to guess multiple choice answers) related to the desired performance. Then, checkpoints should be designed to allow teachers to track how well they are progressing toward the standard. And, somehow, individual differences need to be accommodated so the test really measures, rather than blindly generating meaningless numbers for administrators to gloat over or fume about.

This is not dissimilar to a standard performance-based ISD approach that we (and many others) use to develop custom training programs. It’s frustrating to see a problem continue when the means to fix it is well understood and available.

Without some kind of test, there is no verification of capability. Performance testing is the most accurate but can be difficult to administer. We still have to decide though, is making it easier to process a large number of poor tests really a better solution?

What is a performance test?

A performance test is esssentially a checklist of key performance characteristics that define the criteria for successful performance. The checklist is used during observation of performance (or to review the result or output of performance) to assess whether the performance is acceptable. Performance tests can be used as a “gate” to determine whether performers are ready to “go solo” or simply as a way to verify capability (e.g., in a training course).

By contrast, a knowledge test attempts to assess the learner’s retention and recall of information or, occasionally, the application of rules.
 

Why I like performance tests.

Performance tests have several advantages over knowledge tests.

1. A performance test measures the right things. 

Assuming that your focus is on performance, a performance test is going to tell you what you should want to know. Specifically, it will tell you what people can do overall, and at a more granular level. It doesn’t tell you the learner “knows” the rules of the road…it tells you the learner can follow the rules of the road, steer and stop the car appropriately, use lanes correctly, follow traffic signals and signs, monitor other drivers’ movements, or any other criteria built into the test. As a benefit, it also tells you if they know the rules of the road based on whether or not they follow the rules in the course of performance!

2. A performance tests defines the work and the criteria for performance.

This is not trivial. In almost every case where we have developed performance tests (including work environments where there are detailed SOPs for every task) we have created new knowledge. That is, we have identified or clarified tasks or techniques or sequences that were missing or incorrect in existing documentation. Usually, the criteria for performance we define at the task level has not been previously documented. (In many cases, employees had figured these things out but they had not been communicated or standardized.) This is valuable to the business.

Clarifying performance requirements usually also simplifies the performance. It takes some of the mystique out of ”mastery” but makes it easier for all performers to perform effectively.
 

3. Performance tests connect training to performance.

The actual performance test instrument, as mentioned earlier, is typically a description of the work down to the task level. It includes criteria for successful performance that are as clear and objective as possible–instrument must be able to yield consistent results when used by multiple evaluators. In many cases, the performance test is used as a job aid by learners and a training by coaches and supervisors (in addition to being used for assessment).

4. If the performance test is done well, it is a more accurate test of capability than a knowledge test.

Actually performing almost always requires more than simple recall of information or even application of rules. It requires putting everything together in a real situation. That includes information, use of tools/resources, situational factors, and even “noise” in the environment. Performance often happens in “real time” where knowledge tests are usually off-line (or “stop time”). For example, knowing traffic laws (the “written test”) is not a good test for whether a teenager can actually drive, that is, can actually navigate through traffic and make good decisions in the moment. 

5. You don’t have to hide the answers.

A performance test, including the key performance criteria, can be published to anyone. With a performance test, just because you know what is expected doesn’t mean you can do it. So there is no need to hide or randomize the questions and answers. (This is why it can be used by learners as a job aid.)
 

6. (There is the potential to) get work done during the testing process.

In a business situation, the performance test can often be administered by a “master performer” (who has been trained/qualified to administer performance tests). So while the learner is being tested, he or she is actually doing real work. It may be at a slower rate and may require an additional resource (i.e., the master performer) to evaluate it more closely than normal, it is still resulting in output. And, you would hope that someone would be checking the work of any new (or unqualified) performer anyway so it is not an incremental increase in resource.

7. Managing learner expectations for “going solo.”

Instead of the learner “watching and learning” with the master performer for an undetermined period of time, a specific gate is identified and the learner, master performer, and supervisor will have a clear point in time for when the learner is ready to solo.