Teach to the Test

March 30th, 2010

One of the biggest complaints in schools is that the “No Child Left Behind” act has set up standard test hurdles to be cleared by all schools and students. The idea seems sound — set a standard and then expect everyone to meet it. It allows teachers the freedom to vary their methods but, ultimately, they would be held accountable for results.

In practice, it doesn’t work that way. In the informal conversations I have had with people in the education business, this is seen as a wrong and short-sighted idea. If there is a test, “teachers will not worry about the kids learning…they will just teach to the test.” At its most absurd, an example of this would be, instead of teaching how to add (the core capability) teachers would teach how to add “5 + 7″ because that is on the test.

I would h0pe that the actual test content is not available to allow teachers to directly teach the test answers. But is there any harm in defining in advance the subjects and type of test questions (i.e., the capabilities) to be tested? In fact, that would seem to be the best way to get standardization.

One thing that muddies the waters though, is that teachers are accustomed to having almost no oversight. They are pretty much allowed to do whatever they want in their classroom. In a business, managers who run their organization well, meeting their goals and getting good employee feedback, are often left alone as well. But someone is looking at the results. The standardized tests are the results. So, if they are well-designed tests, they would show which teachers are getting the job done and which are not. Many get concerned because results have never been tracked and reported so publicly before..and anytime you introduce testing it is perceived as threatening because the performers are being asked to 1) trust that the tests are fair and 2) that the results will be used constructively. Often, that is a big leap to expect people to take.

As it turns out, it can be argued that the tests are not completely effective. Of course, to my knowledge, we haven’t taken the step to figure out exactly what we want students to be able to do when they get out of school. If you don’t know what you are shooting for, any attempt to measure whether you have hit the mark is a futile effort.

Another problem is more practical. Tests for large numbers of students are built for electronic grading — so every question has to be multiple choice or some other easy-to-grade format. If life were only multiple choice…it would be so much easier. But figuring out which of four (intentionally unambiguous) options is the right answer isn’t the same as have the capability to do something. It doesn’t prove that you know something other than how to use the process of elimination.

Another problem is that the tests are often administered poorly. In some cases, students with learning disabilities and IEPs (individual learning plans…which means they are intentionally NOT following the same sequence and pace as the standard) are still tested. In one case, students that could not read were forced to take the test but their aide was not allowed to read them the questions. So, they looked at the tests and randomly filled in circles…

Finally, it seems that the tests are intended to measure a minimum standard. But, due to the emphasis placed on them, they are in danger of becoming the actual goal. An effective teacher who is focusing on getting real and important learning to happen should produce students that blow through the simple standardized tests like a trained athlete would ace a basic physical. Administrators and parents wouldn’t need to fret so much about hitting the numbers. (Anecdote: We probably all know at least one teacher who has had an irate parent complain about a poor grade on an elementary school test impairing their child’s chance to get into Harvard. Any society that doesn’t see that as absurd should go slap itself.)

The fix though, is not to discard the tests and go back to the good ole days. The first step should be an analysis of the performance, followed by the design of tests that test capability (not the ability to guess multiple choice answers) related to the desired performance. Then, checkpoints should be designed to allow teachers to track how well they are progressing toward the standard. And, somehow, individual differences need to be accommodated so the test really measures, rather than blindly generating meaningless numbers for administrators to gloat over or fume about.

This is not dissimilar to a standard performance-based ISD approach that we (and many others) use to develop custom training programs. It’s frustrating to see a problem continue when the means to fix it is well understood and available.

Without some kind of test, there is no verification of capability. Performance testing is the most accurate but can be difficult to administer. We still have to decide though, is making it easier to process a large number of poor tests really a better solution?

Excellent Service

March 5th, 2010

I recently had a problem with my computer. A client I was working with said it was a D620 error. I didn’t get it at first but then realized my computer is a Dell Latitude D620. Apparently, this model has had a problem or two…my client’s company actually exchanged a bunch of them when they first came out due to problems.

My company doesn’t have that kind of leverage with Dell though and the computer was three years old when I started having the problem. In fact, I had a three-year warranty and the problem started about a month before the warranty expired. (Of course, I was too busy to deal with it until after the warranty expired but that’s my own fault…)

I was prepared for the worst…which, in this case, would be hours of troubleshooting, some new parts, some frustration trying to get the fix to work, followed eventually by the purchase of a new laptop. But, for once, that is NOT what happened.

Instead, I got a first-hand look at how Dell does service and it was impressive. Dell’s reputation can be spotty, depending on who you talk to. But they are clearly putting a focus on customer satisfaction and service.

First, I tried to chat with a service support person and right after I typed the problem symptoms and the steps I had taken to correct the problem into the chat window, the chat was ended. “I knew it…they couldn’t solve it so the ‘hung up’ on me!” I wasn’t happy but was smart enough to copy my information so I wouldn’t have to re-type it.

Figuring I would try again, I re-logged in but during the process, my phone rang. Picking up I found it was the service support rep I had been chatting with! How about that!

She did some troubleshooting on line and then had me run some diagnostics and called back to check the results. Bad news…I needed some new parts. The motherboard for one thing. (Which sounded like a big deal.)

After looking at some alternatives, she offered sell me an extended warranty which would cover on-site service/installation of the new parts and a year of accident protection for less than half the price of the motherboard (which was not the only part). My suspicions were raised “what’s the catch?” but I went for it.

Long story short…the support rep called to confirm the parts were shipped. A dispatcher called to let me know a tech would be calling me the next morning for an appointment. The next morning the service tech called to schedule a time. In the afternoon, he came to my office and installed about four separate parts (completely dis-assembling the laptop and re-assembling it in about 30 minutes). And, so far, everything works!

What was so great about that service? First of all, the results. If it didn’t work, nothing else would matter. If you are going to provide any service you have to be competent.

Secondly, the response time. I first gave up my superstitious troubleshooting methods (“maybe if I jiggle this cable while touching the monitor…”) and went online for help on a Wednesday morning. The parts were shipped that night. The tech had them the next morning and was in my office on Thursday afternoon.

Third, the value. This all cost a little more than $200! (Plus, if I drop my computer before next March, I can get it fixed again…) This is a three-year old laptop running Windows XP! But, since we have been in business (2002) we have purchased all our IT equipment from Dell (OK, not cellphones or printers). Based on this experience, we will continue to. The value to Dell is customer loyalty.

Fourth, the communication. This one is almost cliche. Communicate your commitments and then meet them. Explain the process to the customer so they know what is going to happen next. Confirm that the customer is satisfied before moving on to the next task. Fundamental customer service is not hard to learn…but it is still not easy to implement.

This level of service is actually a little bit inspiring. I found myself looking at my own business and asking “how would our service be perceived?” Are we adequate? Are we excellent? Where should we make improvements? I made some notes.

How Long Does it Take to Become an Expert?

February 18th, 2010

The answer is simple — 10,000 hours, or about 10 years. In his book, “This is Your Brain on Music,” Daniel Levitin summarized the  results of a study by Anders Ericsson (Florida State). Ericsson researched a number of types of experts, from chess players to musicians. His team defined “expert” as someone who has achieved a high degree of accomplishment relative to other people. That means the designation is somewhat subjective and has a social element. (Maybe it also means that you have to outlast some of your competitors!)

We did the math and figure that it is roughly equivalent to 3 hours a day for 10 years. What does that mean? For one thing, companies that move people from one position to another every two years may be developing experts in the organization and the industry but not at performing a specific role or task. It is interesting to note that jobs requiring a high level of individual performance (such as doctors, attorneys, airline pilots) have practitioners that remain in their roles longer than the average corporate manager. And, in these roles, more complex tasks are carefully limited to people with more time on the job.

Another question that immediately came to mind was “what if you work on it six hours a day for five years…can you get to expert more quickly?”

Of course, as a performance improvement expert (I can say that, by the way…I have been doing this work since 1984) my real first thought was “I bet we could beat that.” And we probably could. Much of what we do is observe and capture mastery performance and then distill that expertise into process, tools, and information. We sometimes even find shortcuts.

Still, if we needed heart surgery, we would pick the ten year veteran over the one-year novice with even if they have a really good job aid. Why?

Maybe because the longer timeframe allows more reflection. Why did something work better? What could we have done differently? We build our training programs to include exercises and even simulations so that learners apply what they learn in as realistic a situation as possible in a training setting. Sometimes, we even prescribe on-the-job practice and assessment. All of this compresses some of the experience learners would otherwise pick up slowly over the years. But it isn’t entirely equivalent.

There is neurological research that shows expertise depends on well-developed neural paths. Meaning that if you do something frequently and reflect on it, you will develop more expertise…you will sort of wear paths into your brain. But it takes time.

Organizationally, the ten year cycle can create a problem if you need to retain these experts. They get expensive after ten years of raises and accumulated vacation. Their market value increases. The smart thing to do is identify where you really need them and then make sure you leverage that know-how. Capture as much of their expertise as possible in reference materials and tools. Engage them in developing the next generation of experts. Respect their contributions. Plan for succession.

Often people who have been in a role a long time aren’t really valued for their expertise…they can be taken for granted as an old-timer whose advancement peaked. But maybe they decided to focus. Maybe they enjoy what they are doing. In this period of cost-cutting and outsourcing, it can be too easy to lose track of why people work…it may not be only about the money.

Practice Makes…Really, Really Good

February 12th, 2010

Recently I was reading about what it takes to become an expert….also, about what differentiates a good performer and an exceptional performer. In a previous post, I mentioned the estimate that it takes about 10,000 hours, or about 3.5 hrs a day over 10 yes, to become an expert.

There was another difference mentioned in Resnak’s book, that exceptional performers will practice things they are already good at. He describes a few cases from different areas (sports, music) and notes that these top-performers resist anything in their performance becoming “automatic.” One example is Michael Jordan practicing ball-handling! Resnak argues that this focus on very small parts of the performance, continual practice and improvement, and never taking anything as good enough, is what makes the difference between good and great.

As a performance consultant, I had to think about that. Much of what we do is to simplify the performance. Strangely, that usually requires looking closely at the performance to find the specific skills or criteria needed as opposed to just trying to think about everything. The focus for a new learner would naturally be different from the experienced professional. But often, if we are working with experienced performers, we hear the master performers and managers advocate “getting back to the fundamentals” while the bulk of the audience wants to move on past that point.

Questions:

  1. Is it realistic, is it worthwhile, to force performers to drill on the fundamentals? If so, how do you get them to do it?
  2. The “greats” are few and far between. What can we do to spread or reinforce that drive to achieve excellence to a larger number of employees? Or, is the drive to become great only internal to the individual performer? (Or, is it so much a part of the individual’s personality that it is either too late or inappropriate to select it as a target for improvement by management or training events?)

My initial thoughts about both questions are that if you want to focus on the fundamentals, they need to be part of the culture and reinforced by both the formal and informal leadership. If Michael Jordan is practicing ball-handling, chances are, the other players will feel OK about it too.

Another aspect is rewards. In professional sports, there are huge rewards for being the best. Performance is tracked, measured, and analyzed constantly. Most employees really aren’t trying to be great…they just want to to their jobs. And employees can be brilliant or average and still end up with only a limited percent increase in pay (if any). So, if management wants people to put in the effort, they have to do some thinking about how they can identify and reward their top performers besides pay raises and bonuses. The good news is that, in many cases, financial rewards for performance are not really very effective…they are good for short-term improvement but less so for long-term.

So the other piece of good news is that energy, attitude, and ideas are pretty much free. They are out there. Those that try harder can become the best…if they really want to do the work.

The End of the Newsletter

October 26th, 2009

Well, we are getting ready to publish our next and last newsletter “Building Capability.” We’ve been publishing the newsletter since we started the business and it’s been both fun and informative (well, we like to think so anyway…).

But it has been difficult to publish consistently — when you get busy, you need to hit the client’s deadlines first. And much of the time creating the newsletter is spent on wordsmithing to fit the available space…not exactly high value add.

On top of that, the world has been changing. More and more communications are shifting to bite-size. Smaller, more frequent communications fit the more busy daily schedules of many professionals today. In fact, we often thought that our newsletter, built for printing, reading on a plane, leaving on a table in a library or breakroom, etc., might be attractive to people who get tired of reading from a computer screen.

But, printing isn’t “green” and publishing every quarter isn’t frequent enough (assumimg we could keep up with that schedule). So, we’re going to try the new communication model, try some new media (like digital video occasionally), and go for short, focussed email “blasts.”

Of course, now that it is working again, we will continue to post on our blog as well. We’re not planning a Facebook site though…maybe in a few years…

Why Performance Tests are Better Than Knowledge Tests

August 25th, 2009

What is a performance test?

A performance test is esssentially a checklist of key performance characteristics that define the criteria for successful performance. The checklist is used during observation of performance (or to review the result or output of performance) to assess whether the performance is acceptable. Performance tests can be used as a “gate” to determine whether performers are ready to “go solo” or simply as a way to verify capability (e.g., in a training course).

By contrast, a knowledge test attempts to assess the learner’s retention and recall of information or, occasionally, the application of rules.
 

 

Why I like performance tests.

Performance tests have several advantages over knowledge tests.

1. A performance test measures the right things. 

Assuming that your focus is on performance, a performance test is going to tell you what you should want to know. Specifically, it will tell you what people can do overall, and at a more granular level. It doesn’t tell you the learner “knows” the rules of the road…it tells you the learner can follow the rules of the road, steer and stop the car appropriately, use lanes correctly, follow traffic signals and signs, monitor other drivers’ movements, or any other criteria built into the test. As a benefit, it also tells you if they know the rules of the road based on whether or not they follow the rules in the course of performance!

2. A performance tests defines the work and the criteria for performance.

This is not trivial. In almost every case where we have developed performance tests (including work environments where there are detailed SOPs for every task) we have created new knowledge. That is, we have identified or clarified tasks or techniques or sequences that were missing or incorrect in existing documentation. Usually, the criteria for performance we define at the task level has not been previously documented. (In many cases, employees had figured these things out but they had not been communicated or standardized.) This is valuable to the business.

Clarifying performance requirements usually also simplifies the performance. It takes some of the mystique out of ”mastery” but makes it easier for all performers to perform effectively.
 

3. Performance tests connect training to performance.

The actual performance test instrument, as mentioned earlier, is typically a description of the work down to the task level. It includes criteria for successful performance that are as clear and objective as possible–instrument must be able to yield consistent results when used by multiple evaluators. In many cases, the performance test is used as a job aid by learners and a training by coaches and supervisors (in addition to being used for assessment).

4. If the performance test is done well, it is a more accurate test of capability than a knowledge test.

Actually performing almost always requires more than simple recall of information or even application of rules. It requires putting everything together in a real situation. That includes information, use of tools/resources, situational factors, and even “noise” in the environment. Performance often happens in “real time” where knowledge tests are usually off-line (or “stop time”). For example, knowing traffic laws (the “written test”) is not a good test for whether a teenager can actually drive, that is, can actually navigate through traffic and make good decisions in the moment. 

5. You don’t have to hide the answers.

A performance test, including the key performance criteria, can be published to anyone. With a performance test, just because you know what is expected doesn’t mean you can do it. So there is no need to hide or randomize the questions and answers. (This is why it can be used by learners as a job aid.)
 

6. (There is the potential to) get work done during the testing process.

In a business situation, the performance test can often be administered by a “master performer” (who has been trained/qualified to administer performance tests). So while the learner is being tested, he or she is actually doing real work. It may be at a slower rate and may require an additional resource (i.e., the master performer) to evaluate it more closely than normal, it is still resulting in output. And, you would hope that someone would be checking the work of any new (or unqualified) performer anyway so it is not an incremental increase in resource.

7. Managing learner expectations for “going solo.”

Instead of the learner “watching and learning” with the master performer for an undetermined period of time, a specific gate is identified and the learner, master performer, and supervisor will have a clear point in time for when the learner is ready to solo.

Paper vs. Electronic

August 25th, 2009

Originally posted on November 15, 2007 by Pete
One of the toughest decisions is whether to use paper or electronic tools for running meetings, managing to-do’s, even taking notes. Paper wins for speed, flexibility, and the ability to look at more of it at once…you can spread it out.

Post-Its

Electronic wins though for portability…and who doesn’t want to eliminate paper?

In general, we use paper for facilitating group work sessions because that is the only way to keep up. Using a computer to enter data creates a drag on the entire process, no matter how fluent the operator is. And it is next to impossible to facilitate the group if you have to look at the screen and think about your typing. And, by papering the walls, you have a visual point of reference to point to during the discussion as needed (as well as edit to incorporate later ideas).

PostItPhoto

For managing to-do’s, it is trickier. There are lots of good task management programs and, by using the computer (e.g., Outlook, Vitalist, etc.) you avoid having to re-write the same items when you cross off most of them but still have a few left. And you can sort by due date, by project, by context (e.g., calls, errands, office, internet, etc.). And, many of the programs work on your cellphone as well as your computer, so you have the information available at pretty much all times.

The most subtle challenge may be in creating output. I have gotten fairly fluent with the mainstream Microsoft Office Suite but somehow it is hard to create the first version of something using the computer. When I need to move quickly, handwriting the original and then having someone enter the data works the best…even if I have to tweak every page.

Probably the root question is organization. When creating output, you need to be able to capture and massage ideas as they are generated…which is not always the final sequence. Paper allows you to jump around, easily go back and make changes, get the gist of the content documented without bogging down to fix minor cosmetic details.

Finally, I had a client mention something I had never thought of…that the handwriten flipchart pages are more pleasant to look at! (Most of the production staff who have had the task of transcribing the data would probably disagree!) But maybe they are easier to look at than a computer or projector screen. Maybe the use of diagrams or multiple colors used make it easier to read/consume the information.