Archive for August, 2009

Why Performance Tests are Better Than Knowledge Tests

Tuesday, August 25th, 2009

What is a performance test?

A performance test is esssentially a checklist of key performance characteristics that define the criteria for successful performance. The checklist is used during observation of performance (or to review the result or output of performance) to assess whether the performance is acceptable. Performance tests can be used as a “gate” to determine whether performers are ready to “go solo” or simply as a way to verify capability (e.g., in a training course).

By contrast, a knowledge test attempts to assess the learner’s retention and recall of information or, occasionally, the application of rules.
 

 

Why I like performance tests.

Performance tests have several advantages over knowledge tests.

1. A performance test measures the right things. 

Assuming that your focus is on performance, a performance test is going to tell you what you should want to know. Specifically, it will tell you what people can do overall, and at a more granular level. It doesn’t tell you the learner “knows” the rules of the road…it tells you the learner can follow the rules of the road, steer and stop the car appropriately, use lanes correctly, follow traffic signals and signs, monitor other drivers’ movements, or any other criteria built into the test. As a benefit, it also tells you if they know the rules of the road based on whether or not they follow the rules in the course of performance!

2. A performance tests defines the work and the criteria for performance.

This is not trivial. In almost every case where we have developed performance tests (including work environments where there are detailed SOPs for every task) we have created new knowledge. That is, we have identified or clarified tasks or techniques or sequences that were missing or incorrect in existing documentation. Usually, the criteria for performance we define at the task level has not been previously documented. (In many cases, employees had figured these things out but they had not been communicated or standardized.) This is valuable to the business.

Clarifying performance requirements usually also simplifies the performance. It takes some of the mystique out of ”mastery” but makes it easier for all performers to perform effectively.
 

3. Performance tests connect training to performance.

The actual performance test instrument, as mentioned earlier, is typically a description of the work down to the task level. It includes criteria for successful performance that are as clear and objective as possible–instrument must be able to yield consistent results when used by multiple evaluators. In many cases, the performance test is used as a job aid by learners and a training by coaches and supervisors (in addition to being used for assessment).

4. If the performance test is done well, it is a more accurate test of capability than a knowledge test.

Actually performing almost always requires more than simple recall of information or even application of rules. It requires putting everything together in a real situation. That includes information, use of tools/resources, situational factors, and even “noise” in the environment. Performance often happens in “real time” where knowledge tests are usually off-line (or “stop time”). For example, knowing traffic laws (the “written test”) is not a good test for whether a teenager can actually drive, that is, can actually navigate through traffic and make good decisions in the moment. 

5. You don’t have to hide the answers.

A performance test, including the key performance criteria, can be published to anyone. With a performance test, just because you know what is expected doesn’t mean you can do it. So there is no need to hide or randomize the questions and answers. (This is why it can be used by learners as a job aid.)
 

6. (There is the potential to) get work done during the testing process.

In a business situation, the performance test can often be administered by a “master performer” (who has been trained/qualified to administer performance tests). So while the learner is being tested, he or she is actually doing real work. It may be at a slower rate and may require an additional resource (i.e., the master performer) to evaluate it more closely than normal, it is still resulting in output. And, you would hope that someone would be checking the work of any new (or unqualified) performer anyway so it is not an incremental increase in resource.

7. Managing learner expectations for “going solo.”

Instead of the learner “watching and learning” with the master performer for an undetermined period of time, a specific gate is identified and the learner, master performer, and supervisor will have a clear point in time for when the learner is ready to solo.

Paper vs. Electronic

Tuesday, August 25th, 2009

Originally posted on November 15, 2007 by Pete
One of the toughest decisions is whether to use paper or electronic tools for running meetings, managing to-do’s, even taking notes. Paper wins for speed, flexibility, and the ability to look at more of it at once…you can spread it out.

Post-Its

Electronic wins though for portability…and who doesn’t want to eliminate paper?

In general, we use paper for facilitating group work sessions because that is the only way to keep up. Using a computer to enter data creates a drag on the entire process, no matter how fluent the operator is. And it is next to impossible to facilitate the group if you have to look at the screen and think about your typing. And, by papering the walls, you have a visual point of reference to point to during the discussion as needed (as well as edit to incorporate later ideas).

PostItPhoto

For managing to-do’s, it is trickier. There are lots of good task management programs and, by using the computer (e.g., Outlook, Vitalist, etc.) you avoid having to re-write the same items when you cross off most of them but still have a few left. And you can sort by due date, by project, by context (e.g., calls, errands, office, internet, etc.). And, many of the programs work on your cellphone as well as your computer, so you have the information available at pretty much all times.

The most subtle challenge may be in creating output. I have gotten fairly fluent with the mainstream Microsoft Office Suite but somehow it is hard to create the first version of something using the computer. When I need to move quickly, handwriting the original and then having someone enter the data works the best…even if I have to tweak every page.

Probably the root question is organization. When creating output, you need to be able to capture and massage ideas as they are generated…which is not always the final sequence. Paper allows you to jump around, easily go back and make changes, get the gist of the content documented without bogging down to fix minor cosmetic details.

Finally, I had a client mention something I had never thought of…that the handwriten flipchart pages are more pleasant to look at! (Most of the production staff who have had the task of transcribing the data would probably disagree!) But maybe they are easier to look at than a computer or projector screen. Maybe the use of diagrams or multiple colors used make it easier to read/consume the information.

What I Don’t Like About GTD

Tuesday, August 25th, 2009
Originalily posted on November 1st, 2007 by Pete | Edit

Well actually, GTD is great. (GTD is a personal productivity system by David Allen…sort of like time management for the internet age.) I bought the book on CD and have listened to it multiple times. (Partly because one hearing was not enough–it is hard to grasp this kind of information by listening–but also because the information was useful.) I have implemented a bunch of the ideas.

But the very first listen flagged a problem for me and it still nags at me. The presumption is that we really can’t control how we spend our time (or, in GTD parlance, the actions we perform). Again, the system works pretty well because capturing everything makes it all visible and then you can “intuitively” make a decision “in the moment” about what you will actually work on. But it seems like this approach still puts you at the mercy of demands and requirements that you can’t control. Which, in many ways, is real. But that is the crux of why people want time management systems in the first place. So they can get out from under outside demands that cause stress and take you away from getting your goals met.

Ultimately, the danger of deciding “in the moment” and “renegotiating commitments” when we can’t meet them is that deadlines get missed. When you don’t plan ALL the steps you need to take to get you to the finish line, you are building in guaranteed future slippage. If everyone is overbooked and only worries about the next action, the entire organization will eventually grind to a halt when they look up at the deadline and only have half the actions completed.

In the end, the only options still end up with us needing to either do things faster or to do fewer things. The key is not to continuously renegotiate commitments (like an endless chain of continuances in a court case) but to make them more carefully in the first place. Ultimately, it means the job of leadership is to provide focus and to exclude unproductive activity so that the time and energy can be applied to meeting the organization’s goals. This is difficult. It is risky. But it is vitally important work that only leadership can do.

Disclaimer: David Allen does talk about project planning and commitment management. I’m pretty sure his focus on the next action is really a strategy to get people to stop talking about things and start doing them. We’ve all been in (usually large) companies where meetings are endless discussion of big picture ideas with little specific next steps to move things forward.

The “Real World” isn’t Simple

Tuesday, August 25th, 2009
Originally posted on April 18th, 2007 by Pete | Edit

One of the biggest complaints we often hear when we design a modular curriculum is that it is too complex.

First of all, every job I have analyzed is pretty complex so why wouldn’t a comprehensive curriculum to capture and distribute that know-how be complex also?

But PLEASE!! Listen to conversations between people who perform the jobs, especially the best performers. If you are uninitiated, you will hear an incomprehensible barrage of jargon, minutae, exceptions, esoteric rules, and assumed understanding of advanced technical concepts. Nothing wrong with that…in fact, you would hope that, after doing a job for a long time in “the big leagues” (i.e., large corporation) you would be an advanced professional. The question is, why ask people doing the training job to keep everything in layman’s terms? Even worse, in terms everyone can relate to from grade school?

The bottom line is that, when you analyze a job, you find that practitioners have a great deal of know-how. When you pile it up in one place, it looks overwhelming. But, they didn’t learn it like that any more than a company learns how to make transmissions or medicines or control systems in one month. That know-how is accumulated over time. But, if you were to try to start a competitive organization in a mature industry today, you need to identify and collect as much of that know-how right away. It would be an immense task so you wouldn’t try to do it in one or two months.

An effective strategy for building capability (e.g., curriculum, knowledge management system, competency system, etc.) has to deal with the complexity that is there in the “real world.” Technology always changes. People have widely varying natural capabilities and interests and backgrounds. Capability can be conveyed to an individual through formal training, on-the-job coaching, unguided experience (i.e., trial and error), by reading, and through tools (like references). That ends up being a large number of decisions so it does get complex. Unless you have a process for working through those decisions, it is easy to get lost and frustrated and, then, to decide that ”it should be easier.” Maybe it would be nice if it was, but it isn’t. 

Does Pressure Improve Performance?

Tuesday, August 25th, 2009
Originally posted on October 12th, 2007 by Pete | Edit

It is much easier to do something well when nobody is bugging you…isn’t it? Look at Rex Grossman, Bears (sometime) quarterback. Every mistake he makes is analyzed in the media until I would have to imagine he can’t do anything without second-guessing himself. If you watch other teams play, it turns out their quarterbacks throw interceptions and fumble occasionally as well.

On the other hand, if there is no oversight and no competition, it often creates an environment that performs horribly. The stereotype is the government office or monopoly business where you wait in interminable lines only to be told when you get to the window that you need a different form and to go wait in a different line.

Somewhere in the middle is probably the “sweet spot” for being both productive and supportive. This is really the challenge of line management. Too much competition and pressure leads to backstabbing, stealing credit for ideas, hoarding ideas, playing politics, and a host of things that make work a stressful nightmare for many.

But if every mistake is excused and management takes the blame for every problem (we didn’t provide good enough instructions, we didn’t plan well enough, they didn’t have access to the right information, responsibilities are unclear, etc.) performance degrades to the lowest common denominator.

Performance technology has a lot to say about pressure and performance.

For one thing, to perform effectively (which is the main point, true?) employees need to know what is expected, how it will be measured, and what will happen to them if they don’t meet or exceed expectations.

Another, the performers need access to the right environmental supports. The right tools. The information they need. Work processes that make sense.

And, the performers need to be capable of performing. They need to have the skills and knowledge they need to perform the tasks they are asked to execute.

If management can see that the above has been met, it seems fair for them to demand results. But they need to be within some subjective range of what is acceptable in the culture. In some workplaces, management can get in employees’ faces and in others that would be harassment. (I’m not advocating getting in people’s faces by the way.) But the culture is where fuzzy lines are drawn about how much pressure is enough and how much is too much. Some raises the bar. Too much frustrates and antagonizes employees.

At a call center we once worked with, there were hourly sales promotions where agents would push specific products and services and management would keep score. The highest-selling agents received points and prizes–very immediate feedback. But also an hourly re-set…if you did poorly in the morning, you still had a chance to win in the afternoon.

In a financial services business, security traders and trade processors have daily deadlines for completing trades and sending payment, etc. Failure to meet the deadline could result in fines to the business. Also, a high-pressure work setting. They normally get it done. Pressure, in this setting, results in performance.

Ultimately, it seems that pressure is a necessary evil. Creative and knowledge workers often lament deadlines and insufficient cycle time that “keep us from doing really do good work.” But, some of the best solutions we’ve come up with have been because we needed to…we had a deadline and, as a result, we were focussed, we eliminated distracting ideas quickly, and we found the shortest path. If you are a manager, ask yourself, are you applying the right amount of pressure in your organization to ensure performance?

Why Six Sigma is Annoying

Tuesday, August 25th, 2009
Posted on October 12th, 2007 by Pete | Edit

Ok…I am all for quality improvement and a systematic approach to in. So in a way, I am actually a fan of Six Sigma…but sometimes it just gets annoying.

1. For one thing, they get all the cool projects. Before there was Six Sigma, you could identify a way to improve things, figure out the cost/benefit to the company, and do the project. (In fact, Human Performance Technology was based on this approach and predates Six Sigma…it just didn’t get the visibility…) But now, with Six Sigma, in many companies if your business case is any good, your idea may end up being turned into a Six Sigma project. So some blackbelt gets all the glory and you go back to the same old same old.

2. Based on personal experience as well as comments from others, there are a lot of Six Sigma practitioners who really don’t know what they are doing. Clearly, many are quite competent. But just having the certification does not mean that you necessarily have the capability. Once, while doing a performance analysis meeting, a Six Sigma person observing from the back came up and said that he could see at least ten projects after observing just one day of our meeting. No kidding…we identified them!!! (Well, technically, the meeting participants, the master performers from the client organization, identified the opportunities…we just facilitated and documented their thinking.)
3. Companies may use Six Sigma as a way to avoid really paying attention to the details of the business operation. Instead, they designate people as Six Sigma experts and then figure process improvement is “handled” and they can “check it off.”
4. Fixation with process instead of results. A standard process only provides a logical template for actions. But, if you think it through, you may just as well come up with a sound process without having to add the extra structure and formality that Six Sigma may entail. Sometimes that structure helps. But sometimes it just creates needless bureaucracy.
Remember when process mapping came out and suddenly everybody had to map every process? Teams were running around mapping processes all over the place…instead of getting work done!! Six Sigma, like anything that becomes a fad or trend, is in danger of the same thing, as is lean manufacturing. Instead of improving the business, the risk is that you just add another layer of work processes and more overhead.
5. In most cases, Six Sigma teams find cost reductions. When used for innovation, there is less success. (Some would argue that targeting innovation is an inappropriate application for Six Sigma…but it happens.)
6. Of course the worst problem with Six Sigma is the same as happened with Y2K and many SAP initiatives–it drains budget and attention that could be used for other important consulting…like human performance improvement, curriculum design, training development!!

The Pace of Change

Tuesday, August 25th, 2009
Originally posted on March 30th, 2007 by Pete

If you think ideas and change are happening more quickly than they used to, you are probably right. On the Conference Board’s website is an article describing how the rate of change is impacting the business of managing organizations. 

“Ideas are circulating faster,” Clark says, “with the consequence that the lifespans of recent management fashions are considerably shorter than those for ideas which came to prominence in earlier periods; their peaks are much higher.” Research confirms this. A recent academic study found that the period of time between the introduction of a fashionable management idea or technique and the peak in its popularity has fallen from a mean average of 14.8 years in the 1950s through the 1970s, to 7.5 years in the 1980s and to 2.6 years in the 1990s.

 (For the entire article, click here http://www.conference-board.org/articles/atb_article.cfm?id=346&pg=4.)

This may mean that we are increasingly impatient for new approaches to deliver results, and when they don’t (or if they take too long) we abandon the approach.  But it may also mean that we have become so addicted to the new and novel that we are ready to jump on whatever the next bandwagon happens to be because we have to be the first in line.

For management consultants, this can result in continuous morphing of your identity. “TQM? Sure, we do that. I mean, Six Sigma? Yes, we have that.” For businesses, it can mean wasting a lot of time and effort training people, creating powerpoint decks, re-positioning initiatives, changing labels on things, and so forth. But little benefit. Running a business requires more than just applying the latest idea. You have to really understand your market, your technology, your value proposition. It isn’t often (ever?) simple.

Too Much Automation

Tuesday, August 25th, 2009
Originally posted on March 28th, 2007 by Pete

There was an article in Wall Street Journal recently about schools overusing automated phone message systems, resulting in some families getting up to five calls in the same night and some with no real information (e.g., “your child is a pleasure to have in class”)!! The desire to communicate but the ease of doing it en masse and automatically.

Even worse than automated calling out systems is the over-reliance on “knowledge bases” and voice recognition for incoming technical help call centers. There is nothing more futile than trying to get a quick answer to a system technical (usually computer-related) question. It will work like this:

1. Waste a bunch of time trying to find your problem in the “knowledge base” of pre-built questions and answers. It is great if they have your question but, if not, you better get comfortable.

2. Waste more time going on a user forum. These have a worse case of the same problem as the knowledge bases–the information is not organized so you have to use a random search and/or browsing strategy which will likely yeild either no results or the teasing promise of results just around the corner of the next click.

3. Waste more time trying to find a phone number to call an actual person. Once you find the number, you will not get a person who can answer on the first call.

You could always email for help on a non-urgent problem. The likely outcome will be an instant reply telling you to check the knowledge base!! (and probably a warning not to reply to that address…and probably a message asking you to take a survey to report how satisfied you were with how they handled your case).

The only thing worse is calling cellular directory assistance. Technology is great but, when you want an answer quickly, technology really can’t match the satisfaction of asking someone who knows. You can clarify, you can discuss other similar situations…in short, you can actually learn how something works so you can do it yourself next time. Using automation for basic, easily answered questions or for triage is fine but have a person who can help when you need it.