A GUIDE TO IDENTIFYING GOOD CONTENT

Welcome to the wonderful world of training content. It seems like vendors are popping up everywhere, especially in the web-based training market. Deciding which content to buy and which to avoid can be as confusing as buying a used car, and unfortunately, it's just as easy to buy a lemon. And just as with buying a used car, you might not know you have a lemon until you've used the content for a while. Now for the good news: whether you're in the market for Instructor-Led, Computer-Based, Web-Based or any other type of training materials, there are a few things you can look for that are more reliable than kicking the tires.

Anybody can design and develop training (NOT!)
Lots of people think they can design and develop training, but in reality, quality instructional design takes education, experience and skills the average person does not have. Unfortunately, there are a lot of training vendors out there that don't have a single Instructional Designer, or ID, involved in the development process. That's a little like having a plumber build your house. Sure, the plumber has seen a lot of houses being built, but he's not a carpenter. I wouldn't want to be there when the building inspector shows up to inspect the framing. So, one of the first things to look for when deciding what content to purchase is: "who designed it?"

Tied closely to instructional design is subject matter accuracy and level of content. In many cases, the ID is not an expert in the subject matter. You've heard the old saying "jack of all trades, master of none". This is often true for an ID. The ID must rely on people who know the subject matter inside and out. These people are typically called Subject Matter Experts, or SME. The ID and the SME should work as a team to determine what to teach, and how to teach it. The SME usually brings first hand, workplace knowledge and experience to the mix. In fact, in many companies, the majority of training is designed and delivered by trainers who are also SME, and who have received some training in basic instructional design and delivery. There are two important areas of content where the SME usually contributes the most: content accuracy, and level of content. Level of content is very important in a training environment because training costs the employer money. Employers must find the balance between ensuring the work force is adequately trained and not spending too much time and money on training. Employers don't want their employees to attend training that is not relevant or necessary. The next question to ask the vendor: "who was the subject matter expert?"

Is there an objective to all this?
Like most things in life, developing good training requires good planning. There are many different tried and true methods for designing and developing training, but most of them have a common thread. When you're looking at training content, you should be able to easily see that thread. Look for three basic attributes: 1) desired outcomes, 2) a means of confidently measuring achievement, and 3) a means of enabling students to achieve the desired outcomes.

Desired outcomes can take many different forms, but in most common training programs they're in the form of goals and objectives. Goals and objectives can cover a wide spectrum of outcomes. For instance, an employer might need an employee to memorize the steps of a procedure, or recognize different colors associated with safety signs. Memorization and recognition are usually considered low level learning objectives. On the other hand, an employer might need an employee to gather evidence at an accident scene and use sophisticated analyses to determine the root causes of the accident. This is a high level objective.

I could spend a lot of time covering the proper way to develop and write training objectives, but for our purposes here, I'll narrow it down considerably. Whether a training program has been designed to help students achieve high level objectives, or low level objectives, or any level in between, the objectives for the program should all have the same characteristics. Objectives should be stated as simply as appropriate for the intended audience, using language and terms that the student should understand. Objectives should be specific and unambiguous, so that there is no question as to the intended outcome. Above all, objectives must be measurable. Training objectives are no different than any other goals we set. We need to be able to figure out if we got there.

"Are we there, yet?"
Anyone who has ever traveled with small children has probably heard that question more than they'd like. In training, the answer to that question is often "close enough". Most workplace training applications use evaluation tools, such as practical evaluations and examinations, to determine if a student has reached an acceptable level of skill or knowledge to be safe and functional in the workplace. In order to be effective, training objectives and evaluations must fit like hand and glove. The objective tells the student what must be achieved and the evaluation tells the student if it was achieved. When you're looking for quality training content, compare the training objectives to the evaluation tool. Make sure each and every objective is measured by the evaluation. Make sure the objectives are measured in proportion to the importance of the objective. For instance, an objective that requires the student to "list the primary colors" is probably less important than an objective that requires the student to "list the color of wire to cut that will defuse this bomb". For the latter, you will want to be VERY sure the student demonstrates the proper knowledge.

Besides making sure the objectives are measured, you will want to make sure the objectives are measured properly. For instance, it is almost impossible to measure a person's physical skills using a multiple-choice examination. You would also not want to waste time and money measuring simple knowledge objectives with elaborate simulations. Be a smart shopper, don't pay for things you don't need.

If you are looking at training content that is primarily knowledge-based (i.e., little or no practical skills), and evaluation is accomplished with examinations or tests, make sure there are enough questions in the test bank. For Instructor-Led training, there should be enough questions to create three or four tests with at least a 50% difference in questions from test to test. If you're reviewing computer-based or web-based training programs, make sure the tests are randomized and there are enough test items so that you're assured of a significant difference from test to test. I recommend there be at least twice as many questions in each objective question pool as the number of questions that will be asked for that objective on each test.

The bottom line: vendors should be able to easily prove that their evaluations confidently measure student achievement against established learning objectives.

"Where's the beef?"
Let's see, what was this article supposed to be about? Oh yeah, choosing the best content! We've finally arrived at the meat of the subject. Up to this point we've analyzed the situation and determined the outcomes we'd like to achieve with our training. We've looked for a program with good, measurable objectives and we've made sure the program has a proper evaluation matched to the objectives. Now all we have to do is make sure the content matches the objectives and evaluation. Sounds simple enough, and if the content was well designed and developed, it will be simple. Remember way back in the beginning when I said that the ID and the SME need to work together as a team. This is where it really shows. Good content design is a balancing act between "not enough" and "too much". In the training world, unbalanced content usually translates into extra cost for the employer. In the table, I've listed four common attributes of training content and the possible effects of "not enough" or "too much".

Deciding whether there is too much or too little of these attributes is not easy. The answer often varies depending on the target audience. When I'm discussing the decision making process for training programs, I'm always reminded of a training class I attended several years ago. It was an eight hour instructor-led program. The instructor was very dynamic. He used every trick in the instructor's handbook and more. He told stories. He acted out little skits. He told jokes. He almost forced the students to participate. In my opinion, he was a little over the top, but he certainly kept everyone riveted throughout the class. Of course, he received rave reviews, with many students saying it was the best class they'd ever taken. After the class, something kept nagging at me. How much had I learned? When you got past the flash, the actual content of the program could easily have been taught in a couple of hours. The lesson I learned from this is that teachers love to teach, trainers love to train. Training content should be designed to give the student the best chance to achieve the objectives of the training program without overkill. Too much content, interactivity, or engagement can be almost as bad as too little, and they both end up costing the employer.

Since technical accuracy is a must for any training, I won't dwell on it here, but keep in mind that YOU are ultimately responsible for the training you provide to your employees. Most vendors will make a good faith effort to keep their content current and accurate. You should keep track of events, such as changes to state or federal regulations, which might affect your training. Make sure any necessary changes are made to your programs.

“Tell me and I’ll forget. Show me, and I may remember. Involve me, and I’ll understand” (Native American Proverb)
The ultimate goal of most training programs is to make sure employees have the skills and knowledge they need to be safe and productive on the job. Good evaluations based on good objectives give you confidence that your employees have an acceptable level of achievement at that time. What about six or eight months down the road? What about two weeks? Obviously, retention of learned skills and knowledge is very important. Interactivity and engagement are training program attributes that can greatly affect retention. A good training program will have activities at key times that require the student to demonstrate or practice relevant skills or re-enforce knowledge. These activities can be as simple as questions or as complicated as full-blown simulations. When we develop training programs, we follow a simple philosophy: at least one interactive activity per training objective. More important objectives might have several activities.

Activities don't have to be difficult to be effective, but they do have to be relevant and support the applicable objective. Especially in computer-based or web-based training, make sure the objective of the activity doesn't get lost in the design of the activity. For instance, we once designed a training program where the objective was to familiarize the student with a series of warning signs and the potential hazards each sign represented. We designed a "concentration" style game where the object was to match signs and reveal the puzzle underneath. While the activity was interesting to program and fun for the students, we soon realized that it was not helping the students achieve the objective. After the activity was completed, the students still couldn't recognize the signs and match them to the hazards. The lesson: interactivity must be used to help students achieve the objectives, not just to add interactivity.

Last, but not least, is engagement. This one can be very tricky. Most of today's workforce grew up with easy access to entertainment. There has been a steady progression from AM radio to the Internet. It seems to take more and more sophisticated programs to keep students engaged because their expectations are so high. I've been involved in the design and development of hundreds of training classes over the last twenty years and I can tell you that designing engagement is almost always the hardest part of the project. We've used comedy, drama, emotion, games, history, music, friendly characters, and more. Of course, the real problem is people themselves. Different things are engaging to different people. Some people are even angered by things that other people find engaging. On one project in particular, we created a humorous medieval theme, complete with knights, monks, damsels, and even horses. We received so much feedback from the students, we decided to do a survey. Surprisingly, the results were almost evenly split; half the students loved it, half hated it, and there didn't appear to be a middle ground. The half that loved it generally said it was very engaging and fun and the half that hated it generally said they thought it was a waste of time and money. Of course, we think the latter half were all probably scientists and engineers.

What did we learn today?
Let's summarize the attributes you should look for when selecting training content.

Check the credentials of the design and development teams. Instructional design education and experience is very desirable, if not a must. Credible SME involvement is also very important.

There has to be a clear connection between learning objectives, evaluations, and content. Objectives should be focused on the needs of the student and they must be measurable. Evaluations must measure achievement of the objectives. Content needs to be focused on the objectives, clear, accurate and complete. Unnecessary content could increase the cost of training with little value added.

Content needs to be interactive, but the interactivity needs to be focused and relevant.

Engaging content is a plus, but not if it's overdone. Evaluate the engagement for your employees and decide if it fits your environment.

I think if you follow this advice, you'll have a lot more confidence in your training content purchase.

About the author:
Lauren Smith is the Manager of Product Development and Production, for Vivid Learning Systems, Inc., Richland, WA. He has been involved in the design, development, and production of Instructor-Led, Computer-Based, and Web-Based training programs since the days of green-screens (1981). In past lives, he was a Technical Instructor and Instructional Designer.