Aunt ADDIE – Assessment
Unit 3, Section 1, Overview
Before moving on to the rest of the content head over to the Unit 3, Section 1, assessment page to answer a few quiz questions.
ADDIE is the foundational model for systems thinking and instructional systems design. It is what is used by performance technologists and instructional designers to create any kind of system, like a website or type of instruction. There’s an easy acronym to help remember a relatively complex process. Let’s take a step back in time to explain why this process makes sense in developing websites.
The history of instructional systems design is said by some to have started at the onset of World War II. President Franklin Roosevelt, realizing that the US was most likely going to have to enter World War II, was faced with a major problem. How could we train millions of troops in a matter of months as the US’s standing army was too small to support conflicts in both Europe and Asia? He called together the nation’s top psychologists, including B.F. Skinner from Harvard, and they created a massive state-of-the-art training system that was performance-based and scalable to millions. A German general was later quoted as saying that they lost the war because they had not expected the US to be able to mobilize that many troops in that short of time.
Instructional systems design in the preliminary concepts of the ADDIE process was born. After World War II, instructional systems design began to be used in the US. Something funny was happening, though. As people were brought into organizations to develop training, they were finding that quite frequently the performance problems they were brought into solve were not related to training but rather to larger organizational issues that had nothing to do specifically with the employees. Instructional systems and its ability to impact the overall performance of organizations in general and not just as an item in instruction and training became known as performance technology. Two of the forefathers of performance technology and instructional systems were colleagues and friends, Drs. Joe Harless and Roger Kaufman. Dr. Harless actually worked under B.F. Skinner for a time and embraced and applied the ADDIE process in many organizations in the US and around the world.
One of their big debates, however, was whether assessment or evaluation came at the end of ADDIE, where the E stands for evaluation, or whether the assessment should occur at the front of ADDIE. Dr. Harless believed that the analyze or in analysis stage, or what he called “front end analysis,” should be the first step. But Dr. Kaufman believed that assessment or preliminary evaluation of the existing performance should be the first stage prior to any analysis. At the 30 years of practice, Dr. Harless finally agreed, however, with Dr. Kaufman that in fact the preliminary assessment should take place followed by front-end analysis.
ADDIE was changed to A ADDIE whereas [sic] Dr. Harless humorously referred to as Aunt-ADDIE. In other words, we need to assess — collect data from stakeholders — then follow through with ADDIE.
I wanted also to take a moment to remember Dr. Harless, who passed away unexpectedly in 2012, and thank him as a friend, a former mentor of mine, and as a gentle and amazing person who contributed so much to the quality of life for so many in the world. You’ll be missed, Joe. And it was a privilege to have the chance to know and work with you and may your ideas and contributions live on in the work of so many you impacted.
ADDIE was formerly created and codified in 1975 at my alma mater, Florida State University.
For the diagram, five steps of analyze, design, develop, implement, and evaluate represents a never ending loop involving continuous feedback and improvement. It actually involves 19 specific steps but we will not discuss this at–that detail at this time. The key to remember is that this is a basic model that represent an easy-to-follow process that will allow you to be systematic in identifying what you want to accomplish and then taking the necessary steps to increasing the likelihood that you’ll do what is necessary to make that happen. We’ll walk thorough each of the steps, allowing your hands-on practice at the end including the assessment for Aunt ADDIE throughout Unit 3.
Let’s start with assessment. Assessment is about understanding current performance conditions. Most importantly, performance needs to be operationalized by the users and the stakeholders of the system itself. So before we can analyze anything, we need to collect valid data from those who will be using the system. Who are the users? What do they need and want? What are the organizational goals? Probably most importantly, the process of assessment helps build in by-in from the users and consensus through user involvement and participation. In other words, the needs of users are reflected from the very beginning rather than [at] the end of the process. In other words, you have to first collect valid data before you can analyze it. This is the Aunt or assessment in A ADDIE.
A fundamental aspect of assessment is to assure you’re actually collecting valid data that is more likely to be accurate and is called data triangulation. Like a puzzle, in order to get a good grasp [of] what reality is, you must develop it from multiple perspectives that collectively fit together to form a more valid picture of reality than though . . . than through just our own perceptions or the perceptions of only a few select people.
What’s a whole? Well, it is sum of its parts. What is a valid perspective of what the current performance needs are of the major stakeholders of any system? Well, this must be defined by reaching out to those major groups so that collectively your system can deliver to [a] more accurate degree what they need and want from you.
So how does one collect this information? Well, in usability, refer to both empirical with users and non-empirical without user’s methods. With users include such data collection methods as interviews, which are one-on-one discussions with individuals; focus groups, which are small group discussions; surveys, which involve both paper-based and online questions that people could fill out anonymously. Natural observations [are] of all of observing people using your system where you do not necessarily interact with them or rather you just watch how they use it. Finally, usability testing, which we’ll do a lot of soon, is a systematic investigation into whether people can actually use your system in an effective, efficient, and satisfing fashion.
Non-empirical measures do not involve users and therefore can be done quicker and more cost effectively. The first is site analytics. In the growing information age real time data collection that informs you with essential data about your system, service, and organization is becoming more and more the norm. Site analytics gives you the information you need on top of probably more information than you ever ask for. Bottom line, though, with site analytics is you have to identify what success looks like and then identify the data you need to measure. Demographics go to basic analysis identifying who is actually using your site and the services you offer. Site analytics surround page views, bounce rates, or the downloads and purchases, etc., will help you get a feel for who and what is most used on your site. Performance metrics closely align with analytics and demographic data. These metrics can be at times outputs and outcomes of your site, like money earned or times something is clicked on, based on the goals of your site. Metrics are the measure of success while analytics is the data itself. Cognitive walkthroughs involves having a site expert walk through the site independently, trying to accomplish the major tasks of different user groups. And finally, heuristic testing compares a site to standard usability rules of thumb to ensure these guidelines are being followed.
Let’s have a quick discussion around valid sampling to increase the likelihood that you’re actually collecting valid data. Random sampling in the research world is one of the gold standards, which means that anyone you select to collect data from has an equal chance of being selected. If done correctly, this means that any results you obtained can be generalized to the greater population with more confidence that truly reflects the ideas, thoughts, and needs of other people you did not actually talk to. This is the process that the US Census Bureau uses so that it could generalize facts about the US.
Purposeful sampling is where you specifically select users or user groups that you want to talk to because they are representative users — influential, insightful, etc. Finally, self-selection sampling, which involves people volunteering to complete your survey or attend the forum, etc. This can, of course, have great bias because people who choose to participate may not necessarily be representative of the general public. But with that being said, however, this reflects a relatively easy way to get a large number of users to provide feedback, especially if an incentive is offered, e.g, free drawing for a price, etc.
So, in summary, Aunt ADDIE is all about assessing your user organizational needs and then following a systematic process. This is a process that was proven valid and very influential in our modern day society and world economy. Once your data is collected, then you can analyze and design systems that are hopefully relevant, useful, and usable to the people you hope will use it. Remember, the concept of “quality in, equality out.” If your data going into this process is not valid, then all the conclusions used by your entire system may prove to be invalid and miss the mark completely. It is critical that your assessment is as valid as you can make it because it informs your entire system.
Well, that’s it for Section 1. And remember to answer the five review questions, review any section readings, discuss this module on the discussion boards of others, and most importantly, do the hands-on activity. Thanks for joining me and I’ll hopefully see you later on in another module. We’re just getting started.