Section 2: Empirical Tests

Empirical Tests

Unit 4, Section 2, Overview

Next Up…
Before moving on to the rest of the content head over to the Unit 4, Section 2, assessment page to answer a few quiz questions.


Video Transcript
Hi. I’m Dr. Anthony Chow, and welcome to Unit 4, Section 2: Empirical Usability, Evaluation, and Testing.

In this module, we’re going to walk through working with actual representative users to help determine the overall usability of a website. I will go through the usability evaluation process that I’ve developed over the years. Let’s begin.

As mentioned in a previous module, the five empirical methods we are going to work through are:

  • interviews
  • focus group
  • surveys
  • natural observations
  • usability testing

Let’s discuss each within the context of a usability evaluation.

Guiding our usability evaluation process are three foundational questions we want answered:

  1. What are the top informational or service needs of our users?
  2. Is our current site meeting their needs in terms of relevancy and ease-of-use?
  3. Can they use our site effectively, efficiently, and in a satisfying fashion?

All of our empirical data collection methods are intended to help us answer these questions using data triangulation or different sources of information that collectively should give us a pretty accurate picture.

So let’s walk through what we have done with our revised department website. First, we conducted a department-wide survey of users. Next, I did a natural observation to see how a typical user uses our site. Then I conducted an online focus group to gather more insight from a larger number of users. We then tested the actual site with users to see if they are actually able to accomplish representative tasks with minimal pain and suffering. Finally, we ended the test with the debriefing interview and survey.

Our survey involved five general questions. We wanted to know their gender and also academic status. This will allow us to filter and sort the data by these variables. We then asked an open-ended question which allows them with no bias to identify what they feel their most important needs are. Then we followed that question up with a checklist of potential types of information and services, which will allow us to compare and contrast with the open-ended questions.

The final two questions about the live site were open-ended questions about strengths and weaknesses. We concluded the survey by showing them a redesigned mock up and asked them to rate their overall satisfaction.

You will note that the preliminary results of the open-ended question around priorities focused again on courses and course information. In addition, faculty contact information and student forms were also mentioned frequently. News and events were also mentioned. Here is the full list.

This slide shows that the mean ratings on a scale of 1 to 7 showed some level of satisfaction with the information provided, a 5.7 out of 7 while the effort level, a 4.3 out of 7, and the overall satisfaction of the site, a 4.8 out of 7, left some opportunities for improvement.

Some participant quotes include, “Information is hidden in odd places. It’s like a big scavenger hunt. Not intuitive.” Another quote: “And it’s everything you might need, but the layout makes no sense whatsoever. It’s also just sort of blah. Your eyes glaze over as you look at it.” Another person said, “The headings often don’t make sense, and the organization of the links often doesn’t make sense. I could eventually find what I need, but it’s usually after I’ve clicked other links that have taken me to the wrong place.”

Certainly we have input that there is, indeed, room for improvement.

The survey results regarding the mock up suggests a marked improvement. Note that all the ratings are in the 88 percent range on six fundamental areas of web design:

  • graphic design
  • information architecture
  • navigation
  • content
  • utility
  • ease of use

Quotes included, “The mock up is much cleaner, and it’s much easier to find information with the drop down menus. I like the mock up better than the current design.” Another participant said, “It seems that the information is a little more intuitively organized than the existing website. I like that everything’s accessible from the home page rather than having to find the separate tabs on different pages.” And someone else said succinctly, “Excellent improvement.”

Let’s take a look at a natural observation. As it sounds, this is a method in which you allow the user to interact with the website with no directions or specific tasks to complete. You just need to watch what they do, and oftentimes you ask them to think aloud so that you can hear what they’re thinking. Where do they go? Are they engaged? How long do they stay on certain pages? Is it what they are expecting?

I conducted an online natural observation in Blackboard Collaborate. Let’s take a look.

My natural observation was of one of my current graduate students. It was biased, of course, because she already knows the site and how to use it, but with that being said, I asked her to think aloud from the home page without any prompts, given the scenario you’re a prospective student and just hit the home page for the first time. Tell us what you think and where you would go.

She immediately mentioned the slideshow graphic. It takes up to one-third of the page above the fold and it represents the only real color and depth on the home page. Clearly, delivery of content in this area is important and has high priority for both the organization and its high potential to catch the user’s eye. She felt the overall graphic design was pleasing. Her eyes then moved down immediately to the Quick Links area. Again, it can be surmised because the links are in blue text and underlined, which drew her eye. She clicked on our Online Information Sessions. Back on the home page, she then looked at the main navigation area on the left side bar and ultimately selected Course Information, which was expected and consistent with all of our previous findings.

This is a screen capture of our session where our participant controlled the page view in Blackboard Collaborate. On the course information page, her eyes were drawn to the three-year plan, but she was surprised when she downloaded an Excel document as opposed to opening another page. Despite the fact that it said “download,” usually downloads are marked with an icon.

This quick natural observation immediately suggests the two elements about the website. First, our user was drawn to color both in terms of the images in the slideshow and also to the links. This also occurred on the subpage, too. Second, the site needs to practice the second highest usability heuristic, which is consistency, and links that lead directly to a download should be more clearly marked in the style that they normally are, file name or icon to the right of the link. She did not read the text that said “download” before clicking, which many of us tend to do.

I had a group of 11 students that I asked to provide feedback in a focus group. We wanted to collect general student feedback on both their current live site and our mock up. Our questions focused on our driving three questions: What are your top information needs? Is the site meeting your needs? And can you use the site of high levels of effectiveness, efficiency, and satisfaction?

Student comments focused, again, on courses and course information. Once student emphasized a need to see course syllabi, and another mentioned that our existing social media information was not explicit enough. That more opportunities and emphasis could be placed in promoting our social community.

Overall, they felt the site did meet their needs, but as one student put it, “That was because they had gotten used to it.” So they mentioned that the information was there somewhere, you just had to know where to find it. Another mention that she did not use the site very often.

Overall, EES for effectiveness, efficiency, and satisfaction was good for the site. It had good quality information. Efficiency, however, needed improvement as the focus group felt they knew the information they wanted was there, but they did not always know where to find it, and it was not that intuitive.

We broke them up into smaller groups, and asked them to look at our mockup of a couple of new designs online. The advantage of doing a focus group online is they are sitting at their own individual computers and can interact with websites directly. Some of the comments were, “This mock up feels like a meadow, space to roam.” Another commented, “It also allows you to kind of think and then search as opposed to being flooded with information, not really knowing where to go or what to click on.” Another said, “I do think that the mock up seems easier to navigate than the current site.” And another student mentioned, “I like the mock up site better. It is easier to look over and select the main links. The mock up has fewer things to click, which means fewer things to have to scan before you decide what to click.”

The focus group was further data to suggest that the revised version was on the right track. The best way, however, to know how usable something is to get representative users to try and accomplish major tasks. Opinions are one thing. Being able to actually accomplish tasks is another. Nielsen’s research suggests that 85 percent of all usability problems with the website can be found by testing with just five users.

We recruited several students and asked them to complete major tasks for each user group. Our protocol made sure to emphasize that we were testing the interface and not actually them. So they did not need to feel any pressure of a test of the knowledge or abilities. The two primary metrics of efficiency are error rate and time on task, and we measured both. Testing can occur either face to face or virtually where screens can be shared.

The test results of our actual HTML mock up looked promising as there were no errors and all tasks were completed within thirty seconds. The participant liked the use of white space but mentioned he would prefer the quick links in a more visible space above the fold. Participant 2 also was very successful in completing her tasks, although one error was made.

Again, all tasks were completed under thirty seconds. There were many participants who were also equally successful in the overall information architecture and interface design, passed tasks usability tests with flying colors.

The ISO standard for usability further defined usability into more precise subfactors. Over the years, I operationalized these into an instrument I call the EES Debriefing Survey. After every usability test, we interview and debrief the participants by asking them to rate each of the eight usability factors as you see here. Please also see the EES form in Excel that you are welcome to use for your own usability tests, and you’ll have an opportunity to do that in the hands-on activity.

In this example taken for an actual usability study we conducted, here are the results of 14 users who attempted to complete seven representative tasks. As far as the error rate, you will note that across all seven tasks an average of 1.6 errors did occur. In terms of time, you’ll notice a positive relationship between error rate and time. In other words, the more false starts, the longer a task took.

In particular, Task 5 and Task 7 clearly represented potential problem tasks, both in terms of error and time. Task 1 and 2 are also slightly troubling, given the amount of time and also slightly higher error rate. Not as efficient as it could be.

Here we have the overall ratings of effectiveness, efficiency, and satisfaction across all 14 participants. These are all satisfaction ratings, and clearly error rate and deviations from the critical path were problem areas. It has also caused an increase in mental effort. Collapsing the seven ratings into the three primary usability factors clearly shows that participants were generally satisfied with both the effectiveness of the site and in general. The issue, again, is the overall efficiency and ease of use. A grand mean of 7.5 on a scale of 10 would suggest a usability of the site as currently in the high C range, but certainly leaves much room for improvement.

Well, that’s it. In summary, then, empirical methods involve users directly, and there are a number to choose from. The more different ways you evaluate your site, the more likely that you will have a valid perspective on the overall usability of your site.

The EES form in particular allows you to test users trying to accomplish major tasks and gives you both quantitative and qualitative measures that will help you identify potential areas to focus on.

In our example, the preliminary mockup of a new site appears to leave students satisfied and able to accomplish major tasks easily with little mental effort. The other study, however, showed some difficulties in terms of the overall efficiency in which participants were able to complete their tasks.

That’s it for this module. Please be sure to engage in the review session, discussion topics, reading, and hands-on activities. Try the EES ratings on your own site with your own scenarios. Thank you for joining me.