In usability testing, you can collect qualitative data by actively asking questions to people or by observing them. When you actively ask questions, you have a control over what to ask and collect data that you want to collect (e.g., opinion toward a prototype, design preference). Data from observations are useful for you to interpret what participants are not aware of (e.g., tacit practices) and underlying feelings in what they say while thinking-aloud.
You can ask open-ended questions or closed-ended questions to gather qualitative data. Open-ended questions include “please tell me your favorite aspect of the design of the user interface” and “if you answered ‘yes’ to the previous question, can you describe X?”
The response to closed-ended questions is a good source of qualitative data too. For instance, you can obtain useful demographic information like gender (i.e., male, female, or other) and types of disability the respondent has. A response to closed-ended questions is not always qualitative. For example, a response to queries like “how old are you?” is quantitative, and a response “how often do you use your smartphone every week?” could be considered quantitative (as it is ordinal). And a response to a Likert scale question like “How much do you agree to the statement, ‘the data entry user interface is easy to use’: ‘strongly agree,’ ‘agree,’ ‘neutral,’ ‘disagree,’ ‘strongly disagree’” is more quantitative than qualitative though it is an ordinal data.
Qualitative data is not only helpful for you to understand the cause of design problems and explore potential solutions, but you can also quote the data to explain design problems to stakeholders.
We introduce two methods to analyze textual data from a transcript of interview or think-aloud recording: affinity diagram (Preece et al., 2015) and content analysis (Lewis, 2015; Preece et al., 2015).
The first step for the both methods is transcription of the recording. This is a time consuming process. If you are informally analyzing the data (e.g., for this course's assignment), consider transcribing only interesting parts (e.g., questionnaire responses, description of a critical incident, interesting quotes).
The process involves writing down interesting ideas from the transcripts on post-it notes. Then clustering similar ideas into piles. Once you assemble interesting ideas, identify relationship between different piles if there are any. Finally, you report the summary of the formed diagram: this diagram is called an affinity diagram. The process is similar to the class activity that you performed in brainstorming; the difference is that the ideas do not come from your imagination but instead from the qualitative data that you collected.
Content analysis procedure
The content analysis is more rigorous and systematic than creating an affinity diagram. The method involves open coding, axial coding, designing a codebook, coding the transcript, and evaluating inter-coder reliability (Lewis, 2015).