Human–Computer Interaction (HCI) studies how people use, experience, and are affected by interactive technologies. For example, given two interface designs that help users perform the same task, we might ask, “Which interface is better?” To answer questions like this, we rely on empirical methods—systematic approaches to collecting and analyzing data about user behavior. These methods provide evidence to assess whether one design is truly better than another, whether users can complete a task, and whether they enjoy doing so. Intuition and design expertise can also inform these decisions, but they depend on experience that is difficult to acquire and on skills that not everyone has. Empirical methods, by contrast, are learnable skills.
At the heart of empirical methods is how you design data collection and analysis. Consider a team redesigning a mobile checkout flow: the designers believe the new three-step flow is simpler than the old five-step flow. That belief is a hypothesis, not a fact. When studying interface and interaction design, we objectively measure whether users complete checkout faster or more successfully with the new design, observe where users hesitate, make errors, or abandon the process, and ask users about their subjective experience and perceived effort. We then analyze the resulting data to draw conclusions that go beyond anecdote. Without empirical evidence, design decisions rest on intuition. With it, we can make informed design decisions that better serve users.
When conducting HCI research studies, we collect data through a variety of settings, depending the goal of the research. To name a few, for example:
The rest of the course covers a variety of data collection techniques, data-gathering approaches, and data analysis methods. Before we dive into individual techniques, the next few notes introduce some fundamental topics in empirical methods for HCI, including formative and summative studies, the philosophical underpinnings of different research approaches, and ethics.
HCI studies generally serve one of two purposes—formative and summative—and recognizing the distinction helps you choose the right approach.
Formative studies aim to inform and improve a design while it is still being developed. Their goal is to uncover problems, understand user needs, and generate ideas—not to produce definitive measurements. Formative studies tend to be qualitative or mixed-method, with small sample sizes and rich, detailed data. They answer questions like: What problems exist? Why do users struggle? What do users need? Examples of formative studies:
| Method | Example |
|---|---|
| Contextual inquiry | Observing nurses using an electronic health record system to identify workflow pain points |
| Think-aloud usability test | Asking users to narrate their thoughts while using an early prototype of a scheduling app |
| Diary study | Having participants log their frustrations with smart home devices over two weeks |
| Interview | Talking to remote workers about what makes video calls exhausting |
Summative studies aim to evaluate a finished design against specific criteria. The goal is to measure performance, compare alternatives, or validate that a design meets its targets. These studies tend to be quantitative, using larger sample sizes and statistical analysis. They answer questions like: Which design is better? Does this design meet our usability target? How large is the improvement? Examples of summative studies:
| Method | Example |
|---|---|
| Lab experiment | Comparing task completion time between two text-entry techniques with 40 participants |
| Online A/B testing | Measuring conversion rates for two checkout page layouts across 10,000 users |
| Standardized questionnaire | Administering the System Usability Scale (SUS) after participants complete a set of tasks |
The distinction is not rigid—a research project can serve both purposes—but a useful rule of thumb is this: formative studies help you build the right thing, while summative studies help you prove it works. In practice, a project often begins with formative work (understanding the problem space) and concludes with summative evaluation (measuring the solution). As a result, research papers often reflect this progression: early sections may be formative, while later sections are more summative.
Regardless of whether a study is formative or summative, qualitative or quantitative, HCI research follows a common process: (i) forming research question(s), (ii) designing a study, (iii) collecting data, (iv) performing analysis, and (v) reporting. Each stage builds on the previous one, but the process can also be cyclical. For instance, someone might start with a rough research idea, design a pilot study, and collect data; but if they identify issues (e.g., the research question is not answerable, or the study design does not allow useful data to be collected), they may need to revisit an earlier step.