Previous:

Next: Data Collection (TODO)

Introduction

With this note, we begin our discussion of quantitative methods for HCI research. When I use the terms “study” or “method,” assume the underlying philosophical stance is realism and objectivism (see Philosophical Foundations).

Every HCI study starts with a design decision: how will you structure the study to answer your research question? Experimental design is the plan that determines how participants are assigned to conditions, what data you collect, and how you control for confounding factors. A well-designed experiment helps you draw meaningful conclusions; a poorly designed one can waste weeks of work and still leave results ambiguous.

Study Methods

HCI researchers run studies in a variety of settings. Each comes with its own strengths and limitations.

• Lab studies bring participants into a controlled environment (a usability lab or a quiet office), where the researcher can manage the hardware, software, lighting, and noise level. Lab studies offer high internal validity—you can be confident that differences you observe are caused by the manipulation rather than environmental noise. The tradeoff is that the artificial setting may not reflect how people use technology in their daily lives.
• Field studies observe or test participants in their natural environments—at their desk, on public transit, or in a hospital ward. These studies have high ecological validity because the context is realistic, but they are harder to control. Interruptions happen, lighting varies, and you can’t standardize the experience as tightly as in a lab.
• Online (remote) studies recruit participants through platforms like Prolific and have them complete tasks on their own devices. These studies make it easy to reach large, diverse samples quickly and affordably. However, you have less control over participants’ hardware and environment, or even whether they are paying attention.

In this note, we will walk through a hypothetical example that compares two input interaction methods and introduces several concepts around empirical methods for HCI. While we will briefly explain these new ideas, the main goal is to introduce key terms and make you aware of them. You do not need to fully understand everything on the first read. Rather, the goal is for you to know these concepts exist and matter, so we can explore them more deeply in future notes.

Steps in Experimental Design

Whether you conduct a lab experiment or a field study, there are a few common steps you should follow when designing your study. These include:

• Formulate the research questions
• Choose the study method
• Operationalization
• Designing the study protocol

In the section below, we take a simple lab experiment to walk through things to consider in experimental design.