Introduction

In HCI research, we often compare conditions to answer questions such as “Does a new user interface reduce input errors?” and “Is this novel input method faster than the existing one?” To address these questions rigorously, we often use null hypothesis significance testing (NHST). In NHST, we begin by assuming there is no difference between conditions—that is, the new interface or interaction method offers no meaningful improvement over the existing one. This assumption is the null hypothesis ($H_0$). We then collect data and ask, “If $H_0$ were true, how unlikely would our observed data be?” If the data would be highly improbable under $H_0$, we reject $H_0$ in favor of the alternative hypothesis ($H_1$), which states that a real difference exists. The logic of NHST is analogous to proof by contradiction: rather than directly proving the new interface is better, we show that our observations are inconsistent with the assumption that it is not.

The p-value is the probability of observing data as extreme as (or more extreme than) what we collected, assuming $H_0$ is true. In an equation, this is: $p = P(\text{observed data} \mid H_0 \text{ is true})$. A small p-value (conventionally < 0.05) suggests that the observed data would be unlikely if $H_0$ were true, leading us to reject $H_0$. Common misconceptions:

A p-value is not the probability that $H_0$ is true. It does not tell you, for example, that "there is a 3% chance the keyboards are equally fast."
A p-value is not the probability of making a mistake. It is conditioned on $H_0$ being true, not on your data.
A smaller p-value does not imply a larger or more important effect. A trivially small difference can produce a tiny p-value with a large enough sample.
A p-value above 0.05 does not prove $H_0$. Absence of evidence is not evidence of absence.

NHST is best understood as a hybrid of two traditions: Fisher’s approach to significance testing and Neyman–Pearson’s approach to hypothesis testing (Perezgonzalez, 2015). From Fisher’s framework, it adopts the p-value, which measures how incompatible the data are with the null hypothesis ($H_0$). From Neyman–Pearson’s framework, it adopts a pre-specified significance level ($\alpha$), control of Type I and Type II errors, and the concept of statistical power. In classroom practice today, NHST is typically presented as a two-phase procedure:

Study design phase (a priori): Before collecting data, you specify your hypotheses and significance level ($\alpha$), choose a statistical test to use, then conduct power and sample-size planning.
Analysis phase (a posteriori): After collecting data, you compute a test statistic—often alongside a p-value or confidence interval—and interpret it to draw conclusions.

There are various tests you can use to draw conclusions in HCI research. In practice, the test you choose depends on the type of data you collect and your experimental design. One of the most commonly used tools in HCI research is the t-test, which compares the means of two groups. It is appropriate when (i) your dependent variable is continuous (e.g., task completion time in seconds), (ii) you are comparing exactly two conditions, and (iii) your data are approximately normally distributed. There are two main variants:

Design	Test	Example
Between-subjects: different participants in each condition	Independent-samples t-test	Group A uses touchscreen keyboard; Group B uses physical keyboard
Within-subjects: same participants in both conditions	Paired t-test	Each participant types on both keyboards

Example: Comparing Two Input Methods

Let’s use a running example to see how a t-test is used in HCI research. Suppose a researcher wants to know whether users complete a text-entry task faster on a physical keyboard than on a touchscreen keyboard.

install.packages(c("tidyverse", "effectsize", "car"))

library(tidyverse)
library(effectsize)
library(car)

Independent-Samples t-Test

We’ll consider a hypothetical between-subjects study. Twenty-four participants are randomly assigned to one of two groups: 12 use a touchscreen keyboard and 12 use a physical keyboard. The dependent variable is task completion time in seconds. We can generate synthetic data as follows:

set.seed(42)

keyboard_data <- tibble(
  participant = paste0("P", 1:24),
  condition   = rep(c("touchscreen", "physical"), each = 12),
  time_sec    = c(
    # Touchscreen group: generally slower
    round(rnorm(12, mean = 48.5, sd = 7.2), 1),
    # Physical keyboard group: generally faster
    round(rnorm(12, mean = 39.0, sd = 6.8), 1)
  )
)

print(keyboard_data, n = 24)

We now formulate the hypotheses. The null and alternative hypotheses would be: