Introduction

In the previous module on t-tests, we discussed hypothetical studies where we compare two user interface conditions—touchscreen and a physical keyboard—on people’s text entry speed. But HCI studies often compare users’ performance across three or more conditions. Suppose you are evaluating three input conditions—touchscreen, normal physical keyboard, and split keyboard—for text-entry speed. You want to know whether task completion time differs across these conditions.

To assess these conditions, you might consider running pairwise comparisons between every combination (touchscreen vs. normal physical keyboard, normal physical keyboard vs. split keyboard, and split keyboard vs. touchscreen). That gives you three tests, and this is not a terrible plan. However, each test carries a 5% chance of a false positive (Type I error); you might find significant difference between conditions just by chance. And finding at least one pair of comparison yielding significant difference is no longer 5%, but it climbs to approximately:

$$ 1 - (1 - 0.05)^3 \approx 0.143 $$

That is a 14.3% chance of at least one false positive. As the number of conditions increases, the number of pairwise comparisons rises rapidly—and with it, the chance of a Type I error. A common approach to analyzing data from multiple conditions—while avoiding the increased Type I error rate from many pairwise comparisons in HCI—is to use Analysis of Variance (ANOVA).

Recall that, in null hypothesis testing, we ask whether we can reject the null hypothesis of no difference between conditions. ANOVA addresses this by testing all groups simultaneously in a single omnibus test; rather than asking, "Do these two groups differ?" it asks, "Is there any difference among these group means?" If the answer is yes, you can then follow up with targeted post-hoc comparisons that control for multiple testing.

One-way ANOVA

Let’s consider a hypothetical user study comparing task completion time across three gesture interaction techniques—swipe, pinch, and tap-and-hold—in a map navigation study.

First, load the required libraries. Install any you do not already have with install.packages().

install.packages(c("tidyverse", "effectsize", "car"))

library(tidyverse)
library(car)
library(effectsize)

We simulate a between-subjects dataset with 30 participants per technique (90 data points total). Each participant is assigned to one technique and completes a series of map navigation tasks. The dependent variable is the mean task completion time in seconds.

set.seed(42)

n_per_group <- 30

gesture_data <- tibble(
  participant = 1:(n_per_group * 3),
  technique = rep(c("swipe", "pinch", "tap_and_hold"), each = n_per_group),
  time = c(
    rnorm(n_per_group, mean = 4.2, sd = 1.0),   # swipe
    rnorm(n_per_group, mean = 5.1, sd = 1.1),   # pinch
    rnorm(n_per_group, mean = 5.8, sd = 1.2)    # tap-and-hold
  )
) %>%
  mutate(technique = factor(technique, levels = c("swipe", "pinch", "tap_and_hold")))

glimpse(gesture_data)

Let us examine group-level summaries:

gesture_data %>%
  group_by(technique) %>%
  summarise(
    n    = n(),
    mean = mean(time),
    sd   = sd(time),
    .groups = "drop"
  )

And visualize the distributions:

ggplot(gesture_data, aes(x = technique, y = time, fill = technique)) +
  geom_boxplot(alpha = 0.7, outlier.shape = 21) +
  geom_jitter(width = 0.15, alpha = 0.4, size = 1.5) +
  labs(
    title = "Task Completion Time by Gesture Technique",
    x = "Gesture Technique",
    y = "Time (seconds)"
  ) +
  theme_minimal() +
  theme(legend.position = "none")

ANOVA relies on three assumptions: