Previous: Experimental Design

Introduction

In the previous note, we introduced the basic concept of experimental design. One key aspect was collecting data to measure the dependent variable and answer our research question. In practice, there are many ways to collect data. Here, we describe a basic technique for collecting data, discuss which events are worth logging in a study, explain how to clean and preprocess the data, and show basic visualizations for exploratory data analysis.

Interaction Logging

Interaction logging is the practice of automatically recording user actions as they interact with a system. You can timestamp and log users’ interactions—such as clicks, keystrokes, scrolling, page transitions, and pauses—for later analysis. Importantly, unlike surveys or interviews, which rely on participants’ self-reports, interaction logs capture what users actually did rather than what they say. Common types of interaction data include:

Data Type	Example	What It Reveals
Clicks	Button presses, link selections	Which features users engage with; navigation choices
Dwell time	Time spent on a page or element	Attention, interest, confusion, or difficulty
Keystrokes	Text input events	Typing speed, error correction patterns, input effort
Errors	Failed actions, invalid inputs	Usability problems, confusing interface elements
Navigation paths	Sequences of pages visited	How users explore the interface; whether they find what they need
Scroll events	Scroll depth and direction	Whether content below the fold is seen
Hover events	Mouse movements over elements	Areas of visual attention (on desktop)

Interaction logging can reveal things that surveys or interviews are ill-suited to capture:

Frequency and duration at scale. A survey might ask "Did you find the search feature useful?" but logs can show that 73% of users never clicked the search button at all.
Temporal patterns. Logs reveal when, during a session, users struggle—for example, a spike in back-button clicks after reaching a particular page.
Unconscious behavior. Users often cannot accurately report how long they spent on a task or how many errors they made. Logs capture this without recall bias.
Longitudinal tracking. Logging can passively capture behavior across days or weeks, which would be impractical with direct observation.

Interaction data generally falls into two categories:

Event-level data: individual records of discrete actions (e.g., "User 12 clicked the Submit button at 14:03:22"). This is the raw material for most analyses.
Aggregated/session-level data: summaries derived from event data (e.g., "User 12 completed 3 of 5 tasks in an average of 42 seconds each"). This is what you typically use for statistical tests.

Hands-On: Logging Clicks and Dwell Times

In a real study, you would implement a mechanism to log interaction data (e.g., by inserting a JavaScript snippet into your web app). For this tutorial, we create a synthetic dataset to learn how to work with log data in a self-contained way. We assume that we are conducting a usability study in which 20 participants each complete a series of tasks on a prototype website.

library(tidyverse)

set.seed(42)

n_users <- 20
pages <- c("/home", "/search", "/product", "/cart", "/checkout")

# Generate event-level log data:
# 60 sessions (3 per user), each containing 10 events
log_data <- tibble(
  session_id = paste0("S", sprintf("%04d", rep(1:60, each = 10))),
  user_id    = paste0("U", sprintf("%03d", rep(rep(1:n_users, times = 3), each = 10))),
  timestamp  = as.POSIXct("2025-03-01 09:00:00") +
               sort(runif(600, min = 0, max = 7200)),
  page       = sample(pages, 600, replace = TRUE,
                       prob = c(0.30, 0.25, 0.25, 0.12, 0.08)),
  event_type = sample(c("click", "click", "click", "error", "navigation"),
                       600, replace = TRUE),
  dwell_ms   = round(rlnorm(600, meanlog = 8, sdlog = 0.8))
)

# Preview the data
log_data

The code above generates 600 rows of data. Each row represents a single interaction event. Every session belongs to exactly one user, and each user contributed three sessions of ten events each. The dwell_ms column indicates how long the user stayed on the page before the event (in milliseconds). The event_type column distinguishes normal clicks from errors and navigation transitions.