pandas and behavioural data

This chapter shows how to move from individual trial rows to summaries that answer a research question. Each step has a purpose: inspect the table, check its columns, filter invalid observations, create useful variables, and summarise by participant or condition. Run the companion notebook alongside the examples so you can compare each intermediate table with your own output.

Use this notebook

Use Notebook 1: Lexical decision data for this chapter: open it in the book or download the notebook.

The small example file is available as sample trial data.

from pathlib import Path
import pandas as pd

trials_path = Path("book/data/trials.csv")  # locate the CSV file
trials = pd.read_csv(trials_path)            # load rows into a DataFrame

Inspect before transforming

trials.head()          # preview the first rows
trials.shape           # number of rows and columns
trials.columns         # available column labels
trials.dtypes          # type stored in each column
trials.isna().sum()    # count missing values by column

Select and filter

reaction_times = trials["reaction_time"]              # select one Series
features = trials[["reaction_time", "correct"]]      # select two columns

correct_trials = trials.loc[trials["correct"]]        # keep correct trials
condition_a = trials.query("condition == 'A'")        # keep condition A

Check with Python: change the filter or grouped column and inspect the table.

import pandas as pd trials = pd.DataFrame({ "participant": ["P01", "P01", "P02", "P02"], "condition": ["A", "B", "A", "B"], "reaction_time": [0.51, 0.73, 0.62, 0.81], "correct": [True, True, False, True], }) correct = trials.loc[trials["correct"]] summary = correct.groupby("condition")["reaction_time"].mean() print(summary) Run Python

Output will appear here.

Group at the correct unit of analysis

Common mistake

After groupby(), one row may represent a participant or condition rather than a trial. State the unit of one row before merging the result or fitting a model.

participant_summary = (
    trials
    .groupby(["participant", "condition"], as_index=False)  # define the unit
    .agg(
        mean_rt=("reaction_time", "mean"),
        accuracy=("correct", "mean"),
        n_trials=("trial", "count"),
    )
)

Scientific question first

Averaging every trial together weights participants with more retained trials more heavily. Decide whether the inferential unit is a trial or participant before aggregating.

Missing values

trials["reaction_time"].isna()                         # Boolean missingness mask
trials.dropna(subset=["reaction_time"])                # remove missing rows
trials["reaction_time"].fillna(                         # replace with a summary value
    trials["reaction_time"].median()
)

Do not impute automatically. First ask why the value is missing.

Merge metadata

participants = pd.read_csv("book/data/participants.csv")  # participant metadata
analysis = participant_summary.merge(
    participants,
    on="participant",                  # matching key in both tables
    validate="many_to_one",             # each participant has one metadata row
)

validate turns an assumption about the relationship into a check.

Exercise: Comment the analysis

This exercise is about reading and explaining code, not writing a new analysis. Work with one or two other people. Download the complete Python script, open it in VS Code, and run it once without changing anything. It should print a summary table and display a figure.

Now it is your turn to add the comments:

from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd


project_dir = Path(__file__).resolve().parent.parent
trials_path = project_dir / "data" / "trials.csv"

trials = pd.read_csv(trials_path)

correct_trials = (
    trials
    .dropna(subset=["reaction_time"])
    .loc[lambda data: data["correct"]]
    .assign(reaction_time_ms=lambda data: data["reaction_time"] * 1000)
)

participant_summary = (
    correct_trials
    .groupby(["participant", "condition"], as_index=False)
    .agg(
        mean_rt_ms=("reaction_time_ms", "mean"),
        n_trials=("trial", "count"),
    )
    .sort_values(["participant", "condition"])
)

print(participant_summary)

condition_summary = (
    participant_summary
    .groupby("condition", as_index=False)
    .agg(
        mean_rt_ms=("mean_rt_ms", "mean"),
        variability=("mean_rt_ms", "std"),
    )
)

fig, ax = plt.subplots(figsize=(6, 4))
ax.bar(
    condition_summary["condition"],
    condition_summary["mean_rt_ms"],
    yerr=condition_summary["variability"],
    capsize=5,
)
ax.set(
    xlabel="Condition",
    ylabel="Mean reaction time (ms)",
    title="Correct-trial reaction time by condition",
)
fig.tight_layout()
plt.show()

Exercise 18 (Add comments for a future collaborator)

Imagine that a new student will use this script next semester. Add comments directly to the downloaded .py file that help them understand the analysis without merely translating Python into English. Do not change the working code.

Your comments should explain:

1. where the input file is located and what one input row represents;

2. which observations are removed;

3. why reaction time is multiplied by 1,000;

4. what .groupby(["participant", "condition"]) makes one output row represent;

5. what each new summary column contains; and

6. why sorting changes the presentation but not the calculated values;

7. why the data are summarised a second time for the plot; and

8. what the bars and error bars represent.

Add a comment above each main step and short end-of-line comments only where they make a particular operation clearer. Do not comment every line. Run the script again to confirm that adding comments has not changed its behaviour.

Before you run it

Predict the columns in participant_summary and whether it will have more or fewer rows than trials. Record the prediction before checking it with Python.

Solution to Exercise 18 (Add comments for a future collaborator)

There is no single correct wording. One useful version is:

# Build a path that works when the project is stored in a different location.
project_dir = Path(__file__).resolve().parent.parent
trials_path = project_dir / "data" / "trials.csv"

# Load trial-level data: each row represents one experimental trial.
trials = pd.read_csv(trials_path)

# Retain correct trials with a recorded reaction time, then convert seconds to ms.
correct_trials = (
    trials
    .dropna(subset=["reaction_time"])
    .loc[lambda data: data["correct"]]
    .assign(reaction_time_ms=lambda data: data["reaction_time"] * 1000)
)

# Produce one row for every observed participant–condition combination.
participant_summary = (
    correct_trials
    .groupby(["participant", "condition"], as_index=False)
    .agg(
        mean_rt_ms=("reaction_time_ms", "mean"),  # mean correct-trial RT
        n_trials=("trial", "count"),               # retained trial count
    )
    # Arrange rows consistently without changing the summary calculations.
    .sort_values(["participant", "condition"])
)

# Average participant-level values so each participant contributes equally to a bar.
condition_summary = (
    participant_summary
    .groupby("condition", as_index=False)
    .agg(
        mean_rt_ms=("mean_rt_ms", "mean"),
        variability=("mean_rt_ms", "std"),  # between-participant standard deviation
    )
)

# Plot the condition means; error bars show between-participant variability.
fig, ax = plt.subplots(figsize=(6, 4))
ax.bar(
    condition_summary["condition"],
    condition_summary["mean_rt_ms"],
    yerr=condition_summary["variability"],
    capsize=5,
)

Good comments communicate the unit of observation, the reason for a transformation, and the meaning of the output. A comment such as # use groupby repeats the code but does not provide that information.

Exercise 19 (Participant summaries)

For correct trials only, calculate each participant’s mean reaction time in each condition. Sort from fastest to slowest.

Hint

Filter rows first, group by both participant and condition, aggregate reaction_time, then sort the resulting column.

Solution to Exercise 19 (Participant summaries)

result = (
    trials
    .loc[trials["correct"]]
    .groupby(["participant", "condition"], as_index=False)
    .agg(mean_rt=("reaction_time", "mean"))
    .sort_values("mean_rt")
)