Notebook 1: Lexical decision data with pandas

Notebook 1: Lexical decision data with pandas#

The lexdec dataset contains 1,659 lexical-decision trials from 21 participants and 79 English nouns. Participants decided whether each stimulus was a word. The variables include response accuracy, log reaction time, trial number, native-language group, word frequency, word length, and semantic class.

Source: languageR, Baayen (2008).

Hide code cell source

 
from pathlib import Path
import sys

# Find the repository from the notebook folder, solutions folder, or project root.
for _candidate in (Path.cwd().resolve(), *Path.cwd().resolve().parents):
    if (_candidate / "notebooks" / "workshop_checks.py").is_file():
        PROJECT_ROOT = _candidate
        break
else:
    raise FileNotFoundError(
        "Open this notebook inside the cloned workshop repository; "
        "notebooks/workshop_checks.py is required."
    )

sys.path.insert(0, str(PROJECT_ROOT / "notebooks"))

from workshop_checks import Check, run_checks

check = Check()

1: Load the data#

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

data_path = PROJECT_ROOT / "book" / "data" / "real" / "lexical_decision.csv"
if not data_path.is_file():
    raise FileNotFoundError(f"Missing workshop data: {data_path}")

trials = ...  # load the CSV
trials.head()
Hint

Use pd.read_csv(data_path).

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_6", locals())

2: Inspect the DataFrame#

Inspect .shape, .columns, and .dtypes. Assign the number of trials, participants, and words to the three variables below.

print("shape:", trials.shape)
print("columns:", trials.columns.tolist())
print("types:\n", trials.dtypes)

n_trials = ...
n_participants = ...
n_words = ...
Hint

Use len(trials) for rows and .nunique() on Subject and Word.

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_11", locals())

3: Select columns#

Create analysis_columns containing these columns in this order: Subject, Word, RT, NativeLanguage, Correct, Frequency, Length, and Class.

analysis_columns = ...
Hint

Select multiple columns with trials[["first", "second", ...]].

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_16", locals())

4: Accuracy and Boolean filtering#

Count correct and incorrect responses. Then create correct_trials containing only correct responses.

n_correct = ...
n_incorrect = ...
correct_trials = ...
Hint

Compare trials["Correct"] with the strings "correct" and "incorrect". Use the resulting Boolean Series to filter rows.

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_21", locals())

5: Convert reaction time#

RT is the natural logarithm of reaction time in milliseconds. Add RT_ms to correct_trials by applying np.exp to RT.

correct_trials = correct_trials.copy()
correct_trials["RT_ms"] = ...
Hint

The inverse of the natural logarithm is np.exp.

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_29", locals())

6: Summarise language groups#

Calculate the median correct-trial reaction time for the two NativeLanguage groups. Return a Series indexed by NativeLanguage.

median_rt = ...
Hint

Group by NativeLanguage, select RT_ms, and call .median().

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_34", locals())

7: Word frequency and reaction time#

Create one row per word with its frequency and mean correct reaction time. Plot word frequency against mean reaction time.

Then answer:

  1. What pattern is visible?

  2. Why should the 1,659 trials not be treated as independent observations?

  3. Which variables might confound a comparison between native-language groups?

word_summary = ...

# Create a scatter plot of Frequency and mean_rt_ms.


# Interpretation:
Hint

Group by Word, Frequency, and Length with as_index=False. Use named aggregation to create mean_rt_ms. Plot one point per row of the summary.

Hide code cell source

 
run_checks("01_lexical_decision_pandas_cell_42", locals())