Notebook 3: From behavioural summaries to a classifier

Notebook 3: From behavioural summaries to a classifier#

This notebook uses the lexical-decision data again, but at a new unit of analysis: one row per participant. The aim is to practise the scikit-learn interface and learn where leakage can enter an analysis.

Hide code cell source

 
from pathlib import Path
import sys

# Find the repository from the notebook folder, solutions folder, or project root.
for _candidate in (Path.cwd().resolve(), *Path.cwd().resolve().parents):
    if (_candidate / "notebooks" / "workshop_checks.py").is_file():
        PROJECT_ROOT = _candidate
        break
else:
    raise FileNotFoundError(
        "Open this notebook inside the cloned workshop repository; "
        "notebooks/workshop_checks.py is required."
    )

sys.path.insert(0, str(PROJECT_ROOT / "notebooks"))

from workshop_checks import Check, run_checks

check = Check()

1: Load and prepare trial-level variables#

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

data_path = PROJECT_ROOT / "book" / "data" / "real" / "lexical_decision.csv"
if not data_path.is_file():
    raise FileNotFoundError(f"Missing workshop data: {data_path}")

trials = ...
trials["is_correct"] = ...
trials["RT_ms"] = ...
Hint

Load with pd.read_csv. Compare Correct with "correct", and undo the natural logarithm in RT with np.exp.

Hide code cell source

 
run_checks("03_model_workflow_cell_6", locals())

2: Make one row per participant#

Create participants with Subject and NativeLanguage, plus mean reaction time, accuracy, mean word frequency, and mean word length. Use named aggregation.

participants = ...
participants.head()
Hint

Group by Subject and NativeLanguage with as_index=False. Aggregate RT_ms, is_correct, Frequency, and Length with .mean().

Hide code cell source

 
run_checks("03_model_workflow_cell_11", locals())

3: Separate features and target#

Let X contain the four numerical summaries and let y contain NativeLanguage.

feature_names = ["mean_rt_ms", "accuracy", "mean_frequency", "mean_length"]
X = ...
y = ...
Hint

Select the feature list with square brackets. Select the target with one column name so that it remains a Series.

Hide code cell source

 
run_checks("03_model_workflow_cell_16", locals())

4: Split before learning preprocessing parameters#

Use 30% of participants as a test set. Set random_state=42 and stratify by y.

X_train, X_test, y_train, y_test = ...
Hint

Call train_test_split(X, y, test_size=0.3, random_state=42, stratify=y).

Hide code cell source

 
run_checks("03_model_workflow_cell_21", locals())

5: Fit a pipeline#

Create a pipeline containing StandardScaler() followed by LogisticRegression(max_iter=1000). Fit it on the training rows and predict the test rows.

model = ...
model.fit(...)
predictions = ...  # predict the test rows
Hint

Use make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)); then call .fit(X_train, y_train) and .predict(X_test).

Hide code cell source

 
run_checks("03_model_workflow_cell_26", locals())

6: Inspect the errors as well as the score#

Create a 2 × 2 confusion matrix with labels ordered as English, Other. Then count how many test predictions are correct.

matrix = ...
n_correct_predictions = ...  # count matches
matrix
Hint

Use confusion_matrix(y_test, predictions, labels=["English", "Other"]). Compare the two arrays and sum the Boolean results.

Hide code cell source

 
run_checks("03_model_workflow_cell_31", locals())