Notebook 4: Text to features

Notebook 4: Text to features#

Six short stories stand in for documents. The notebook follows the path from text to a document–term matrix, TF–IDF vectors, and a three-dimensional token representation.

Hide code cell source

 
from pathlib import Path
import sys

# Find the repository from the notebook folder, solutions folder, or project root.
for _candidate in (Path.cwd().resolve(), *Path.cwd().resolve().parents):
    if (_candidate / "notebooks" / "workshop_checks.py").is_file():
        PROJECT_ROOT = _candidate
        break
else:
    raise FileNotFoundError(
        "Open this notebook inside the cloned workshop repository; "
        "notebooks/workshop_checks.py is required."
    )

sys.path.insert(0, str(PROJECT_ROOT / "notebooks"))

from workshop_checks import Check, run_checks

check = Check()

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

documents = [
    "attention selects relevant visual information",
    "visual attention changes reaction time",
    "memory retrieval depends on context",
    "working memory supports language comprehension",
    "language models learn contextual representations",
    "reaction time measures lexical processing",
]

1: Build a document–term matrix#

Fit a CountVectorizer to documents. Store the sparse matrix in counts and the feature names in terms.

vectorizer = ...
counts = ...  # transform the documents
terms = ...
Hint

Create CountVectorizer(), call .fit_transform(documents), then call .get_feature_names_out() on the fitted vectorizer.

Hide code cell source

 
run_checks("04_nlp_text_features_cell_7", locals())

2: Read the matrix#

Calculate the number of counted tokens in each document. Then find the column index for reaction and extract that column as a one-dimensional array.

document_lengths = ...
reaction_index = ...
reaction_counts = ...  # extract one term column
Hint

Sum counts across columns with axis=1 and use .A1 to obtain an array. Find a matching term with list(terms).index(...); select that matrix column and use .toarray().ravel().

Hide code cell source

 
run_checks("04_nlp_text_features_cell_12", locals())

3: TF–IDF and document similarity#

Fit TfidfVectorizer and calculate the full document-by-document cosine-similarity matrix.

tfidf = ...
similarities = ...
Hint

Use TfidfVectorizer().fit_transform(documents), then pass the resulting matrix twice to cosine_similarity.

Hide code cell source

 
run_checks("04_nlp_text_features_cell_17", locals())

4: Transfer the axis reasoning to token vectors#

The toy array below has shape documents × tokens × embedding features. Average over tokens to create one vector per document.

rng = np.random.default_rng(7)
token_vectors = rng.normal(size=(6, 8, 4))
document_vectors = ...
Hint

Tokens are axis 1. Averaging that axis should leave documents × embedding features.

Hide code cell source

 
run_checks("04_nlp_text_features_cell_22", locals())