Chapter 11 Experiment 11: Latent Semantic Analysis (LSA) and the SVD

Guide to the Experiment¶

The experiment has two parts:

Part 1 (a toy corpus): using a small $30 \times 15$ matrix, we take LSA apart step by step so that the geometric meaning of the SVD is completely visible; an interactive experiment lets you feel the effect of the truncation dimension $k$ directly.

Part 2 (a real corpus): using the 20 Newsgroups dataset from sklearn (2151 news posts from 4 of its topics, with 4000 terms), we see how the behavior of the SVD changes when the size of the matrix jumps from $30 \times 15$ to $4000 \times 2151$ .

import numpy as np
import plotly.graph_objects as go
from ipywidgets import widgets

np.set_printoptions(precision=4, suppress=True, linewidth=100)

def print_header(title):
    print("=" * 60)
    print(f"  {title}")
    print("=" * 60)

def print_step(step, desc):
    print(f"\n▶ Step {step}: {desc}")
    print("-" * 40)

def compare_print(label, actual, expected_desc):
    print(f"\n[{label}]")
    print(f"  Computed value: {actual}")
    print(f"  Expected value: {expected_desc}")

# --- Color scheme: book palette ---
C_BG   = "#F8F8F8"
C_GRID = "#D6D6D6"
C_V1   = "#57068C"   # linear algebra (violet)
C_V2   = "#006385"   # quantum mechanics (deep blue)
C_WARN = "#FF5D47"   # machine learning (orange)
C_AUX  = "#AB82C5"   # auxiliary color (light violet)
C_T1   = "#2AD2C9"   # accent color (teal)
C_AXIS = "#000000"

TOPIC_COLORS = {"Linear algebra": C_V1, "Quantum mechanics": C_V2, "Machine learning": C_WARN}

Part 1: A Toy Corpus (30 Terms × 15 Documents)¶

§1 Building the Term-Document Matrix¶

The corpus consists of 5 documents on each of three topics (15 documents in all), and the vocabulary contains 30 terms (10 per topic):

Topic	Vocabulary (10 terms)
Linear algebra	matrix, vector, eigenvalue, singular value, decomposition, determinant, linear equation, subspace, basis, orthogonal
Quantum mechanics	quantum state, superposition, entanglement, measurement, Hamiltonian, wave function, operator, eigenenergy, density matrix, unitary
Machine learning	neural network, gradient, loss function, training, feature, classification, regression, dim reduction, data, optimization

Term-document matrix $\mathbf{A} \in \mathbb{R}^{30 \times 15}$ : entry $(i,j)$ = the number of times term $i$ occurs in document $j$ . Each document contains only terms from its own topic, and terms from different topics never co-occur—this is the key assumption, and we will see later how it affects the result of the SVD.

# ============================================================
print_header("§1 | Building the 30×15 term-document matrix")
# ============================================================

np.random.seed(7)

TOPICS = {
    "Linear algebra": ["matrix","vector","eigenvalue","singular value","decomposition","determinant","linear equation","subspace","basis","orthogonal"],
    "Quantum mechanics": ["quantum state","superposition","entanglement","measurement","Hamiltonian","wave function","operator","eigenenergy","density matrix","unitary"],
    "Machine learning": ["neural network","gradient","loss function","training","feature","classification","regression","dim reduction","data","optimization"],
}

# --- Step 1: generate the corpus ---
# each document is guaranteed to contain its topic's anchor term, plus 5–7 more random terms from the same topic
print_step(1, "Generate the corpus (5 documents per topic, 15 in all)")
docs, labels = [], []
for tname, words in TOPICS.items():
    anchor = words[0]                          # anchor term (linear algebra = matrix): locks in co-occurrence within the topic + supports §2
    rest   = words[1:]
    for i in range(5):
        k = np.random.randint(5, 8)            # then draw 5–7 of the remaining 9 terms
        extra = list(np.random.choice(rest, k, replace=False))
        docs.append([anchor] + extra)
        labels.append(tname)
        
for i in [0, 5, 10]:
    print(f"  document {i:2d} [{labels[i]}]: {docs[i]}")

# --- Step 2: build the vocabulary and the term-document matrix ---
print_step(2, "Build the 30×15 term-document matrix A (integer term counts)")
VOCAB = sorted(set(w for t_words in TOPICS.values() for w in t_words))
word2idx = {w: i for i, w in enumerate(VOCAB)}
n_terms, n_docs = len(VOCAB), len(docs)

A_raw = np.zeros((n_terms, n_docs), dtype=float)
for j, doc in enumerate(docs):
    for w in doc:
        A_raw[word2idx[w], j] += 1

print(f"  Matrix shape: {A_raw.shape} ({n_terms} terms × {n_docs} documents)")
print(f"  Nonzero entries: {int((A_raw>0).sum())} / {n_terms*n_docs} = {(A_raw>0).mean():.1%} (the matrix is very sparse)")
print()
print("  The full matrix A_raw (rows = terms, columns = documents):")
# Print with vocab labels - show transposed for readability
header = "  term\\doc         " + "  ".join(f"d{j:02d}" for j in range(n_docs))
print(header)
print("  " + "-"*70)
for i, word in enumerate(VOCAB):
    row = A_raw[i].astype(int)
    row_str = "  ".join(f"  {v} " for v in row)
    print(f"  {word:<16} {row_str}")

§2 TF-IDF Weighting¶

Raw term counts have a problem: the term “matrix” occurs in every linear algebra document, and this very “generality” lowers its discriminating power. TF-IDF weighting damps frequent general-purpose terms and amplifies rare but discriminating ones:

a_{ij} \leftarrow \underbrace{\frac{c_{ij}}{\sum_{i'} c_{i'j}}}_{\text{TF (normalized term frequency)}} \times \underbrace{\left(\log\frac{n+1}{df_i+1} + 1\right)}_{\text{IDF (inverse document frequency)}}

(1)

Here $c_{ij}$ is the raw count, $df_i$ is the number of documents in which term $i$ occurs, and $n=15$ is the total number of documents.

# ============================================================
print_header("§2 | TF-IDF weighting")
# ============================================================

def tfidf_weight(A):
    tf  = A / (A.sum(axis=0, keepdims=True) + 1e-12)
    df  = (A > 0).sum(axis=1, keepdims=True)
    idf = np.log((A.shape[1] + 1) / (df + 1)) + 1
    return tf * idf

A = tfidf_weight(A_raw)

print_step(1, "Compare before and after weighting (\"matrix\" vs \"unitary\" as examples)")
for word in ["matrix", "unitary"]:
    idx = word2idx[word]
    df_val = int((A_raw[idx] > 0).sum())
    print(f"  \"{word}\": occurs in {df_val:2d} documents")
    print(f"    before weighting (term count): mean of nonzero values = {A_raw[idx][A_raw[idx]>0].mean():.4f}")
    print(f"    after weighting (TF-IDF): mean of nonzero values = {A[idx][A[idx]>0].mean():.4f}")

compare_print(
    "‖A‖_F² (after TF-IDF weighting)",
    f"{np.linalg.norm(A,'fro')**2:.6f}",
    "= Σᵢ σᵢ², to be checked with the SVD below"
)

§3 The SVD of the Term-Document Matrix¶

\mathbf{A} = \mathbf{U}\,\boldsymbol{\Sigma}\,\mathbf{V}^{\top} \qquad \mathbf{U}\in\mathbb{R}^{30\times 15},\; \boldsymbol{\Sigma}\in\mathbb{R}^{15\times 15},\; \mathbf{V}\in\mathbb{R}^{15\times 15}

(2)

Column $i$ of $\mathbf{U}$ , $\mathbf{u}_i$ : the $i$ -th left singular vector; row $t$ of $\mathbf{U}$ gives the coordinates of term $t$ in the SVD semantic space
Column $i$ of $\mathbf{V}$ , $\mathbf{v}_i$ : the $i$ -th right singular vector; row $j$ of $\mathbf{V}$ gives the coordinates of document $j$ in the SVD semantic space
(or, equivalently, $(\boldsymbol{\Sigma}\mathbf{V}^{\top})_{:,j}$ is the weighted low-dimensional coordinate vector of document $j$ )
$\sigma_i$ : the importance of the $i$ -th “semantic topic” (the larger, the more important)

After truncation to $k$ dimensions, only the first $k$ singular values are kept, and the best rank- $k$ approximation of the matrix $\mathbf{A}$ is

\mathbf{A}_k = \sum_{i=0}^{k-1}\sigma_i\,\mathbf{u}_i\mathbf{v}_i^{\top}

(3)

# ============================================================
print_header("§3 | The SVD and the singular value spectrum")
# ============================================================

U, s, Vt = np.linalg.svd(A, full_matrices=False)
# U  : (30, 15)  semantic coordinates of the terms
# s  : (15,)     singular values
# Vt : (15, 15)  semantic coordinates of the documents (transposed)

print_step(1, "Singular value spectrum σ₀ ≥ σ₁ ≥ ... ≥ σ₁₄")
print(f"  {s.round(4)}")

print_step(2, "Check the Frobenius norm identity  ‖A‖_F² = Σ σᵢ²")
compare_print(
    "‖A‖_F²",
    f"{np.linalg.norm(A,'fro')**2:.6f}",
    f"Σ σᵢ² = {(s**2).sum():.6f}"
)

print_step(3, "Cumulative energy: share of the total energy in the first k singular values")
energy_cum = np.cumsum(s**2) / (s**2).sum()
for k_val in [1, 2, 3, 5, 10]:
    bar = "█" * int(energy_cum[k_val-1] * 30)
    print(f"  k={k_val:2d}: {energy_cum[k_val-1]:.1%}  {bar}")

print_step(4, "For k=3: each of the three topics occupies one singular-vector direction")
coords3 = np.diag(s[:3]) @ Vt[:3, :]   # (3, 15) document coordinates
for tname in TOPICS:
    idxs = [i for i, l in enumerate(labels) if l == tname]
    c = coords3[:, idxs].mean(axis=1)
    print(f"  {tname} centroid: {c.round(3)}")
print("  → Each of the three topics \"occupies\" one coordinate-axis direction, because the vocabularies of different topics in the corpus do not overlap at all.")

§4 Interactive Experiment: The Truncation Dimension $k$ and Semantic Queries¶

What the two panels show:

Left panel (singular-value energy plot): the bar chart shows the energy $\sigma_i^2$ of each singular value; the first $k$ singular values, which are kept, are shown in dark violet, and the discarded part is shown in light violet. Adjusting $k$ shows you directly “how much information is kept.”

Right panel (semantic similarity): choose a term from the drop-down menu, and the plot shows the 5 terms with the highest cosine similarity to it in the current $k$ -dimensional LSA space:

\text{sim}(i, j) = \frac{\mathbf{U}_k[i,:]^{\top}\,\mathbf{U}_k[j,:]}{\|\mathbf{U}_k[i,:]\|\,\|\mathbf{U}_k[j,:]\|}

(4)

What to look for:

$k=1$ : there is only one semantic direction, all the terms are “squeezed together,” and the similarities are almost all 1
$k=3$ : the three topics separate exactly; same-topic terms have similarity ≈ 1, and cross-topic terms have similarity ≈ 0
$k>5$ : noise singular vectors come in, and cross-topic “contamination” begins to appear

Suggested queries: try “matrix” (linear algebra) and “entanglement” (quantum mechanics). How do their most similar terms differ between $k=3$ and $k=10$ ?

# Colab: enable the custom widget manager so that FigureWidget and the sliders are interactive; skipped elsewhere
try:
    from google.colab import output
    output.enable_custom_widget_manager()
except ImportError:
    pass

# ============================================================
print_header("§4 | Interactive experiment")
# ============================================================

def cos_sim(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))

def get_top5(word, k):
    Uk = U[:, :k]
    q  = Uk[word2idx[word]]
    if np.linalg.norm(q) < 1e-10:
        return []   # this term is the zero vector in the k-dimensional space

    ranked = sorted(
        [(cos_sim(q, Uk[i]), VOCAB[i]) for i in range(n_terms) if VOCAB[i] != word],
        reverse=True
    )[:5]
    return ranked

# --- Initial parameters ---
k_init    = 3
word_init = "matrix"

# --- Left panel: bar chart of singular-value energies ---
energy_vals = s**2 / (s**2).sum()
colors_init = [C_V1 if i < k_init else C_AUX for i in range(len(s))]

bar_energy = go.Bar(
    x=list(range(len(s))),
    y=energy_vals,
    marker_color=colors_init,
    name="σᵢ² / ‖A‖_F²",
    text=[f"{v:.3f}" for v in energy_vals],
    textposition="outside",
    textfont=dict(size=9),
)

# --- Right panel: bar chart of semantic similarities ---
top5_init  = get_top5(word_init, k_init)
sim_words  = [w for _, w in top5_init]
sim_vals   = [v for v, _ in top5_init]
topic_of   = {}
for tname, words in TOPICS.items():
    for w in words:
        topic_of[w] = tname
sim_colors_init = [TOPIC_COLORS.get(topic_of.get(w, ""), C_AUX) for w in sim_words]

bar_sim = go.Bar(
    x=sim_words,
    y=sim_vals,
    marker_color=sim_colors_init,
    name=f"similarity to \"{word_init}\"",
    text=[f"{v:.3f}" for v in sim_vals],
    textposition="outside",
    textfont=dict(size=11),
    xaxis="x2",
    yaxis="y2",
)

fig = go.FigureWidget(
    data=[bar_energy, bar_sim],
    layout=go.Layout(
        title=dict(
            text=f"LSA semantic space (k = {k_init}, energy retained {energy_vals[:k_init].sum():.1%})",
            font=dict(size=13),
        ),
        plot_bgcolor=C_BG,
        paper_bgcolor=C_BG,
        width=940,
        height=400,
        showlegend=False,
        # left axes: energy plot
        xaxis=dict(
            domain=[0.0, 0.44],
            title="Singular value index i",
            tickvals=list(range(len(s))),
            gridcolor=C_GRID, gridwidth=0.8,
        ),
        yaxis=dict(
            title="Relative energy σᵢ² / ‖A‖_F²",
            range=[0, energy_vals.max() * 1.25],
            gridcolor=C_GRID, gridwidth=0.8,
        ),
        # right axes: similarity plot
        xaxis2=dict(
            domain=[0.56, 1.0],
            anchor="y2",
            title="Term",
            gridcolor=C_GRID, gridwidth=0.8,
        ),
        yaxis2=dict(
            anchor="x2",
            title="Cosine similarity",
            range=[0, 1.25],
            gridcolor=C_GRID, gridwidth=0.8,
        ),
    ),
)

# --- Controls ---
k_slider = widgets.IntSlider(
    value=k_init, min=1, max=15, step=1,
    description="Truncation k:",
    style={"description_width": "90px"},
    continuous_update=False,
    layout=widgets.Layout(width="380px"),
)

word_dropdown = widgets.Dropdown(
    options=VOCAB,
    value=word_init,
    description="Query:",
    style={"description_width": "60px"},
    layout=widgets.Layout(width="200px"),
)

info_label = widgets.HTML(value=(
    f"<span style='font-size:13px; color:#57068C;'>"
    f"Energy retained: <b>{energy_vals[:k_init].sum():.1%}</b></span>"
))

def on_change(_):
    k    = k_slider.value
    word = word_dropdown.value

    # update the colors of the energy bars
    new_colors = [C_V1 if i < k else C_AUX for i in range(len(s))]

    # update the similarity bars
    top5      = get_top5(word, k)
    new_words = [w for _, w in top5]
    new_vals  = [v for v, _ in top5]
    new_sc    = [TOPIC_COLORS.get(topic_of.get(w, ""), C_AUX) for w in new_words]

    retained = energy_vals[:k].sum()
    info_label.value = (
        f"<span style='font-size:13px; color:#57068C;'>"
        f"Energy retained: <b>{retained:.1%}</b></span>"
    )

    with fig.batch_update():
        fig.data[0].marker.color = new_colors
        fig.data[1].x            = new_words
        fig.data[1].y            = new_vals
        fig.data[1].text         = [f"{v:.3f}" for v in new_vals]
        fig.data[1].marker.color = new_sc
        fig.data[1].name         = f"similarity to \"{word}\""
        fig.layout.title.text    = (
            f"LSA semantic space (k = {k}, energy retained {retained:.1%})"
        )

k_slider.observe(on_change,    names="value")
word_dropdown.observe(on_change, names="value")

ctrl = widgets.HBox(
    [k_slider, word_dropdown, info_label],
    layout=widgets.Layout(gap="20px", align_items="center"),
)
widgets.VBox([ctrl, fig])

Part 2: A Real Corpus—20 Newsgroups¶

The matrix of the toy corpus is $30 \times 15$ , small enough to see with the naked eye. What does a real-world text matrix look like?

We use the 20 Newsgroups dataset: a classic standard NLP dataset drawn from newsgroups of the 1990s, with 20 topics in all (we select 4). We first look at the size of the matrix, and then compare it with the toy corpus.

# ============================================================
print_header("Part 2 | The real 20 Newsgroups corpus")
# ============================================================

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from scipy.sparse.linalg import svds

print_step(1, "Download and load the data (4 topics, training set only)")
CATEGORIES = [
    "sci.space",          # space science
    "comp.graphics",      # computer graphics
    "rec.sport.baseball", # baseball
    "talk.religion.misc", # religion discussion
]

# CATEGORIES = [
#     "sci.space", "sci.electronics", "sci.med",                       # science group
#     "comp.graphics", "comp.windows.x",                               # computer group (many shared technical terms)
#     "rec.sport.baseball", "rec.sport.hockey",                        # sports group
#     "alt.atheism", "talk.religion.misc", "soc.religion.christian",   # religion/atheism group (sharply opposed, with very many shared words)
# ]

ng = fetch_20newsgroups(
    subset="train",
    categories=CATEGORIES,
    remove=("headers", "footers", "quotes"),
    random_state=42,
)
print(f"  Number of documents: {len(ng.data)}")
print(f"  Topics: {ng.target_names}")

print_step(2, "TF-IDF vectorization (max_features=4000, English stop words removed)")
vec = TfidfVectorizer(max_features=4000, stop_words="english", min_df=3)
X   = vec.fit_transform(ng.data)   # shape: (n_docs, n_terms)
terms = vec.get_feature_names_out()
print(f"  TF-IDF matrix shape: {X.shape} ({X.shape[0]} documents × {X.shape[1]} terms)")
print(f"  Compared with the toy matrix (30×15), the size has grown by about "
      f"{X.shape[0]//15} × {X.shape[1]//30} times")
print(f"  Sparsity (fraction nonzero): {X.nnz / (X.shape[0]*X.shape[1]):.3%}")

print_step(3, "Compute a truncated SVD with scipy.sparse.linalg.svds (k=50)")
print("  (A sparse matrix does not need a full SVD; svds directly gives the first k singular values)")
k_ng = 50
# svds is applied to X.T (terms × documents) and returns ascending order, which must be reversed
U_ng, s_ng, Vt_ng = svds(X.T, k=k_ng)
order = np.argsort(s_ng)[::-1]
U_ng, s_ng, Vt_ng = U_ng[:, order], s_ng[order], Vt_ng[order, :]
# U_ng : (4000, 50)  semantic coordinates of the terms
# Vt_ng: (50, n_docs) semantic coordinates of the documents

print(f"  First 10 singular values: {s_ng[:10].round(2)}")

print_step(4, "Identifying the common mode σ0: why is singular value 0 so large?")
ratio01 = s_ng[0] / s_ng[1]
u0_sign = max((U_ng[:, 0] > 0).mean(), (U_ng[:, 0] < 0).mean())
print(f"  σ0/σ1                  = {ratio01:.2f}      (σ0 is clearly larger than the rest)")
print(f"  share of u0 with one sign = {u0_sign:.1%}      (nearly all the same sign)")
print("  → The u0 belonging to σ0 is almost entirely positive: it is the \"average/common-mode\" direction shared by all documents,")
print("    encoding the overall term frequency and document length, unrelated to any topic—this is precisely the")
print("    general dimension that LSA customarily discards. The real topic structure lies beyond σ1.")

print_step(5, "Compare the decay after aligning the levels: toy (from σ0) vs real (from σ1)")
tail_cum_ng = np.cumsum(s_ng[1:]**2) / (s_ng[1:]**2).sum()   # real: renormalize after discarding the common mode
toy_cum     = np.cumsum(s**2) / (s**2).sum()
print(f"  Real (common mode removed, from σ1): cumulative energy of the first 3 dimensions: {tail_cum_ng[2]:.1%}")
print(f"  Toy (disjoint, no common mode, from σ0): cumulative energy of the first 3 dimensions: {toy_cum[2]:.1%}")
print("  → Toy topics are disjoint → concentrated in the first 3 dimensions, then a cliff;")
print("    real topics overlap → even after the common mode is removed, a gradual decline with no break.")

# ============================================================
print_header("Part 2 | Real corpus: comparing semantic queries")
# ============================================================

term2idx_ng = {w: i for i, w in enumerate(terms)}

def query_ng(word, k, top_n=6):
    """Query the semantic neighbors in the k-dimensional LSA space of 20ng."""
    if word not in term2idx_ng:
        print(f"  \"{word}\" is not in the vocabulary.")
        return []
    Uk = U_ng[:, :k]
    q  = Uk[term2idx_ng[word]]
    ranked = sorted(
        [(cos_sim(q, Uk[i]), terms[i]) for i in range(len(terms)) if terms[i] != word],
        reverse=True
    )[:top_n]
    return ranked

print_step(1, "Query the semantic neighbors of \"space\" for different k")
for k_val in [5, 20, 50]:
    results = query_ng("space", k_val, top_n=5)
    top_words = [w for _, w in results]
    print(f"  k={k_val:2d}: {top_words}")

print_step(2, "Query the semantic neighbors of \"baseball\" for different k")
for k_val in [5, 20, 50]:
    results = query_ng("baseball", k_val, top_n=5)
    top_words = [w for _, w in results]
    print(f"  k={k_val:2d}: {top_words}")

print_step(3, "Toy vs real: comparing cross-topic similarities for k=3")
print("  Toy corpus (topics do not overlap at all):")
Uk3_toy = U[:, :3]
pairs_toy = [("matrix","entanglement"), ("matrix","gradient"), ("quantum state","optimization")]
for w1, w2 in pairs_toy:
    sim = cos_sim(Uk3_toy[word2idx[w1]], Uk3_toy[word2idx[w2]])
    print(f"    sim({w1},{w2}) = {sim:.4f}")

print()
print("  Real corpus (k=20, topics share vocabulary):")
pairs_ng = [("space","orbit"), ("space","baseball"), ("god","jesus")]
Uk20 = U_ng[:, :20]
for w1, w2 in pairs_ng:
    if w1 in term2idx_ng and w2 in term2idx_ng:
        sim = cos_sim(Uk20[term2idx_ng[w1]], Uk20[term2idx_ng[w2]])
        print(f"    sim({w1},{w2}) = {sim:.4f}")
print()
print("  → In the real corpus the similarity of cross-topic terms is not 0, reflecting the overlap of topics in real language.")

# ============================================================
print_header("Part 2 | Comparing singular value spectra: toy vs real (common mode removed)")
# ============================================================

import plotly.graph_objects as go

fig2 = go.Figure()

# toy corpus: disjoint topics, no common mode; the topics start at σ0
s_toy_norm = s**2 / (s**2).sum()
fig2.add_trace(go.Scatter(
    x=list(range(len(s))),
    y=s_toy_norm,
    mode="lines+markers",
    name="Toy (disjoint topics, from σ₀)",
    line=dict(color=C_V1, width=2),
    marker=dict(size=8),
))

# real corpus: first discard the common mode σ0 (general dimension), then renormalize from σ1 on
s_ng_tail = s_ng[1:]**2 / (s_ng[1:]**2).sum()
fig2.add_trace(go.Scatter(
    x=list(range(1, len(s_ng_tail)+1)),
    y=s_ng_tail,
    mode="lines+markers",
    name="Real 20ng (common mode σ₀ removed, from σ₁)",
    line=dict(color=C_WARN, width=2),
    marker=dict(size=5),
))

fig2.update_layout(
    title=dict(text="Comparing singular-value energy distributions (common mode σ₀ removed from the real corpus)", font=dict(size=13)),
    xaxis=dict(title="Singular value index (aligned to the level of topic structure)", gridcolor=C_GRID, gridwidth=0.8),
    yaxis=dict(title="Relative energy", gridcolor=C_GRID, gridwidth=0.8),
    plot_bgcolor=C_BG,
    paper_bgcolor=C_BG,
    width=720,
    height=380,
    legend=dict(x=0.5, y=0.95),
)
fig2.add_annotation(
    x=0.5, y=1.0, xref="paper", yref="paper", showarrow=False,
    text="σ₀ (the common mode) of the real corpus has been removed, so that the \"average direction\" is not mistakenly compared with topic structure",
    font=dict(size=10, color=C_V2), yshift=18,
)
fig2.show()

print()
print("Observations (aligned to the level of \"topic structure\"):")
print("  Toy corpus: topics completely disjoint → no common mode; from σ0 on we have the three topics, with a cliff-like drop after the first 3 dimensions.")
print("  Real corpus: σ0 is the common mode (the average document direction) and has been discarded first; from σ1 on, the energy declines gradually with no break.")
print("  Key point: the earlier impression that \"the real corpus has a cliff too\" is in fact an artifact of the common mode σ0, not topic separation;")
print("       only after the common mode is removed does the difference between real (gradual) and toy (cliff) truly show.")

An Easily Misread Trap: The Common Mode σ0

When the SVD is applied to a nonnegative matrix (TF-IDF entries are all nonnegative), the largest singular value $\sigma_0$ often corresponds to a nearly all-positive “average/common-mode” direction: every document has a positive projection onto it, and what it encodes is the overall term frequency and document length, unrelated to any topic. This is the general dimension that is commonly discarded in LSA. Geometrically, it is the same as “PCA without centering: principal component 0 merely points to the average offset of the data cloud.”

Therefore the pattern “ $\sigma_0$ is especially large and then drops sharply,” common in real corpora, is not evidence of topic separation; the real topic structure only becomes clear from $\sigma_1$ onward, or after the average document has been subtracted. In the toy corpus the topics are completely disjoint and there are no globally shared terms, so the matrix is block-diagonal and has no such common mode—its $\sigma_0$ corresponds directly to a topic.

The core differences between toy and real:

	Toy corpus (30×15, disjoint topics)	20 Newsgroups (4 overlapping topics)
Global common mode $\sigma_0$	None (no terms shared across topics; the matrix is block-diagonal)	Present ( $\sigma_0$ is clearly the largest, $\sigma_0/\sigma_1 \approx 1.54$ ; $\mathbf{u}_0$ nearly all positive = the average document direction)
Where the topic structure lies	From $\sigma_0$ on (the leading singular vectors = the topics themselves)	From $\sigma_1$ on (the common mode must be discarded first)
Decay of the topic structure	Cliff-like (drops after the first 3 dimensions; $k=3$ suffices)	Gradual (no break after the common mode is removed; a larger $k$ is needed)
Cross-topic similarity	Exactly 0	Nonzero (reflecting the fuzziness of language)
The “magic” of the SVD	Mathematically inevitable (block-diagonal)	Statistically effective (but not exact)

Summary of the Experiment¶

Three Roles of the SVD, Seen through LSA

1. De-noiser
The truncated SVD $\mathbf{A}_k = \sum_{i=0}^{k-1}\sigma_i\mathbf{u}_i\mathbf{v}_i^{\top}$ keeps the main semantic structure and discards the random noise of individual documents. This is a direct application of the Eckart-Young theorem of §11.2.2: $\mathbf{A}_k$ is the best Frobenius approximation to $\mathbf{A}$ among matrices of rank $\leq k$ .

2. Topic extractor
Each singular value $\sigma_i$ represents the strength of a “latent semantic direction”; the corresponding $\mathbf{u}_i$ (term coordinates) and $\mathbf{v}_i$ (document coordinates) together describe the profile of this topic. This agrees completely with “singular vectors = principal directions” in the PCA of §11.3.

3. Semantic metric space
In the $k$ -dimensional LSA space, cosine similarity becomes a tool for measuring semantic distance. Two terms that never appear in the same document can still be close in the LSA space because “they co-occur with the same group of terms.” This is precisely what “latent” means: the SVD reveals the semantic structure hidden behind a sparse matrix.