How the models compare: one dataset, many methods¶
Every model in topica makes a different bet about how text is generated. The way to see those bets clearly is to build a corpus where we know the answer, then watch which methods recover it and which do not. This page walks through exactly that: a single synthetic corpus of municipal council statements, attacked by the whole roster.
The corpus and the scoring scripts live in
benchmarks/synthetic/;
python benchmarks/synthetic/run_all.py reproduces every number below.
Why synthetic, and how it is built¶
Real corpora never come with an answer key, so on real text we can only compare models to each other, never to the truth. Synthetic data removes that ceiling: we plant the structure, so we can measure recovery directly.
The design rests on one rule: separate the answer key from the prose. A deterministic, seeded sampler assigns every document its latent structure (its topic mixture, its covariates, its author, the topic of each sentence). That assignment is the ground truth, and no language model ever touches it. Language models do only the surface work, turning each sentence's assigned topic into fluent prose. The result reads like real council debate while remaining fully labeled underneath.
The statements are short, roughly 150 words over seven sentences. A sample, with each sentence's planted topic shown in brackets:
[housing] Rent increases are pushing tenants toward eviction faster than we can process permits, and a universal, subsidized housing mandate is overdue. [environment] Every wetland we pave over adds to the emissions problem, and this council needs a collective climate resilience strategy for the watershed. [transit] Adding rail service and bike-share stations along the corridor, with subsidized fares for every rider, would finally give this city universal transit coverage.
Three signals are woven through every sentence: topic vocabulary (permits, wetland, corridor), the speaker's party framing (tenants, climate, emissions), and the speaker's market-versus-public leaning (universal, subsidized, collective). Each is planted independently, so a model that recovers one should not be picking up another by accident.
Finding the topics¶
The starting point for every method is recovering the topics themselves. We fit a model, align its topics to the planted ones, and measure the cosine between the recovered and true word distributions.
LDA recovers the six planted topics at a topic-word cosine of 0.80, and the
other admixture models (CTM, NMF, ProdLDA) land in the same range. This is
table stakes: any usable topic model clears it on clean data. The interesting
questions are everything the plain topic-word view cannot see.
Mixed membership versus single membership¶
The deepest split in the roster is whether a model treats a document as a blend
of topics or as one topic. LDA, STM, and CTM are admixture models: each
document is a distribution over topics. GSDMM is a mixture model: each document
is generated by a single topic. The distinction is invisible in topic-word
recovery and obvious in document-topic recovery.
We built a second corpus in which the number of topics per document is planted directly, from one up to five, and scored how well each model recovers the true document-topic mixture as documents blend more topics:
| topics per document | LDA | CTM | GSDMM |
|---|---|---|---|
| 1 (single-topic) | 0.985 | 0.998 | 0.833 |
| 2 | 0.988 | 0.987 | 0.627 |
| 3 | 0.992 | 0.985 | 0.611 |
| 4 | 0.991 | 0.990 | 0.535 |
| 5 | 0.986 | 0.978 | 0.539 |
The admixture models are flat and near-perfect across the whole range: more blend
to recover, and they recover it. GSDMM starts respectably on single-topic
documents and then falls away, because one label per document cannot represent a
mixture. This is not a defect. GSDMM is built for short texts where each
document really is about one thing, and on that terrain its single-topic
assumption is an advantage, not a handicap. Its recovery of single-sentence
documents in the main corpus (adjusted Rand index 0.73) is strong. The lesson
is to match the assumption to the text: reach for a mixture model when documents
are short and focused, an admixture model when they genuinely span topics.
Covariates: prevalence and content¶
Council statements carry metadata (region, party, year), and covariate models ask how that metadata relates to the topics. There are two distinct questions, and different models answer them.
Prevalence asks how often each group discusses a topic. We planted a rising
attention to housing over four years and region-specific emphases (the coast on
transit, the north on environment). STM recovers the housing trend with the
correct positive slope, and DMR maps every planted covariate to its topic with a
perfect 5 of 5 hit rate. When the question is "who talks about what, and how
has that changed," these are the tools.
Content asks how a group words a topic differently. We planted opposing
party frames inside each topic (one party frames housing as development and
supply, the other as affordability and tenants). SAGE and STM's content
covariate recover this cleanly: the party contrast in wording separates the
planted frame words with a large log-ratio (+4.4). A prevalence-only model
would miss this entirely, because both parties discuss housing at similar rates
while describing it in opposite vocabularies.
The practical point is that "effect of party" is ambiguous until you say whether you mean prevalence or content. topica lets you estimate both, and they can tell different stories about the same corpus.
Time¶
When topics themselves drift, the question is not prevalence but vocabulary. We
planted a transit topic whose words move across the four years, from bus and
parking early to rail and electric late. DTM tracks it: the probability of
rail rises across the slices while bus falls. DTM holds the topic's identity
fixed while letting its words evolve, which is the right model when a subject
persists but its language changes. It is the wrong model when the mix of topics
changes but each topic's vocabulary is stable; that is a prevalence question for
STM with a time covariate.
Position within a document¶
Most models ignore word order entirely; a document is a bag of words. NarrativeTM
does not. It asks where inside a text each topic tends to appear. We planted an
ordered arc into a set of longer documents, twenty-four sentences that move through
procedure, then housing, budget, transit, policing, and environment in sequence.
NarrativeTM recovers the order exactly: the peak position of each topic's
trajectory matches the planted sequence with a rank correlation of 1.00.
This axis carries the clearest length lesson in the suite. On the short main
corpus, where the only positional signal is procedural framing at the opening and
close, NarrativeTM recovers only a weak U-shape. A narrative-trajectory model
needs documents long enough to have a narrative. Give it transcripts, articles,
or opinions, and it reads their arc; give it a tweet, and there is no arc to read.
Author position¶
IdealPointTM and Wordfish ask a different question again: not what the topics
are, but where each author sits on a latent axis inferred from word choice. We
planted a market-versus-public leaning for each of twenty-four council members,
deliberately orthogonal to their party, expressed only through a handful of tilt
words. IdealPointTM recovers the planted positions at a correlation of 0.72,
from a signal thinner than either topic or party. These models are the tool when
the quantity of interest is a continuous trait of the speaker rather than the
content of the text.
Across languages¶
InfoCTM aligns topics across two languages using a bilingual dictionary. We
translated the corpus into Spanish and supplied a glossary linking the key terms.
InfoCTM aligns all six topics across English and Spanish: topic k in one
language denotes the same theme as topic k in the other. This is the model for a
corpus that spans languages and a question that should not, such as tracking one
policy debate across an English-language and a Spanish-language press.
Choosing the number of topics¶
Every parametric model asks for K, the number of topics, and the honest question
is what happens when K is wrong. We fit LDA across a range and measured two
things: coverage, how well the six true topics are each represented, and
redundancy, the fraction of fitted topics that are nobody's best match.
| fitted K | coverage | redundancy |
|---|---|---|
| 2 | 0.63 | 0.00 |
| 4 | 0.82 | 0.00 |
| 6 (true) | 0.99 | 0.00 |
| 8 | 0.98 | 0.25 |
| 12 | 0.97 | 0.50 |
| 16 | 0.97 | 0.62 |
The asymmetry is the whole story. Setting K too low is destructive: true topics
get merged and coverage collapses. Setting K too high is survivable: every true
topic is still found, and the cost is redundant topics that can be read and
pruned. When unsure, err high. topica's search_k
automates the scan.
The nonparametric alternative is to let the model discover K. On this corpus
that alternative disappoints. HDP, which is supposed to infer the topic count,
collapses to a single topic under its default concentration and, as that
concentration is raised, jumps from three topics straight to nine, skipping six
entirely. Its answer is governed by its prior, not the data. The reliable path to
choosing K here is the parametric scan with coverage and redundancy in view, not
a nonparametric count taken on faith.
Hierarchy¶
HLDA learns a tree of topics rather than a flat set. We built a corpus with a
planted two-level structure, three "urban" topics and two "civic-order" topics,
each domain sharing vocabulary that should surface at an internal node. HLDA
does place each domain's shared vocabulary at an internal node, which is the core
hierarchical claim. But its nonparametric tree over-splits into roughly thirty
nodes rather than the planted handful, so it does not partition documents cleanly
by domain. As with HDP, the nonparametric structure search is sensitive to its
concentration parameters and does not settle on the planted shape without tuning.
Hierarchy recovery is the one axis in the suite that only partially succeeds, and
that honest edge is worth more than a benchmark on which everything passes.
The scorecard¶
Reading down the roster, the pattern is that parametric models matched to a clear question recover their structure cleanly and degrade gracefully, while the nonparametric structure-discoverers are fragile on this data.
| Capability | Best-suited models | Recovery |
|---|---|---|
| Find topics | LDA, CTM, NMF, ProdLDA | 0.80 cosine |
| Document-topic mixture | LDA, CTM, STM | flat ~0.99 across mixedness |
| Single-topic short text | GSDMM | 0.73 ARI; degrades on blends |
| Prevalence covariates | STM, DMR | 5/5 covariate hits |
| Content covariates | SAGE, STM content | +4.4 frame separation |
| Vocabulary drift over time | DTM | rail up, bus down |
| Narrative position | NarrativeTM | 1.00 on long docs |
| Author ideal point | IdealPointTM, Wordfish | 0.72 correlation |
| Cross-lingual alignment | InfoCTM | 6/6 topics aligned |
| Choosing K | search_k (not HDP) | graceful; HDP unreliable |
| Hierarchy | HLDA | partial (over-splits) |
A caveat on synthetic data¶
Synthetic data flatters models whose assumptions match the generator. Our
documents are cleaner than any real corpus: topics are well separated, covariate
effects are large, and noise is mild. A model that fails here, on data built to
be recoverable, is in real trouble, which is what makes the HDP and HLDA
results informative. But passing here is necessary, not sufficient. Real text is
messier, and the honest use of this benchmark is as a floor: a method that cannot
recover planted structure under ideal conditions will not find it in the wild.