How the models compare: one dataset, many methods

Every model in topica makes a different bet about how text is generated. The way to see those bets clearly is to build a corpus where we know the answer, then watch which methods recover it and which do not. This page walks through exactly that: a single synthetic corpus of municipal council statements, attacked by the whole roster.

The corpus and the scoring scripts live in benchmarks/synthetic/; python benchmarks/synthetic/run_all.py reproduces every number below.

Why synthetic, and how it is built

Real corpora never come with an answer key, so on real text we can only compare models to each other, never to the truth. Synthetic data removes that ceiling: we plant the structure, so we can measure recovery directly.

The design rests on one rule: separate the answer key from the prose. A deterministic, seeded sampler assigns every document its latent structure (its topic mixture, its covariates, its author, the topic of each sentence). That assignment is the ground truth, and no language model ever touches it. Language models do only the surface work, turning each sentence's assigned topic into fluent prose. The result reads like real council debate while remaining fully labeled underneath.

The statements are short, roughly 150 words over seven sentences. A sample, with each sentence's planted topic shown in brackets:

[housing] Rent increases are pushing tenants toward eviction faster than we can process permits, and a universal, subsidized housing mandate is overdue. [environment] Every wetland we pave over adds to the emissions problem, and this council needs a collective climate resilience strategy for the watershed. [transit] Adding rail service and bike-share stations along the corridor, with subsidized fares for every rider, would finally give this city universal transit coverage.

Three signals are woven through every sentence: topic vocabulary (permits, wetland, corridor), the speaker's party framing (tenants, climate, emissions), and the speaker's market-versus-public leaning (universal, subsidized, collective). Each is planted independently, so a model that recovers one should not be picking up another by accident.

Finding the topics

The starting point for every method is recovering the topics themselves. We fit a model, align its topics to the planted ones, and measure the cosine between the recovered and true word distributions.

LDA recovers the six planted topics at a topic-word cosine of 0.80, and the other admixture models (CTM, NMF, ProdLDA) land in the same range. This is table stakes: any usable topic model clears it on clean data. The interesting questions are everything the plain topic-word view cannot see.

Mixed membership versus single membership

The deepest split in the roster is whether a model treats a document as a blend of topics or as one topic. LDA, STM, and CTM are admixture models: each document is a distribution over topics. GSDMM is a mixture model: each document is generated by a single topic. The distinction is invisible in topic-word recovery and obvious in document-topic recovery.

We built a second corpus in which the number of topics per document is planted directly, from one up to five, and scored how well each model recovers the true document-topic mixture as documents blend more topics:

topics per document	LDA	CTM	GSDMM
1 (single-topic)	0.985	0.998	0.833
2	0.988	0.987	0.627
3	0.992	0.985	0.611
4	0.991	0.990	0.535
5	0.986	0.978	0.539

The admixture models are flat and near-perfect across the whole range: more blend to recover, and they recover it. GSDMM starts respectably on single-topic documents and then falls away, because one label per document cannot represent a mixture. This is not a defect. GSDMM is built for short texts where each document really is about one thing, and on that terrain its single-topic assumption is an advantage, not a handicap. Its recovery of single-sentence documents in the main corpus (adjusted Rand index 0.73) is strong. The lesson is to match the assumption to the text: reach for a mixture model when documents are short and focused, an admixture model when they genuinely span topics.

Covariates: prevalence and content

Council statements carry metadata (region, party, year), and covariate models ask how that metadata relates to the topics. There are two distinct questions, and different models answer them.

Prevalence asks how often each group discusses a topic. We planted a rising attention to housing over four years and region-specific emphases (the coast on transit, the north on environment). STM recovers the housing trend with the correct positive slope, and DMR maps every planted covariate to its topic with a perfect 5 of 5 hit rate. When the question is "who talks about what, and how has that changed," these are the tools.

Content asks how a group words a topic differently. We planted opposing party frames inside each topic (one party frames housing as development and supply, the other as affordability and tenants). SAGE and STM's content covariate recover this cleanly: the party contrast in wording separates the planted frame words with a large log-ratio (+4.4). A prevalence-only model would miss this entirely, because both parties discuss housing at similar rates while describing it in opposite vocabularies.

The practical point is that "effect of party" is ambiguous until you say whether you mean prevalence or content. topica lets you estimate both, and they can tell different stories about the same corpus.

Time

When topics themselves drift, the question is not prevalence but vocabulary. We planted a transit topic whose words move across the four years, from bus and parking early to rail and electric late. DTM tracks it: the probability of rail rises across the slices while bus falls. DTM holds the topic's identity fixed while letting its words evolve, which is the right model when a subject persists but its language changes. It is the wrong model when the mix of topics changes but each topic's vocabulary is stable; that is a prevalence question for STM with a time covariate.

Position within a document

Most models ignore word order entirely; a document is a bag of words. NarrativeTM does not. It asks where inside a text each topic tends to appear. We planted an ordered arc into a set of longer documents, twenty-four sentences that move through procedure, then housing, budget, transit, policing, and environment in sequence. NarrativeTM recovers the order exactly: the peak position of each topic's trajectory matches the planted sequence with a rank correlation of 1.00.

This axis carries the clearest length lesson in the suite. On the short main corpus, where the only positional signal is procedural framing at the opening and close, NarrativeTM recovers only a weak U-shape. A narrative-trajectory model needs documents long enough to have a narrative. Give it transcripts, articles, or opinions, and it reads their arc; give it a tweet, and there is no arc to read.

Author position

IdealPointTM and Wordfish ask a different question again: not what the topics are, but where each author sits on a latent axis inferred from word choice. We planted a market-versus-public leaning for each of twenty-four council members, deliberately orthogonal to their party, expressed only through a handful of tilt words. IdealPointTM recovers the planted positions at a correlation of 0.72, from a signal thinner than either topic or party. These models are the tool when the quantity of interest is a continuous trait of the speaker rather than the content of the text.

Across languages

InfoCTM aligns topics across two languages using a bilingual dictionary. We translated the corpus into Spanish and supplied a glossary linking the key terms. InfoCTM aligns all six topics across English and Spanish: topic k in one language denotes the same theme as topic k in the other. This is the model for a corpus that spans languages and a question that should not, such as tracking one policy debate across an English-language and a Spanish-language press.

Choosing the number of topics

Every parametric model asks for K, the number of topics, and the honest question is what happens when K is wrong. We fit LDA across a range and measured two things: coverage, how well the six true topics are each represented, and redundancy, the fraction of fitted topics that are nobody's best match.

fitted K	coverage	redundancy
2	0.63	0.00
4	0.82	0.00
6 (true)	0.99	0.00
8	0.98	0.25
12	0.97	0.50
16	0.97	0.62

The asymmetry is the whole story. Setting K too low is destructive: true topics get merged and coverage collapses. Setting K too high is survivable: every true topic is still found, and the cost is redundant topics that can be read and pruned. When unsure, err high. topica's search_k automates the scan.

The nonparametric alternative is to let the model discover K. On this corpus that alternative disappoints. HDP, which is supposed to infer the topic count, collapses to a single topic under its default concentration and, as that concentration is raised, jumps from three topics straight to nine, skipping six entirely. Its answer is governed by its prior, not the data. The reliable path to choosing K here is the parametric scan with coverage and redundancy in view, not a nonparametric count taken on faith.

Hierarchy

HLDA learns a tree of topics rather than a flat set. We built a corpus with a planted two-level structure, three "urban" topics and two "civic-order" topics, each domain sharing vocabulary that should surface at an internal node. HLDA does place each domain's shared vocabulary at an internal node, which is the core hierarchical claim. But its nonparametric tree over-splits into roughly thirty nodes rather than the planted handful, so it does not partition documents cleanly by domain. As with HDP, the nonparametric structure search is sensitive to its concentration parameters and does not settle on the planted shape without tuning. Hierarchy recovery is the one axis in the suite that only partially succeeds, and that honest edge is worth more than a benchmark on which everything passes.

The scorecard

Reading down the roster, the pattern is that parametric models matched to a clear question recover their structure cleanly and degrade gracefully, while the nonparametric structure-discoverers are fragile on this data.

Capability	Best-suited models	Recovery
Find topics	LDA, CTM, NMF, ProdLDA	0.80 cosine
Document-topic mixture	LDA, CTM, STM	flat ~0.99 across mixedness
Single-topic short text	GSDMM	0.73 ARI; degrades on blends
Prevalence covariates	STM, DMR	5/5 covariate hits
Content covariates	SAGE, STM content	+4.4 frame separation
Vocabulary drift over time	DTM	rail up, bus down
Narrative position	NarrativeTM	1.00 on long docs
Author ideal point	IdealPointTM, Wordfish	0.72 correlation
Cross-lingual alignment	InfoCTM	6/6 topics aligned
Choosing K	search_k (not HDP)	graceful; HDP unreliable
Hierarchy	HLDA	partial (over-splits)

A caveat on synthetic data

Synthetic data flatters models whose assumptions match the generator. Our documents are cleaner than any real corpus: topics are well separated, covariate effects are large, and noise is mild. A model that fails here, on data built to be recoverable, is in real trouble, which is what makes the HDP and HLDA results informative. But passing here is necessary, not sufficient. Real text is messier, and the honest use of this benchmark is as a floor: a method that cannot recover planted structure under ideal conditions will not find it in the wild.

Reading agreement on real data: the self-agreement ceiling

On real text there is no answer key, so the only check available is agreement: how closely two fits line up, scored here by aligned topic-word cosine (each topic Hungarian-matched to its nearest counterpart, then the mean cosine). We use it two ways — topica against a reference implementation (benchmarks/full_model_run.py scores every model this way), and one model against another. Both are bounded by the same thing, and it is easy to misread a number without it.

The bound is that a stochastic model does not even reproduce itself. Refit the same model on the same corpus with a different random seed and the two runs settle into different-but-comparable topics; their aligned cosine is below 1.0. That "self-agreement" is the ceiling: no cross-implementation comparison can beat how well a method agrees with itself, and how high the ceiling sits depends almost entirely on the inference family. Refitting each model twice (seeds 1 and 2) on the 5,000-document political-blog corpus at K=10:

Inference family	Models (self-agreement)	Ceiling
Matrix factorization	NMF 1.00, LSA 0.99	~1.00
Batch variational (EM / VAE)	CTM 1.00, ProdLDA 0.99	~1.00
Collapsed Gibbs (MCMC)	DMR 0.83, LDA 0.79	~0.80
Online / minibatch VB	gensim LdaModel 0.77, OnlineLDA 0.77	~0.77

The spread is large and it is not "variational versus sampled." Matrix methods and batch variational fits (CTM, ProdLDA) converge to essentially one optimum, so they reproduce themselves almost exactly. Collapsed-Gibbs samplers wander between comparable modes, landing around 0.80. Online (minibatch-stochastic) VB is the most seed-sensitive of all — its noisy per-minibatch gradient explores different local optima, so even gensim's reference LdaModel only agrees with itself around 0.77.

The consequence for reading the benchmark: score a model against its family's ceiling, not against 1.0. The same cosine means different things across families. Take OnlineLDA, which recovers the reference topics at ~0.67 against gensim — a number that looks alarming until you place it:

• online-VB self-agreement is only ~0.77 (gensim does not beat that against itself), so ~0.67 is roughly 90% of the achievable agreement, not 67% of a reachable 1.0;
• OnlineLDA agrees with topica's own collapsed-Gibbs LDA (~0.72) better than gensim does (~0.67);
• its topic coherence matches Gibbs LDA, and its top words are clean.

So ~0.67 is a healthy online-VB result. The same 0.67 from NMF, whose ceiling is 1.00, would instead be a red flag. This is why the harness frames accuracy against a per-model reference ceiling rather than a flat threshold.

These numbers are illustrative, not constants: the ceiling also moves with the corpus, with K, and with how far each fit is run toward convergence (a lightly-iterated Gibbs run self-agrees less than a long one). The ordering across families, though — deterministic and batch-variational at the top, online VB at the bottom — is the stable, transferable lesson.

Comparing two fits statistically: topica.compare

The ceiling above is exactly what makes a raw distance between two topics uninterpretable on its own: is a drop from cosine 1.00 to 0.80 real movement, or just the Monte-Carlo wander any refit shows? topica.compare(fit_a, fit_b) answers that question as a first-class operation rather than a manual diff.

cmp = topica.compare(fit_a, fit_b)

cmp.aligned            # Hungarian 1-to-1 matched topic pairs above `threshold`
cmp.unmatched_a        # topics that vanished (only in A) — never force-paired
cmp.unmatched_b        # topics that appeared (only in B)
cmp.splits, cmp.merges # one-to-many outcomes (overlay), named honestly (esp. across different K)
cmp.drift              # per-pair distance + whether it exceeds the reseed range
cmp.prevalence_shift   # change in topic prevalence, with a posterior-spread uncertainty
print(cmp.render())    # HTML card (same house as the manifest render()); to_markdown() too

Alignment reuses align_topics, which pairs two topics by the Hungarian 1-to-1 assignment and keeps a pair when its similarity clears threshold, so a topic with no honest counterpart is reported as appeared / vanished, not paired to its least-bad neighbor. Splits and merges (one topic in A → two in B, or vice versa) are an overlay, flagged only when an extra partner is close to a topic's own best match relative to the fit's cross-topic similarity floor — so a correlated-topic fit (STM/CTM) compared with itself reports full stability instead of near-total splits/merges, and genuine one-to-many outcomes across different K are still named (issue #642). (The reseed null below reads the Hungarian self-assignment align_topics also exposes, so every A-topic has a self-match to measure wander against.)

Drift needs a null. Give compare a reseed baseline and each matched pair is flagged when it moves beyond the range of self-agreement A shows across the reseeds — a heuristic band, not a calibrated significance test (the floor is the worst self-match over n_reseed refits, so a truly-null pair trips it at a rough ~1/(n_reseed+1) rate, and only A is reseeded):

# refit A on the same corpus under fresh seeds; compare builds the per-topic floor
cmp = topica.compare(fit_a, fit_b, refit=lambda s: topica.LDA(num_topics=20, seed=s).fit(corpus))

# or pass refits you already have, or a flat similarity floor (a family ceiling)
cmp = topica.compare(fit_a, fit_b, reseed_fits=[a2, a3, a4])
cmp = topica.compare(fit_a, fit_b, baseline=0.70)   # e.g. an online-VB ceiling

The floor for each A-topic is the worst self-agreement it keeps across the reseeds; an A↔B match below that floor moved more than reseeding A alone did, so its pair reads drifted=True. Without a null, distances are still reported but drifted is None (honestly "unknown") — never a false verdict.

Three uses, one tool: track which topics are stable / drift / are new across two corpora or time slices; measure how much apparent change is just seed wander; and use it as a library regression test — refit a canonical corpus across topica versions and flag when a "harmless" refactor silently moves the topics. It describes and tests difference; it does not build a consensus model (that is ensemble).

Comparing two manifests, without the models

When the models themselves are gone but their analysis manifests remain, compare can still align them — as long as each fit was recorded with its top words retained:

ra = topica.record_fit(fit_a, corpus_a, topic_words_n=25)   # opt-in: top words are content
rb = topica.record_fit(fit_b, corpus_b, topic_words_n=25)
cmp = topica.compare(ra, rb)                                 # same CompareResult, no refit

Manifests store only each topic's top-N words and mean prevalence, so the manifest path aligns by Jaccard overlap of the top-word sets (metric="jaccard", with a default threshold matched to that scale — lower than cosine's) and yields the same matched / vanished / appeared / split / merge structure. With enough words retained, this recovers the matching the live cosine path finds — but top-word set overlap is coarser than the full-distribution cosine: seed-varying fits (e.g. Gibbs LDA) churn their top-word lists between runs, so a genuinely-stable topic can share too few of its top-10 words to clear the threshold and be mislabelled vanished + appeared. Retain generously (topic_words_n≈25, not 10) so real matches sit well above the noise; compare also flags a "near-miss" on any vanished/appeared topic whose best cross-similarity sits just under the threshold, so a false disappearance is surfaced rather than silent. Two further caveats follow from having only top words: there is no prevalence-shift uncertainty (no posterior is stored), and a manifest cannot be refit — so drift is judged only against a baseline= floor, not a reseed null. Mixing a live model with a manifest is refused. Retaining top words is off by default (topic_words_n=0) because they are corpus-derived content; a manifest recorded without them raises a clear error rather than comparing on nothing.

Record both manifests with the same topic_words_n. Jaccard over sets of different sizes is systematically depressed (a 10-word set nested in a 25-word one scores at most 10/25 = 0.4), so if the two windows differ compare warns and falls back to the common top-N rather than reporting spurious drift.