Preprocessing

topica takes pre-tokenized documents, a list[list[str]], or a Corpus. You control tokenization and vocabulary, because those choices are part of your method (see Build a defensible corpus).

Tokenize

from topica import tokenize

stop = open("stoplist.txt").read().split()        # a list, not a set
tokens = tokenize(text, stopwords=stop, min_length=3)

tokenize lowercases, applies a regex, drops stopwords and short tokens. It does not stem (stemming hurts interpretability); lemmatize in your own pipeline if you need it.

Stopword lists (58 languages)

topica.ENGLISH_STOPWORDS is a short, stable English default. For other languages — or a fuller English list — topica.stopwords(lang) serves the stopwords-iso lists (58 languages, MIT licensed, bundled in the wheel). Accepts an ISO 639-1 code or an English name:

import topica

fr = topica.stopwords("fr")            # or "french"; case-insensitive
corpus = topica.from_dataframe(df, text_col="texte", stopwords=fr)

topica.stopword_languages()            # ['af', 'ar', 'bg', ..., 'zh']

Unknown languages raise with the list of available codes. For the cross-lingual models (InfoCTM, ZeroShotTM), pass the matching list per language. Anything not covered: supply your own list.

Readable topic words: lemmatize, don't stem

Stemming truncates words to a root (military → militari, economy → economi), so top-word tables read as broken. If your text is not already stemmed, topica keeps the surface forms as-is. To merge inflections and keep readable words, lemmatize — and because from_dataframe (and tokenize) take a tokenizer callable, you can drop a lemmatizer straight in:

import topica
from nltk.stem import WordNetLemmatizer   # pip install nltk; nltk.download("wordnet")

_lemm = WordNetLemmatizer()
def lemmatize(text):
    return [_lemm.lemmatize(w)
            for w in topica.tokenize(text, stopwords=topica.ENGLISH_STOPWORDS, min_length=3)]

corpus = topica.from_dataframe(df, text_col="text", tokenizer=lemmatize)
# top words now read "military", "economy" — not "militari", "economi"

If your corpus arrives already stemmed (some bundled datasets and stm's poliblog do), there is no way to recover the original words — that is the data, not topica. Re-process from the raw text if you want readable labels.

Build a Corpus and prune the vocabulary

from topica import Corpus

corpus = Corpus.from_documents(
    docs,
    min_doc_freq=10,        # keep words in >= 10 documents
    max_doc_fraction=0.5,   # drop words in > 50% of documents
    min_cf=0,               # collection-frequency cutoff
    rm_top=20,              # drop the N most frequent residual words
)
print(corpus.num_docs, corpus.num_words, corpus.total_tokens)

The vocabulary is compiled in Rust, so even multi-gigabyte corpora build quickly. A Corpus can also load from disk (one document per line, or MALLET-style TSV).

Cap the vocabulary size (max_features)

To keep only the most frequent terms, pass max_features. It caps the vocabulary to the N most frequent surviving word types, applied after the other filters, and matches scikit-learn's CountVectorizer(max_features=):

corpus = Corpus.from_documents(docs, max_features=10_000)   # keep the 10k most frequent terms

Ties are broken deterministically (by frequency, then by first appearance). Note that scikit-learn ranks by collection (total) frequency, while gensim's keep_n ranks by document frequency; topica follows scikit-learn.

Apply a fixed vocabulary, or vectorize held-out documents

Two related tasks need the vocabulary held fixed rather than learned from the data.

Pin the vocabulary to a predetermined, ordered term list with vocabulary= (scikit-learn's vocabulary=). Out-of-vocabulary tokens are dropped, the column order follows your list, and the frequency filters are not applied (so vocabulary cannot be combined with min_doc_freq, max_features, and friends):

corpus = Corpus.from_documents(docs, vocabulary=["climate", "policy", "economy"])

To score held-out documents with a model you already fit, vectorize them against the training corpus with transform. The result shares the training vocabulary exactly (same terms, order, and ids, at full width), so the model's topic_word columns stay aligned:

corpus = Corpus.from_documents(train_docs, min_doc_freq=5)
model = topica.LDA(num_topics=20).fit(corpus)

heldout = corpus.transform(test_docs)          # same vocabulary as `corpus`
theta = model.transform(heldout)               # held-out document-topic mixtures

This is scikit-learn's vectorizer.transform and gensim's doc2bow on new text. A held-out document with no in-vocabulary tokens is dropped; its surviving index is recorded in heldout.kept_indices, so external labels can be realigned the same way as after pruning.

Detect phrases

Fixed expressions carry meaning together. Detect collocations and rewrite the tokens before modeling:

import topica
phrases = topica.learn_phrases(docs, min_count=8, threshold=12.0)
docs = topica.apply_phrases(docs, phrases)            # "health care" -> "health_care"

Split long documents

Long, heterogeneous documents violate the bag-of-words assumption. Segment them into comparable chunks, copying each source's metadata onto every chunk:

chunks, chunk_meta = topica.split_documents(
    texts, metadata, max_words=200, min_words=50,
)
# chunk_meta[j] = the source row + {"parent": i, "chunk": j}

Chunks from the same source are nested, so use clustered standard errors when you model effects.