Skip to content

Datasets

Bundled example datasets for quickstarts and worked examples. Small datasets ship inside the wheel and load offline; larger ones are downloaded once from GitHub on first use and cached locally (under ~/.cache/topica/datasets, or TOPICA_DATA_HOME). The text loaders return a pandas DataFrame ready for from_dataframe; pass return_path=True for the cached CSV path without pandas.

import topica

df = topica.datasets.load_gadarian()
corpus = topica.from_dataframe(
    df, text_col="open.ended.response", stopwords=topica.data.ENGLISH_STOPWORDS
)

The embedding-native models want vectors, not a token corpus. load_ng20_minilm covers that case: a 20-Newsgroups subset with MiniLM sentence embeddings precomputed for both documents and vocabulary, so ProdLDA/FASTopic/BERTopic/Top2Vec run offline with no sentence-transformers/torch install. It returns a Bunch (attribute access), not a DataFrame.

b = topica.datasets.load_ng20_minilm()
docs = [t.split() for t in b.texts]
# The default density clusterer (UMAP -> HDBSCAN) finds only ~3 topics here, with
# ~80% of the documents in one topic. That is NOT a topica defect: the reference
# umap-learn + HDBSCAN pipeline finds the same few-topic structure on this corpus at
# this min_cluster_size (see parity/bertopic_umap_default_compare.py). It is genuine,
# coarse density structure. For a finer or fixed number of topics, lower
# min_cluster_size, or use a fixed-K clusterer with reduce_frequent, e.g.:
bt = topica.BERTopic(
    clusterer="kmeans", num_clusters=5, reduce_frequent=True, seed=1,
).fit(docs, b.doc_embeddings)

topica.datasets.load_gadarian 

load_gadarian(*, return_path: bool = False, as_bunch: bool = False)

Load the Gadarian & Albertson immigration experiment (341 documents).

The canonical stm prevalence example. Open-ended survey responses with an anxiety-prime treatment and pid_rep (Republican identification). Raw, untokenized text lives in the open.ended.response column, so build a corpus with stopword removal::

df = topica.datasets.load_gadarian()
corpus = topica.from_dataframe(
    df, text_col="open.ended.response", stopwords=topica.ENGLISH_STOPWORDS
)

This dataset is bundled in the wheel and loads offline. Pass return_path=True for the CSV path instead of a DataFrame, or as_bunch=True for a :class:Bunch whose .df is this table (the uniform shape shared with :func:load_ng20_minilm).

topica.datasets.load_poliblog 

load_poliblog(*, return_path: bool = False, as_bunch: bool = False)

Load the CMU 2008 political blog corpus (a 2,000-document sample).

The text in the text column is already tokenized and stemmed (space-separated), as in the stm poliblog vignette, so no stopword removal is needed::

df = topica.datasets.load_poliblog()
corpus = topica.from_dataframe(df, text_col="text")

Covariates: rating (Liberal/Conservative), day, blog. Downloaded once and cached. Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table.

topica.datasets.load_dubois 

load_dubois(*, return_path: bool = False, as_bunch: bool = False)

Load Du Bois-era articles from The Crisis, 1910-1934 (704 documents).

Raw text in the text column; covariates year, decade, volume, issue, author, subjects. Build a corpus with stopword removal::

df = topica.datasets.load_dubois()
corpus = topica.from_dataframe(
    df, text_col="text", stopwords=topica.ENGLISH_STOPWORDS
)

The corpus holds a few (3) exact-duplicate articles reprinted across issues; drop them with df.drop_duplicates("text") if a fit should not double-count them. Downloaded once and cached. Pass return_path=True for the CSV path.

Note on author for :class:~topica.AuthorTopic: this field is dominated by Du Bois (about 675 of 704 articles) and contains delimited composites ("Du Bois; Gruening") and name/initial variants ("Du Bois" vs "Du Bois, W.E.B."). Split composites ([s.split("; ") for s in df.author]) and normalize variants before using it as an author-topic input, or a co-authored article becomes a phantom author and one person splits across several rows.

Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table (the uniform shape shared with :func:load_ng20_minilm).

topica.datasets.load_congress 

load_congress(*, return_path: bool = False, as_bunch: bool = False)

Load U.S. House press releases, 2013-2024 (3,120 documents).

A balanced sample of 260 releases per year across twelve years (1,560 Democratic, 1,560 Republican), so the party and year covariates are both well supported. This is the canonical Structural Topic Model example where prevalence depends on a group and on time::

import numpy as np
df = topica.datasets.load_congress()
corpus = topica.from_dataframe(
    df, text_col="text", strip_html=True,
    stopwords=topica.ENGLISH_STOPWORDS, min_doc_freq=10,
)
# party (contrast vs Democrat) + a smooth trend in year
X, names = topica.design_matrix(
    "~ party + spline(year, df=4)", corpus.metadata
)
model = topica.STM(num_topics=20, seed=13)
model.fit(corpus, prevalence=X, prevalence_names=names)

Columns: raw text (press-release body — pass strip_html=True to :func:topica.from_dataframe, since some releases carry markup); date (YYYY-MM-DD) and year for the time covariate; party (Democrat/Republican); and state, member, bioguide_id, title for reference. Chamber is not a column: the sample is House-only, so party and year are the covariates that vary. (Chamber is taken from the source's member metadata, which mislabels a handful of members, so treat the House scope as approximate.) Some releases are in Spanish; a bilingual member's output can surface as its own topic.

Downloaded once and cached. Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table.

Source: Derek Willis's congress-press <https://github.com/dwillis/congress-press>_ (MIT licensed); the underlying press releases are U.S. government works. For the full multi-year archive (raw JSONL) start from that repository — the examples/congress_tutorial.py script walks the raw-to-STM pipeline on it.

topica.datasets.load_reviews 

load_reviews(*, return_path: bool = False, as_bunch: bool = False)

Load Yelp business reviews, balanced across the star scale (1,500 documents).

Three hundred reviews at each of the five star ratings, so the ordinal stars covariate (1–5) is the sentiment signal. Ships inside the wheel and loads offline. The canonical example for a covariate topic model whose outcome is valence::

df = topica.datasets.load_reviews()
corpus = topica.from_dataframe(
    df, text_col="text",
    stopwords=topica.SENTIMENT_STOPWORDS,   # keep 'not'/'no'/'very'
)
X = (df["stars"].to_numpy(float) - 3.0).reshape(-1, 1)   # centered ordinal
model = topica.DMR(num_topics=12, seed=13).fit(corpus, X, feature_names=["stars"])

Pass stopwords=topica.SENTIMENT_STOPWORDS rather than the default: the default ENGLISH_STOPWORDS strips not/no/very, which would collapse "not clean" into "clean" in exactly the study whose outcome is sentiment.

Raw review text in text; the ordinal covariate in stars. Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table. Derived from the Yelp Open Dataset.

topica.datasets.load_ng20_minilm 

load_ng20_minilm(*, return_path: bool = False)

Load 20-Newsgroups with precomputed MiniLM embeddings (5 groups).

The embedding-native counterpart to the text datasets: the same corpus the ProdLDA/FASTopic/BERTopic/Top2Vec examples use, with sentence-transformers all-MiniLM-L6-v2 vectors already computed for every document and every vocabulary term. This lets the embedding topic models run offline, with no sentence-transformers/torch install::

b = topica.datasets.load_ng20_minilm()
bt = topica.BERTopic(reducer="umap", n_components=5).fit(
    [t.split() for t in b.texts], b.doc_embeddings
)
tv = topica.Top2Vec(n_components=5).fit(
    [t.split() for t in b.texts], b.doc_embeddings,
    word_embeddings=b.word_embeddings, vocabulary=b.vocab,
)

Returns a :class:Bunch with attribute access to:

  • texts — list of documents (space-joined in-vocab tokens)
  • labels — newsgroup name per document (numpy object array)
  • doc_embeddings — (n_docs, 384) float16 MiniLM vectors
  • vocab — list of vocabulary terms
  • word_embeddings — (vocab, 384) float16 MiniLM vectors
  • meta — provenance string

Embeddings are stored as float16 to keep the download small. topica's own models accept them directly (inputs are coerced internally); cast to float32 only for an external tool that needs it. Downloaded once and cached. Pass return_path=True for the cached .npz path instead of the Bunch.

Threaded discussion needs the reply tree, not just a text table. load_threads is the ThreadTM vignette: two subreddits with every comment's parent index preserved, returned as a Bunch whose documents and parents line up for a turnkey fit. See the threaded conversations example.

topica.datasets.load_threads 

load_threads(*, return_path: bool = False)

Load the two-subreddit threaded Reddit corpus (5,042 comments, 171 trees).

The :class:~topica.ThreadTM reply-tree vignette. Two subreddits, chosen to make the model's point honestly:

  • askscience — technical Q&A; replies genuinely answer their parent, so the reply tree carries topic structure and persistence is identifiable.
  • pokemontrades — the deepest reply trees in the source corpus, yet its replies coordinate trades ("added you on DS") rather than respond on-topic, so persistence is not identifiable. Tree depth is not persistence.

Unlike the flat text datasets, threaded data cannot go through :func:topica.from_dataframe (that discards the reply tree), so this returns a :class:Bunch whose rows stay aligned to the parents index. Fit is turnkey::

b = topica.datasets.load_threads()
topica.enable_experimental()  # ThreadTM is experimental
model = topica.ThreadTM(8, coupling="parent").fit(
    b.documents, parents=b.parents, covariates=b.subreddit
)
model.persistence()   # read `reliability` before claiming persistence

The Bunch carries:

  • documents — token lists (lowercased, letters-only, min length 3, English stopwords removed), one per row and in row order (empty rows are kept so parents stays valid).
  • texts — the raw, untokenized comment text (retokenize this yourself for a different vocabulary; keep every row to preserve the parents index).
  • parents — list of ints: the 0-based row index of the comment each row replies to, or -1 for a thread root. A parent's index is always smaller than its child's, so the array is safe to pass straight to fit.
  • subreddit — the per-row subreddit, the prevalence/content covariate.
  • thread_root — the root comment id shared by every row in a tree.
  • timestamp — unix seconds (may be null).
  • df — the full table as a DataFrame.

Source: ConvoKit reddit-corpus-small (Chang et al. 2020); see examples/build_reddit_threads.py to regenerate. Downloaded once and cached. Pass return_path=True for the cached CSV path instead of the Bunch.

topica.datasets.get_data_home 

get_data_home() -> Path

Return the directory where downloaded datasets are cached.

Defaults to ~/.cache/topica/datasets; override with the TOPICA_DATA_HOME environment variable. The directory is created if it does not exist.

topica.datasets.clear_cache 

clear_cache() -> None

Delete every cached (downloaded) dataset file. Vendored datasets are unaffected; the next load_* call re-downloads what it needs.