Skip to content

Datasets

Bundled example datasets for quickstarts and worked examples. Small datasets ship inside the wheel and load offline; larger ones are downloaded once from GitHub on first use and cached locally (under ~/.cache/topica/datasets, or TOPICA_DATA_HOME). The text loaders return a pandas DataFrame ready for from_dataframe; pass return_path=True for the cached CSV path without pandas.

import topica

df = topica.datasets.load_gadarian()
corpus = topica.from_dataframe(
    df, text_col="open.ended.response", stopwords=topica.ENGLISH_STOPWORDS
)

The embedding-native models want vectors, not a token corpus. load_ng20_minilm covers that case: a 20-Newsgroups subset with MiniLM sentence embeddings precomputed for both documents and vocabulary, so ProdLDA/FASTopic/BERTopic/Top2Vec run offline with no sentence-transformers/torch install. It returns a Bunch (attribute access), not a DataFrame.

b = topica.datasets.load_ng20_minilm()
docs = [t.split() for t in b.texts]
# The default density clusterer (UMAP -> HDBSCAN) finds only ~3 topics here, with
# ~80% of the documents in one topic. That is NOT a topica defect: the reference
# umap-learn + HDBSCAN pipeline finds the same few-topic structure on this corpus at
# this min_cluster_size (see parity/bertopic_umap_default_compare.py). It is genuine,
# coarse density structure. For a finer or fixed number of topics, lower
# min_cluster_size, or use a fixed-K clusterer with reduce_frequent, e.g.:
bt = topica.BERTopic(
    clusterer="kmeans", num_clusters=5, reduce_frequent=True, seed=1,
).fit(docs, b.doc_embeddings)

topica.datasets.load_gadarian 

load_gadarian(*, return_path: bool = False, as_bunch: bool = False)

Load the Gadarian & Albertson immigration experiment (341 documents).

The canonical stm prevalence example. Open-ended survey responses with an anxiety-prime treatment and pid_rep (Republican identification). Raw, untokenized text lives in the open.ended.response column, so build a corpus with stopword removal::

df = topica.datasets.load_gadarian()
corpus = topica.from_dataframe(
    df, text_col="open.ended.response", stopwords=topica.ENGLISH_STOPWORDS
)

This dataset is bundled in the wheel and loads offline. Pass return_path=True for the CSV path instead of a DataFrame, or as_bunch=True for a :class:Bunch whose .df is this table (the uniform shape shared with :func:load_ng20_minilm).

topica.datasets.load_poliblog 

load_poliblog(*, return_path: bool = False, as_bunch: bool = False)

Load the CMU 2008 political blog corpus (a 2,000-document sample).

The text in the text column is already tokenized and stemmed (space-separated), as in the stm poliblog vignette, so no stopword removal is needed::

df = topica.datasets.load_poliblog()
corpus = topica.from_dataframe(df, text_col="text")

Covariates: rating (Liberal/Conservative), day, blog. Downloaded once and cached. Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table.

topica.datasets.load_dubois 

load_dubois(*, return_path: bool = False, as_bunch: bool = False)

Load Du Bois-era articles from The Crisis, 1910-1934 (704 documents).

Raw text in the text column; covariates year, decade, volume, issue, author, subjects. Build a corpus with stopword removal::

df = topica.datasets.load_dubois()
corpus = topica.from_dataframe(
    df, text_col="text", stopwords=topica.ENGLISH_STOPWORDS
)

The corpus holds a few (3) exact-duplicate articles reprinted across issues; drop them with df.drop_duplicates("text") if a fit should not double-count them. Downloaded once and cached. Pass return_path=True for the CSV path.

Note on author for :class:~topica.AuthorTopic: this field is dominated by Du Bois (about 675 of 704 articles) and contains delimited composites ("Du Bois; Gruening") and name/initial variants ("Du Bois" vs "Du Bois, W.E.B."). Split composites ([s.split("; ") for s in df.author]) and normalize variants before using it as an author-topic input, or a co-authored article becomes a phantom author and one person splits across several rows.

Pass return_path=True for the CSV path, or as_bunch=True for a :class:Bunch whose .df is this table (the uniform shape shared with :func:load_ng20_minilm).

topica.datasets.load_ng20_minilm 

load_ng20_minilm(*, return_path: bool = False)

Load 20-Newsgroups with precomputed MiniLM embeddings (5 groups).

The embedding-native counterpart to the text datasets: the same corpus the ProdLDA/FASTopic/BERTopic/Top2Vec examples use, with sentence-transformers all-MiniLM-L6-v2 vectors already computed for every document and every vocabulary term. This lets the embedding topic models run offline, with no sentence-transformers/torch install::

b = topica.datasets.load_ng20_minilm()
bt = topica.BERTopic(reducer="umap", n_components=5).fit(
    [t.split() for t in b.texts], b.doc_embeddings
)
tv = topica.Top2Vec(n_components=5).fit(
    [t.split() for t in b.texts], b.doc_embeddings,
    word_embeddings=b.word_embeddings, vocabulary=b.vocab,
)

Returns a :class:Bunch with attribute access to:

  • texts — list of documents (space-joined in-vocab tokens)
  • labels — newsgroup name per document (numpy object array)
  • doc_embeddings — (n_docs, 384) float16 MiniLM vectors
  • vocab — list of vocabulary terms
  • word_embeddings — (vocab, 384) float16 MiniLM vectors
  • meta — provenance string

Embeddings are stored as float16 to keep the download small. topica's own models accept them directly (inputs are coerced internally); cast to float32 only for an external tool that needs it. Downloaded once and cached. Pass return_path=True for the cached .npz path instead of the Bunch.

topica.datasets.get_data_home 

get_data_home() -> Path

Return the directory where downloaded datasets are cached.

Defaults to ~/.cache/topica/datasets; override with the TOPICA_DATA_HOME environment variable. The directory is created if it does not exist.

topica.datasets.clear_cache 

clear_cache() -> None

Delete every cached (downloaded) dataset file. Vendored datasets are unaffected; the next load_* call re-downloads what it needs.