AI & MLNLTKPython

Exploring Reddit Discussions with Tokens and Collocations

A discussion can be described through individual words, but many of its useful expressions contain more than one word. A notebook in my Reddit Canada collection explores both views.

It uses PRAW to collect posts and NLTK for tokenization, stopwords, and bigram collocations, alongside word-cloud exploration. This is classical text analysis rather than a generative-language application.

From posts to tokens

Tokenization determines the units that will be counted. Stopword handling then affects which units remain visible in frequency summaries.

Those choices can change the apparent subject of a discussion. Removing common words can make a cloud more readable, but excessive filtering can also erase context needed to interpret a phrase.

Looking for words that occur together

The notebook includes NLTK's BigramCollocationFinder. A bigram analysis examines adjacent word pairs, which can help retain expressions that an individual-word count splits apart.

That does not make every frequent pair meaningful. Repeated titles, quotations, boilerplate, and a small number of prolific contributors can influence the results.

For a useful interpretation, I would return to representative posts and inspect the phrases in context.

What a word cloud can show

A word cloud is a compact view of a chosen counting rule. It can suggest themes to investigate, but it does not establish why people used a term or how representative the collection is.

The next useful step would be to document the collection window, selection rules, and preprocessing, then compare individual-word and phrase summaries on the same sample.

This notebook belongs in my AI history as groundwork in text representation and interpretation. Those questions remain relevant when a later system uses embeddings or asks an LLM to summarize the same discussions.

Background