I'm planning a small full-text index on a 2 TB SSD, with the data recompressed into 1 MB zstd blocks on a 12 TB HDD. Finding matches should be fast, but a common word could mean millions of posts to fetch.

Is an HDD fine for that, or did you need SSDs for everything? Any real numbers (drive type, index size, query time) would help a lot. Thanks!

IM DESPERATE

I'm hoping someone here with more experience can point me in the right direction because I've been going in circles trying to figure this out.

I'm working on a personal music discovery project. I want to search approximately five years of discussions in a particular subreddit and build a massive list of artists and bands that people have recommended over the years.

The biggest challenge is that I don't want to rely on Reddit's limited search results or just collect the most popular posts. I'm trying to dig as deep as possible, including older discussions, overlooked threads, obscure recommendations, and names buried in comment replies.

Ideally, I'd like to:

- Search hundreds or thousands of historical threads using multiple keywords.

- Retrieve complete comment sections, including nested replies.

- Identify recommendation-related discussions while filtering out unrelated posts.

- Export the collected information into CSV files for further processing.

- Extract and deduplicate artist names to create one comprehensive list.

Here's what I've already tried:

I installed Python, PRAW, and Selenium on my Windows computer.

Unfortunately, Reddit requires API approval for my intended use, and my application was denied. I also tried Selenium with Firefox, but Reddit returned a network security block.

I've looked into a few open-source projects, but I'm unsure which ones are reliable or currently working.

What I'm looking for is a completely free solution that can handle a large amount of historical data. I don't mind letting a script run for several hours if necessary.

Has anyone successfully collected several years of Reddit posts and comments recently?

Are there any open-source tools, public Reddit archives, downloadable datasets, or other legitimate approaches you'd recommend?

I'm particularly interested in methods that can retrieve older discussions and their full comment trees rather than stopping after the first few hundred results.

This is strictly for personal music discovery, not commercial use or collecting personal information.

I'd really appreciate hearing what has actually worked for people rather than just theoretical suggestions.

Thanks in advance!

Reddit's own search caps out (250 results per query/sort, about 490 on the newer endpoint, 250 for comments). Combining about 25 sort/time combinations got me roughly 3,000 at best, so I've given up on that route.

What I want is every post and every comment containing a given keyword, across Reddit's entire history.

I've tried Arctic Shift's search API. It requires a subreddit or author, and comment search often times out (422) even when I go slowly.

So my questions for people who know the archives:

- Is there any way to run a keyword search across all subreddits without downloading the full dumps?

- For comment search on Arctic Shift or PullPush within a large subreddit, what settings or pacing work best?

- If the dumps really are the only complete option, can I avoid the full set, for example by using per-subreddit files?

- Do any paid services actually give complete results, or are they all built on the same archives?

Does PushShift respond to submitted requests? they seem to be ignored

I would rather do it now when they are asking nicely and not in the future when they ask in some less nice way. They haven't said, but I assume pullpush and arctic shift have received similar requests, and will probably get the less nice request in not too long.

Reddit recommends r/reddit4researchers for all research use. I can't say I've seen much success with that approach, but it will soon be the only one left.

Hi!

I'm doing right now the thesis for my degree, to analyse the posts of some subreddits related to mental health, and the pushshift data is perfect for this. I was wondering if I could be able to use this data only for academic purposes to do an investigative thesis.

I'm not really familiar with the T&C so I wanted to know what do I have to do to be able to use pushshift, thanks!

Hi everyone,

I’m currently trying to work with the Pushshift Reddit dataset, but I’m running into a major issue and could really use some help.

I’m not able to retrieve any data at all — the torrent links either don’t start downloading or just fail completely. I’ve tried multiple links and different setups, but nothing seems to be working.

Is there an updated or working source for the dataset? Or any workaround that others have used successfully to access the data?

If anyone has faced this before or knows what might be going wrong, I’d really appreciate your guidance.

Thanks in advance!

Hi all, I'm trying to scrape a subreddit and comments for a linguistics project. I don't know much about it and I'm flying pretty blind. Anyway I created a reddit account and gave it access. I have a confirmation mail and it's in my permitted apps. When I authorize I get the below message:

{"detail":"User is not an authorized moderator."}

Any idea what I'm doing wrong?

Are there ways to recover an image from a user-deleted post? Wayback machine doesn't have the pictures

Hello everyone,

We are currently experiencing a major outage of the Pushshift API due to issues at our physical colocation space. New data or responses may not be received at the moment. We apologize for the inconvenience and will keep you posted with updates. The data will be backfilled once all the systems are up and operational. Thank you for your patience,

Team Pushshift support

Update: At 5:50 PM Eastern, all services were resumed and operational. Data is being backfilled and all operations continue as usual. Thank you for your patience.

So I use push shift I have a login but the interface is a nightmare and it’s a buggy. I hate using it. For years I was using Arctic shift and pull push but now those don’t show deleted posts and comments. Is there a push shift alternative that will take my login that is less buggy and more reliable? Or is there a way to login to Arctic shift to get more info?

https://github.com/wheynelau/pushshift-rs

Just wanted to share a tool I've been using for my own personal processing. Hope it helps someone out.

The name is a little misleading it's only for the reddit data. There's also no filters to catch redact or anything.

What it does:

The usual monthly uploads are for all subreddits. It is currently only a command line tool. This tool has two use cases:

1. It filters out the subreddit you specify.

2. Additional process command that can be used to build data for LLM processing. Every text output is a full reddit thread from the post to an answer.

More details can be found in the repo.

Good evening everyone, can anyone recommend a method that allows me to sample Reddit posts from October 2023 to July 2025?

I am disabled and want to be a speech pathologist. I was in crisis after a family member said it is respectable for disability parents to wish their disabled children whose conditions are not deadly to wish their kids would die. I was frightened given that when I argued against them they said I was being disrespectful of disability parents’ struggling and being dumb. I foolishly decided to ask on here disability parents who wish this to explain whether there are supports that would make them stop wishing that but didn’t clarify the reason I was asking is that I am disabled and scared. I deleted within an hour upon realizing my mistake and apologized publicly in both forums. If I were to get doxxed, I wouldn't want that to affect my career, since it does not reflect who I am in real life in any way shape or form; just a moment of crisis and poor judgement. If I opt out from pushshift now, will that get deleted? How do I opt out from pushshift? Can I still get that deleted post deleted from pushshift?

I downloaded a subreddit and would like to use its data as a source in my notebooklm notebook. I see there is comments and submissions, i was thinking of just converting them to markdown format, but I'm having issues using the tool "markitdown".

Or should I be formatting it in a better format for LLM's to consume?

For my master thesis I am trying to collect data from subreddits on series. For instance, I want to now how many comments and submissions have been done on the r/strangerthings page in the first month after the release (july 2016). I have tried to write a script in Python with Praw, but everything I do ends up with 0 submissions and 0 comments as a result.

I tried with Pushshift as well but that doesn't seem to work either so I came here and read something about the dumps. Is the information I need in there?

Or is something going wrong with my script in Python? Anybody with more knowledge on this subject that can help me?

Thanks in advance!

Are there any good FAQs or Quick Start guides/posts to reference when getting started with a project involving this data?

I work for a hospital, writing queries to their EHR system, so I'm familiar with data in general. Pretty comfortable with writing SQL queries and the like, though I'm less experienced with the steps prior to that.

For this data format, are there any recommended guides how best to load it in and prep it for analysis? I've heard DuckDB recommended in regards to how to store it, but wanted to ask other users of this data what they did before trying to reinvent the wheel.

I'm working on a university research project that analyzes discussions in personal-finance subreddits. Right now, I'm using the per-subreddit data dumps up through 2024.

For my study, having full daily conversations is important, including submissions and comments that were deleted or removed at some point. Is there a way to retrieve that missing data?

Any help on this is highly appreciated. Thanks in advance!