Now that I know everything I post, deleted or not, gets archived and is searchable by anybody, I no longer feel safe participating on reddit.com. Thank you pushshift! And thanks for making it extremely easy to opt-out with your super convenient megathread that always is promptly responded to :) Just kidding, obviously.

inb4 "Google does it too."

inb4 "What you post is public information."

Just so confusing why it's down now when we could still be using it for 99% of things. Any info?

Hello!

As context, I am doing medical research for my PhD and a portion of my project involves scraping posts from a particular subreddit and analyzing them. At first, I was using Praw and my Reddit credentials, but I wasn't able to scrape as may posts as I need for robust data. (I'm trying to get at least 10k posts from the past 5 years off of a one subreddit.) I wasn't able to scrape more than 200 at a time, and at one point, I noticed a lot of posts I scraped were duplicated in the dataset.

Now I'm thinking I really need to use Pushshift, but I am unable to pull because I am not a moderator on Reddit. I am wondering if anyone can help me, or alternative ways around? As context, I'm totally new to coding. Thank you!!!

Hello,

A post I made recently on a subreddit was removed due to my comment history from a different subreddit. The 2 subreddits have nothing to do with each other so there is no overlap. Said Comments were deleted by myself, and I haven't been able to find them on the popular archive websites. I have several questions

1. How was this mod able to see my deleted Comments?

2. If I make a removal request, will my deleted reddit history still be easily accessible?

I'm aware nothing is ever truly gone, but the fact that this mod was able to use my deleted comment history against me is rather concerning.

So I used the .json addition to the url and I looked at the API for the subreddit of a post, and the title shows up but I can't find the author of the post! Can someone help me out?

There is a reasonable expectation that some content on a public website will be scraped, archived, and saved. There is no reasonable expectation that literally all of it will. Aside from being a possible GDPR violation, it's just downright disgusting to hoard all this data.

I imagine the token response here is going to be "Well people posted content on a public site, therefore they should expect it to live forever". This is incredibly disingenuous for three reasons:

1. Most major public sites don't have API's that allow developers to ingest all user data. Twitter, for example, doesn't allow developers to ingest all user data. They can only take in a specific portion. If you don't have that many followers and you delete your twitter, no one is going to have your tweets, most likely. This service, on the other hand, digests all user data every few minutes. Even if no one saw your posts and you deleted them a few hours later, they're still stored.

2. Reddit provides a deceptive 'delete' button to lull users into a false sense of security that they can control their content to a degree. This is obviously useless, since their data is essentially archived the moment they submit anything. It would actually be better to have no delete button, since users would then know that they can't edit anything, and thus will choose to post more carefully. The way it is now is just blatant lying, and if people knew that their data was being saved every few minutes, they would think twice about posting. If you deny this, you are delusional.

3. There is a disturbing amount of teenagers/young adults who post private information temporarily, reasonably thinking that no one is going to save it. For instance, they might post something in their local city subreddit for a couple hours about something mundane, and then delete it. Why would someone save a mundane question on a local city subreddit of less than a couple thousand users, after all? What they don't know is that their data is being saved regardless, and they're unintentionally leaving a bread crumb to their real identity. A bread crumb that anyone with malicious intent can weaponize at any moment. (Some will say that this is their fault - but guess what, they're teenagers, they don't think things through. As it is now, you're enabling potential future abuse)

4. Reddit's own guidelines state not to repost deleted information. This service runs right in the face of that.

I'm tired of this developer pretending that he cares about people's privacy. If he really did, he would prioritize deletion requests, and would have built a simple bot that detects when user's delete their accounts and then updates the database accordingly.

Unironically revolting.

Hello. I am on a wild goose chase from here. I am curious how pushshift handles privacy.

Reddit posts contain volumes of personally identifying information belonging to EU citizens, but you don't seem to anonymize their names in your database. Are you exempt from the GDPR?

If the GDPR applies, do the legal bases by which Reddit processes information of EEA members also apply to pushshift?

Also, are you sure long-term storage of User Content (the IP of the user), including what they have deleted, is permitted by license? (I assume you are using the Reddit API and bound by its terms, but frankly this is over my head).

Reddit Pushshift is so useful that there should be an app for it (especially since the reddit built in search function is so bad) I wonder if developers are working on it?

The title says it all. The megathread appears to not work with most of the requests going unheard. What's going on?

Can you have comments deleted like you can have account data deletion?

I believe there was some discussion of maybe adding resources for researching parler activity, and I know there are definitely researchers scraping archival material and posting to the internet archive. Anything available or being developed through pushshift?

One of the main things that breaks us from building meaningful communities on Reddit is that you can't really ban anyone. Of course, you can't really ban anyone anywhere ( there will always be Tor and VPNs and fake phone numbers and whatnot ), but it would be good if mods could raise the bar on the technical skills necessary for evading bans on their subs.

So we need a service that would be capable of telling you that a certain group of accounts from China and Russia are astroturfing on your subreddit, or that two or more users look suspisciously like the same person.

Alright, so here are the nuts and bolts:

I believe you shouldn't mess with NLP. I mean, you can do that if people are willing to pay for it, and if you can convince SparkNLP and ThunderSVM to do the job in a way that you won't go bankrupt, but I believe that would be overkill.

Most people who evade bans are not actively trying to hide their footprints, so I think it'd be reasonable to simply compare timestamps and community intersections. I wrote a short article here where I do some manual analysis of these aspects.

So the trick would be to run either Dynamic time warping and/or Connectionist Temporal Classification Loss over the timestamps of everyone on Reddit. I never tested this CTC model, maybe it's so efficient that you won't have to mess around with anything else.

Think of every 5 minute interval as being a "word" and every 7 days as being a "phrase". If an user posts on an interval, it gets a "one" label, if he doesn't, it gets a "zero" label. You can get more fancy than this but I think this example is enough to see that we can transport the same techniques which are used for speech recognition to this problem domain.

Hi I am currently doing a social data science project where I want to look at the comments from the subreddit (WallStreetBets) WSB and use a sentiment analysis to see if the sentiment regarding specific stock will provide any predictive value for those stocks. I have bit of coding experience but not specifically with retrieving comments from reddit before. I have tried to run some python code using pushshift but so far I haven't gotten precisely what I was looking for. I have managed to collect approximately 23k comments but only dating back a day or so from the subreddit WSB. I am looking for a way to collect comments for at least the last 2 month, but preferable for longer.
Is this a possible task and if so what do I need to do.
All help is appreciated, thank you.

Who would we contact for bug-reports? Is there a github that anyone knows of?

So I'm trying to find out a certain /r/askscience comment I had saved because it had some great info. That sub has very strict rules about comments being sciencey or something so it got removed. I was able to get the link for the comment through inspect element. Here it is

/r/askscience/comments/wei5x0/why_does_coding_work/iiowwz4/

I don't know the author or the content of it so wasn't able to search for it with pushshift.

Do you have to be a subreddit moderator to gain access to Pushshift? This page, where you go if you want to request access, seems to imply that you need to be a moderator to get access to Pushshift. I'm not a moderator; I simply want to search particular subreddit posts and their comments for particular phrases I'm interested in. Thank you.

I'm researching some specific users, and it appears they have deleted some posts. This would be hard to demonstrate if I hadn't paid attention to their post history in the past.

Any tips on discovering deleted submissions? I was pretty sure pushshift stored this info, so maybe I'm doing something wrong.

Is there any way to access these? Are people just using the Reddit API instead?

From what i read in the docs pushshift scrapes comments quickly and doesn't count edits. What if we want to see upvotes of comments? Is pushshift showing the upvotes ? giving at least a few days for people to vote?

So I’ve made this long post on /r/drugs with the idea to either regularly repost it. Or somehow figure a way out for /r/pushshift shift to always include the post for anyone using an api to download all posts.

Besides ethical reasons the subreddit is about a controversial topic and we’re still heavily being stigmatized. To help reduce this we want to remind the researchers we’re real people and deserve to be treated as such. What do you guys think of this? Should there be stricter control from reddit about who can and can’t use pushshift? Am I being to aggressive?

Any constructive feedback is welcome.

Original post bellow the line

Dear researchers scraping data from

this subreddit. Please follow these guidelines and message the moderators about our research

Reddit is a massive social network and heavily used for all kinds of research, especially regarding drugs and drug use. This also includes research chemicals, recovery, activities related to them, etc. Because this is allowed by the ToS/User Agreement of Reddit there is little we can do to stop this from happening. However, we can and are asking for any researcher that uses data scrapes from this subreddit to announce themselves to the moderators. This is not only to because we the moderators are open to cooperate with them, but it's also the right thing to do ethically. Furthermore, we have access to additional information that is not publicly available like traffic statistics, removed posts/comments, etc.
If you're using data from here then send a modmail or private message to me (/u/cyrilio) with what you're researching and what you will use the data for.

Thanks for understanding and cooperating,

/u/cyrilio and the other mods

For good measure I'm mentioning some key words here so anyone trying to filter in this post so anyone will get to see it anyway:
suicide, opiods, heroin, fentanyl, flualprazolam, MDMA, drug checking, pill testing, addicted, help, cocaine, speed, where to buy, bitcoin, looking for, what are the effects of, how much should I take, psychedelics, LSD, 1V-LSD, 1p-LSD, depressed, is 10 pills a deadly dose?

Are you a researcher and do you need more information about ethics? Then read this:

The Ethics of Web Crawling and Web Scraping in Cybercrime Research: Navigating Issues of Consent, Privacy, and Other Potential Harms Associated with Automated Data Collection ^source

Taking such a considered approach to privacy—before, but also during and after data collection has occurred—is vital to minimize any potential harms to subjects. The consolidation of a significant quantum of personal data has the potential to uncover associations or reveal subjects through collection of various information across different platforms. This can include data obtained through forums or social media, such as a list of contacts (or friends), the correspondence between parties, photographs, videos, “tags” and other metadata (Alim, 2013; Gold & Latonero, 2018). In circumstances where criminological researchers collect data pertaining to individuals who are desirable to law enforcement (e.g., users discussing the commission of a crime on web forums, those selling illicit items on cryptomarkets, and people sharing illicit media files), pressure could be applied to the researcher to provide such data. This could facilitate the arrest or prosecution of individuals (Israel, 2004; Martin & Christin, 2016) as well as other harms through instances of internet vigilantism (Chang, 2018). Elsewhere, the collection of personal data also presents a risk for victimization of new crimes—where any such information made publicly available could be used for spamming, phishing, and/or identity theft purposes (Giles et al., 2010; Menczer, 2011; Thelwall & Stuart, 2006). Finally, where scraped data could be analyzed, published, and subsequently read by and cause trauma to the subject, there is potential for re-victimization (Gueta et al., 2020). For example, in the examination of commonly experienced crimes and victimization on social media(Gorro et al., 2017), reproduction of profiles, photos, posts, and stories may be easily encountered by the victim or other individuals known to the victim. While the likelihood of such activities occurring is low given the expansive and global nature of web-based activities, this risk is nevertheless real and requires that the researcher approach with caution.

Given the potentially sensitive nature of personal data being scraped (regardless of data type), (criminological) researchers seeking to employ such methods must take several steps to minimize the potential for harm against subjects (e.g., those buying/selling items on digital marketplaces, posting comments or other media online, and even those persons contained within media files). This should be accomplished through a process of anonymizing individual outputs, avoiding analysis of identifiable information (Magdy et al., 2017; Xu et al., 2020), and securing the storage and transmission of any sensitive data (Tsatsou, 2014). This includes not only a subject’s “user name” (where applicable), but also any other personal data (e.g., verbatim quotes, extracted biometric data) that might infer information back to a particular subject or even the source website (Décary-Hétu & Aldridge, 2015; Fussell, 2019). Practically, when researchers may not be able to strip all personal identifiers from data without compromising its useability (Israel, 2004), data should be reported on an aggregated level (Alim, 2014; Bouwman et al., 2013). This may prove particularly problematic for data sharing among researchers and in many cases will prohibit such practices. Furthermore, researchers must exercise care beyond the data collection process, and be attentive to data security, particularly as it pertains to data storage and processing procedures (Chiauzzi & Wicks, 2019; Magdy et al., 2017; Xu et al. 2020). To mitigate any potential harm, researchers are advised to maximize confidentiality and implement robust security safeguards, which include both strict access controls and data encryption (Alim, 2014; Bouwman et al., 2013; Gold & Latonero, 2018; Tsatsou, 2014). Finally, researchers need also be aware of any reporting requirements (e.g., being a mandatory reporter in a particular jurisdiction) and be mindful of those requirements prior to, during, and after data collection.

More resource:

• Ethical issues in research using datasets of illicit origin

• Using Twitter as a Data Source: An Overview of Ethical, Legal, and Methodological Challenges

• Ethics in Online Research; Evaluating the ESRC Framework for Research Ethics Categorisation of Risk

• AoIR on Ethics (includes multiple guides)

PS On a personal note. If I find out that you have seen this post or could/should've been aware of it I'm making it my goal to trash your research so hard no one will ever want to read it or believe what was written.
That's a promise.

Researchers wanting to post a survey

Due to the large number of poorly designed surveys in /r/Drugs, it is required for researchers to message the moderators before posting to ensure that the questions are well designed.

You can read about our guidelines here: Link

Thanks

EDIT 27-12-2021

Came across this paper that sums up the state of affairs when it comes to using reddit data in research.

Quote:
Unfortunately, many attempts at ethical disguise fail. Reagle (2021b) interrogated 22 Reddit research reports, with 19 claiming to reword phrases to disguise their sources, with only 8 of those succeeding. Some researchers simply failed to reword phrases. One researcher collected and reported data they said they would not. Two others failed to scrub their reports of locatable information after they opted for heavier disguise during their reports’ review and editing. Some failed to introduce enough change into their phrases. We have no evidence Redditors were affected by these failures, but the potential for their harm (e.g., embarrassment, harassment, or changes to employment and legal status) and to the researchers’ reputations exist.

Source: Spinning words as disguise: Shady services for ethical research? By Joseph Reagle, Manas Gaur

EDIT 1: I still don't know if it's allowed to use the reddit api to scrape data for other use than intended (research). Would be nice if reddit could somehow better aid anyone wanting to use data from reddit to do research with. Like helping create some kind of anonymized dateset to prevent potential harms (like the ones mentioned in the paper referenced above).

Furthermore,
Looking at rule 2 d of the reddit Api terms.

Quote:
>You will comply with any requirements or restrictions imposed on usage of User Content by their respective owners, which may include "all rights reserved" notices, Creative Commons licenses or other terms and conditions that may be agreed upon between you and the owners.

If I add "all rights reserved" to anything I post/comment on reddit. Does that mean legally I can deny anyone (other than reddit and 3rd party reddit apps) to use my posts and comments?

Does reddit enforce this? Or at least provide tools or take some action to minimize abuse?

Note: I edited my original post on /r/redditmodCouncil so copy pasted them over here.

all rights reserved ©

PS I've been working on a letter to the API team and legal of reddit. I think this is a serious issue that needs addressed.

how do i use the data dumps instead ?

I recently requested removal for a deleted account and I wanted to know how long does it take.