I am currently doing a massive analysis of Reddit's entire publicly available comment dataset. The dataset is ~1.7 billion JSON objects complete with the comment, score, author, subreddit, position in comment tree and other fields that are available through Reddit's API.

I'm currently doing NLP analysis and also putting the entire dataset into a large searchable database using Sphinxsearch (also testing ElasticSearch).

This dataset is over 1 terabyte uncompressed, so this would be best for larger research projects. If you're interested in a sample month of comments, that can be arranged as well. I am trying to find a place to host this large dataset -- I'm reaching out to Amazon since they have open data initiatives.

EDIT: I'm putting up a Digital Ocean box with 2 TB of bandwidth and will throw an entire months worth of comments up (~ 5 gigs compressed) It's now a torrent. This will give you guys an opportunity to examine the data. The file is structured with JSON blocks delimited by new lines (\n).

____________________________________________________

One month of comments is now available here:

Download Link: Torrent

Direct Magnet File: magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&dn=RC%5F2015-01.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969

Tracker: udp://tracker.openbittorrent.com:80

Total Comments: 53,851,542

Compression Type: bzip2 (5,452,413,560 bytes compressed | 31,648,374,104 bytes uncompressed)

md5: a3fc3d9db18786e4486381a7f37d08e2 RC_2015-01.bz2

____________________________________________________

Example JSON Block:

{"gilded":0,"author_flair_text":"Male","author_flair_css_class":"male","retrieved_on":1425124228,"ups":3,"subreddit_id":"t5_2s30g","edited":false,"controversiality":0,"parent_id":"t1_cnapn0k","subreddit":"AskMen","body":"I can't agree with passing the blame, but I'm glad to hear it's at least helping you with the anxiety. I went the other direction and started taking responsibility for everything. I had to realize that people make mistakes including myself and it's gonna be alright. I don't have to be shackled to my mistakes and I don't have to be afraid of making them. ","created_utc":"1420070668","downs":0,"score":3,"author":"TheDukeofEtown","archived":false,"distinguished":null,"id":"cnasd6x","score_hidden":false,"name":"t1_cnasd6x","link_id":"t3_2qyhmp"}

UPDATE (Saturday 2015-07-03 13:26 ET)

I'm getting a huge response from this and won't be able to immediately reply to everyone. I am pinging some people who are helping. There are two major issues at this point. Getting the data from my local system to wherever and figuring out bandwidth (since this is a very large dataset). Please keep checking for new updates. I am working to make this data publicly available ASAP. If you're a larger organization or university and have the ability to help seed this initially (will probably require 100 TB of bandwidth to get it rolling), please let me know. If you can agree to do this, I'll give your organization priority over the data first.

UPDATE 2 (15:18)

I've purchased a seedbox. I'll be updating the link above to the sample file. Once I can get the full dataset to the seedbox, I'll post the torrent and magnet link to that as well. I want to thank /u/hak8or for all his help during this process. It's been a while since I've created torrents and he has been a huge help with explaining how it all works. Thanks man!

UPDATE 3 (21:09)

I'm creating the complete torrent. There was an issue with my seedbox not allowing public trackers for uploads, so I had to create a private tracker. I should have a link up shortly to the massive torrent. I would really appreciate it if people at least seed at 1:1 ratio -- and if you can do more, that's even better! The size looks to be around ~160 GB -- a bit less than I thought.

UPDATE 4 (00:49 July 4)

I'm retiring for the evening. I'm currently seeding the entire archive to two seedboxes plus two other people. I'll post the link tomorrow evening once the seedboxes are at 100%. This will help prevent choking the upload from my home connection if too many people jump on at once. The seedboxes upload at around 35MB a second in the best case scenario. We should be good tomorrow evening when I post it. Happy July 4'th to my American friends!

UPDATE 5 (14:44)

Send more beer! The seedboxes are around 75% and should be finishing up within the next 8 hours. My next update before I retire for the night will be a magnet link to the main archive. Thanks!

UPDATE 6 (20:17)

This is the update you've been waiting for!

The entire archive:

magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

Please seed!

UPDATE 7 (July 11 14:19)

User /u/fhoffa has done a lot of great work making this data available within Google's BigQuery. Please check out this link for more information: /r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/

Awesome work!

Please read the community article: https://huggingface.co/blog/tensonaut/the-epstein-files

I've processed all the text and image files (~25,000 document pages/emails) within individual folders released last friday into a two column text file. I used Googles tesseract OCR library to convert jpg to text.

You can download it here: https://huggingface.co/datasets/tensonaut/EPSTEIN_FILES_20K

For each document, I've included the full path to the original google drive folder from House oversight committee so you can link and verify contents.

The Pornhub Dataset provides a comprehensive collection of data sourced from ph, encompassing various details from MANYYY videos available on the platform. The file consists of 742.133 lines of videos.

This dataset contains a diverse array of languages, with video titles indicating that it is 53 different languages.

Note: This dataset contains sensitive content and is intended solely for research and educational purposes. 😉 Please ensure compliance with all relevant regulations and guidelines when using this data. Use responsibly. 😊

Pornhub Dataset ❤️

You have probably seen most of these, but I thought I'd share anyway:

Spreadsheets and Datasets:

• https://www.worldometers.info/coronavirus/

• John Hopkins University Github confirmed case numbers.

• Google Sheets From DXY.cn (Contains some patient information [age,gender,etc] )

• Kaggle Dataset

• Strain Data repo

• https://covid2019.app/ (Google Sheets, thanks /u/supertyler)

• ECDC (Daily Spreadsheets, Thanks /u/n3ongrau)

Other Good sources:

• BNO Seems to have latest number w/ sources. (scrape)

• What we can find out on a Bioinformatics Level

• DXY.cn Chinese online community for Medical Professionals *translate page.

• John Hopkins University Live Map

• Mutations (thanks /u/Mynewestaccount34578)

• Protein Data Bank File

• Early Transmission Dynamics Provides statistics on the early cases, median age, gender etc.

[IMPORTANT UPDATE: From February 12th the definition of confirmed cases has changed in Hubei, and now includes those who have been clinically diagnosed. Previously China's confirmed cases only included those tested for SARS-CoV-2. Many datasets will show a spike on that date.]

There have been a bunch of great comments with links to further resources below!
[Last Edit: 15/03/2020]

The full Epstein document dump from justice.gov is publicly available but practically unsearchable. I indexed all 1.4 million files and built a search interface over them.

Also used this GitHub repo which has extra metadata, transcriptions for scanned docs, and organized file listings: https://github.com/rhowardstone/Epstein-research-data

Search interface: https://epstein.lasearch.app

Disputes about whether web scraping is legal have been going on for a long time. And now, a couple of months ago, the scandalous case of web scraping between hiQ v. LinkedIn was completed.

You can read about the progress of the case here: US court fully legalized website scraping and technically prohibited it.

Finally, the court concludes: "Giving companies like LinkedIn the freedom to decide who can collect and use data – data that companies do not own, that is publicly available to everyone, and that these companies themselves collect and use – creates a risk of information monopolies that will violate the public interest”.

I scraped metadata for 2.6M pornhub videos based on the 320k most recently updated playlists as of mid-July 2019. In total, the data is 350MB in compressed json form, separated into playlist, video, and matching/cross-referencing files. They're directly downloadable from these links:

https://datahub.io/racydata/final/r/dfp.json.gz (14MB)

https://datahub.io/racydata/final/r/dfv.json.gz (120MB)

https://datahub.io/racydata/final/r/dfm.json.gz (210MB)

And here's an example jupyter notebook that uses the matching data (40 million pairs of (videoid,playlistid)) to make a sparse matrix with dimensions of (number of unique videos x number of playlists) and reduce the dimensionality with SVD. Then you can get "recommendations" for playlists/videos similar to a particular playlist/video based on the distance in this reduced dimensional space.

https://gist.github.com/racydata/92ae85ea47da7c2d0bf50442bb0e83ea

The notebook also shows what columns you can play with in those three files.

Kaggle dataset page

Scope of Data

• All electronically-available Texas Appeals Court cases filed since 1900 (as of 2021-08-01).

• Courts included: Texas Supreme Court, Court of Criminal Appeals, 14 Appeals Courts (regional)

• Total cases: 696,036.

• Full dataset approx 12GB.

• Sample dataset approx 10MB.

• Download 500 sample rows or full dataset -- in either SQL or JSON format.

• Data source: https://search.txcourts.gov/CaseSearch.aspx?coa=cossup

• Aggregated by: https://www.judyrecords.com

• Parsed fields include caseId, createdAt (Unix timestamp), siteCaseNum, courtKey, and httpReq.

• Detailed case information is not parsed within each case currently, but can be parsed from standardized HTML structure.

• Sample case from data source: https://search.txcourts.gov/Case.aspx?cn=01-20-00103-CV&coa=coa01

• Court structure of Texas: https://www.txcourts.gov/media/1452084/court-structure-chart-february-2021.pdf

Case Count by Court

https://gofile.io/d/YQ2g4j (1.8MB)

Attrs: ["id","date","tweet", "insult_x","target_x",..]

Extracted from the NYT story: https://www.nytimes.com/interactive/2021/01/19/upshot/trump-complete-insult-list.html

UPDATE

• Dataset is now one unique tweet per row.

• insult_x and target_x exploded into separate columns (sparse matrix).

• Ordered by date ASC.

• Other minor issues fixed (random newlines, other dupes).

• Internal quotes are ""double quoted"".

https://github.com/jihoo-kim/Coronavirus-Dataset/

If you want those merged in the same schema with Singapore and Hong Kong, we did that on DoltHub:

https://www.dolthub.com/repositories/Liquidata/corona-virus/data/master/case_details

That has 7658 cases currently tracked. Dolt data sync with upstreams hourly.

The Big Porn Dataset is the largest and most comprehensive collection of adult content available on the web. With an amount of 23.686.411 Video URLs it exceeds possibly every other Porn Dataset.

I got quite a lot of feedback. I've removed unnecessary tags (some I couldn't include due to the size of the dataset) and added others.

Use Cases

Since many people said my previous dataset was a "useless dataset", I will include Use Cases for each column.

• Website - Analyze what website has the most videos, analyze trends based on the website.

• URL - Webscrape the URLs to obtain metadata from the models or scrape comments ("https://pornhub.com/comment/show?id={video_id}}&limit=10&popular=1&what=video"). 😉

• Title - Train a LLM to generate your own titles. See below.

• Tags - Analyze the tags based on plattform, which ones appear the most, etc.

• Upload Date - Analyze preferences based on upload date.

• Video ID - Useful for webscraping comments, etc.

Large Language Model

I have trained a Large Language Model on all English titles. I won't publish it, but I'll show you examples of what you can do with The Big Porn Dataset.

Generated titles:

• F...ing My Stepmom While She Talks Dirty

• Ho.ny Latina Slu..y Girl Wants Ha..core An.l S.x

• Solo teen p...y play

• B.g t.t teen gets f....d hard

• S.xy E..ny Girlfriend

(I censored them because... no.)

Note: This dataset contains sensitive content and is intended solely for research and educational purposes. 😉 Please ensure compliance with all relevant regulations and guidelines when using this data. Use responsibly. 😊

More information on Huggingface and Twitter:

https://huggingface.co/datasets/Nikity/Big-Porn

https://x.com/itsnikity