r/AssistantBOT • u/kungming2 Creator • Jun 25 '21
Announcement Artemis v2.3 Maple is now deployed! (with a replacement for Pushshift post aggregations)
This version has few obvious changes from the outside, but there's a replacement for Pushshift's aggregations - for posts, to allow Artemis to resume providing certain statistics about the posts and submitters of a subreddit.
More than six months ago, the Pushshift database disabled aggregations for everyone, which meant that some of the activity data that Artemis provided could not be provided. This data was:
Most Active Days for Submissions/Comments
Top Submitters/Commenters
At first, I (and other bot writers) figured this would be a temporary issue as the announcement post stated that "this won't be a permanent change," and that once aggregations went back up, everything would go back to normal. But of course, it's been a while now and aggregations show no sign of being re-enabled, and the "replacements" that have popped up don't seem reliable. Therefore, I decided to create my own local solution to replace Pushshift for* submissions* aggregations queries - Artemis Stream - something I outlined a few months ago as a solution.
Artemis Stream
Artemis Stream is a separate script that is run regularly and ingests/fetches all new posts from only the subreddits moderated by Artemis instances. A very small subset of those posts' data is cached in a local database that the statistics routine can query - in fact, Stream mimics Pushshift's reponse in providing that data back to the statistics function. Stream was actually deployed at the end of May and has been fetching posts for a few weeks now, and I upgraded Artemis's setup with an NVMe SSD to allow for faster reads/writes with Stream.
Stream allows me to restore half of the data previously provided by Pushshift aggregations:
Most Active Days for Submissions
Top Submitters
Artemis v2.3 Maple was deployed a few days ago, so you should have already seen this data re-appear in your June statistics! Stream also allows me to provide three new and simple, but useful information points for a subreddit's monthly statistics with v2.3:
Number/percentage of text posts
Number/percentage of NSFW posts
Number/percentage of spoilered posts
There are other data points that Stream caches and I may write more tools to surface them in the future.
Notes
Unfortunately, due to their sheer amount Reddit comments are outside the scope of the Stream project, and I won't be building a local replacement for comments aggregations.
Stream started tracking the number of text/NSFW posts in the middle of June, so those "total" numbers will be different this month from spoilered posts (and should not vary from next month onward).
Stream's cache won't be permanent (unlike Pushshift) due to storage and utility limitations; I plan to implement a 45-day buffer.
As always, if you are able and would like to contribute to an important resource for the web, please check out Pushshift's Patreon page.
1
u/YoloMice Oct 23 '21
AssistantBot is the top mod on my subreddit and is behaving badly. Can you remove it?
1
u/leneay Aug 16 '22
Hi, sorry to bother. I'm not sure if I did something wrong, but AssistantBOT hasn't updated the user flair statistics since July 1. Could you tell me how I might fix that? Thank you.
1
u/SolariaHues Jun 30 '22
Hi
AssistantBot hasn't updated the stats in r/gardenwild recently (not since the 5th of May) - is there something wrong, do I need to do anything?
Also, I might be premature reporting this, but we've just added AssistantBot1 to r/NewToReddit, and it's passed the time it said it would populate the wiki page, and nothing has happened. Any help very much appreciated!