r/learnpython • u/glof2 • Jan 14 '20
I've made a python program that let's you mass download images from a given subreddit!
The program/script can be found here.
Any advice would be appreciated, thanks in advance!
This is my first code over 100 lines, so it may be messy, sorry for that.
------------------------------
How to use?
Go to the link given above, and download the program. (The first page given, there's no releases or anything)
Once you got it downloaded unzip it somewhere, and open usersettings.py file with any text editor, then fill out the information.
Once that is out of the way open cmd, and download PRAW library. (pip install PRAW), then head over to the directory where sid.py is located, and type in sid.py subreddit_name sorting (Sorting can be top/hot/new/rising).
------------------------------
24
u/xelf Elf Jan 14 '20
Can't look at the code from work, will check it out later.
Did you account for duplicate images? What about duplicate names for differing images? Or vice versa?
One thing that might be useful (if you didn't already do it) is finding a way to capture useful metadata like the poster or vote count, perhaps in the name of the file.
19
u/glof2 Jan 14 '20 edited Jan 14 '20
Didn't check for duplicates, the images are named by the program like this:
image{i}.png, withibeing incremented every image, so the names can't repeat.12
u/xelf Elf Jan 14 '20
Sounds pretty cool. Looking forward to seeing it when I get home! Another possible improvement is capturing the subreddit name, either in a a folder you're putting all the images into, or as part of a hierarchical name construct.
5
u/glof2 Jan 14 '20
Oh, that's a good idea, thanks! Also how can I look for duplicates in these images? Any idea? Thank you!
5
u/xelf Elf Jan 14 '20
If you wanted to be hardcore you could compare the images, but you might find it easier just to compare the URL the image is stored at?
Oh another thought: being able to save the photos with a reverse timestamp might be nice too.
image.spacepics.2020.01.14.11.03_1.png image.spacepics.2020.01.14.11.03_2.png2
u/glof2 Jan 14 '20
Well that would be some protection, but what if the image is stored at different urls? Thank you!
19
Jan 14 '20
Compare their md5 ? :)
I did a mass downloader too once and I avoided duplicates like that.
8
u/lordfwahfnah Jan 15 '20
Just doesn't work of the image is recropped or compressed with jpeg....AGAIN
3
u/Flobaer Jan 15 '20
In that case they are arguably not the same image
3
u/lordfwahfnah Jan 15 '20
Don't know about you, but I don't want the same picture multiple times with decreasing quality
3
4
3
3
Jan 15 '20
There are tons of ways to do analysis to determine if the image is a duplicate. The immediate one that comes to mind is using a computer vision package such as OpenCV to analyze the pixels in the image. Though because of compression/lossy image formats I'm not positive this would work with certain types of images such as .jpg
A quick Google search leads to this response where someone handles it with searching for duplicate filenames/file sizes. If the file size is the same then they hash the image and compare hashes (I assume - didnt read the code).
2
u/j03 Jan 15 '20
There are measures like SSIM (https://en.wikipedia.org/wiki/Structural_similarity) which can be used to compare two images while not requiring pixel-perfect matches. SSIM between pairs of images (resized to be the same dimensions) with a threshold above which to discard would probably get you a long way.
26
Jan 14 '20
Very nice! Good program. I'd add a few suggestions:
Use requirements.txt to allow anyone to install the correct version of your python packages (e.g. praw). Google "requirements.txt python" for instructions.
Use "if name main" and a shebang.
Use argparse for more robust command line argument options.
Try adding some Error handling! What if requests returns a 404 response? Boom!
Avoid using the global keyword unnecessarily. Rather, pass things into your functions as arguments! It makes those functions more modular (e.g. reusable)
6
u/glof2 Jan 14 '20
Will do! Thank you!
8
u/Vitaman02 Jan 14 '20 edited Jan 15 '20
You could also install packages within your script, so the user doesn't need to install packages from the cmd
try: # imports except ImportError: import subprocess import sys subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'PRAW']) finally: # imports7
u/Username_RANDINT def get_random_int(): return "RANDINT" # TODO Jan 15 '20
Don't do this! This should be done by the user, not automatically. What if the user likes to pass the
--user(or other) flags? I should check, but it's possible this'll fail on Linux as it'll try to install in the system packages but doesn't have permissions.3
u/j03 Jan 15 '20 edited Jan 15 '20
It would be much better to package your project using setuptools, and specify
install_requires=['PRAW>=6.5.1']to enforce presence of packages (along with minimum versions) rather than shelling off to pip this way.2
1
u/baubleglue Jan 15 '20
could also install packages within your script,
WTF? Never do that. Never decide for the user what he/she need.
0
Jan 15 '20
[deleted]
1
1
u/baubleglue Jan 15 '20
If you do it for friends and they know about it, than it's ok, but if you share code with unknown people it isn't. You can also provide setup script, that will be more appropriate option for all cases.
1
0
Jan 14 '20
[deleted]
6
u/glof2 Jan 14 '20
You should use code block, in comments It's under the three dots. Thank you!
0
Jan 14 '20
[deleted]
5
u/callmelucky py34int Jan 15 '20
For a code block, put an extra 4 spaces at the start of every line.
That Gets You This lookFor inline code
like this, surround the text with backticks `like this`0
Jan 15 '20
[deleted]
3
u/callmelucky py34int Jan 15 '20
No worries. I noticed you had edited a code block in once is written most of that comment, but figured you might have done that one a PC and not known how to do it manually. Decided to go ahead and post anyway in case it's useful to any passersby :)
2
0
3
u/strkr101 Jan 14 '20
I've been working on a similar project, and I've run into issues trying to download anything that's not an image. Is your program able to download gifs?
2
u/glof2 Jan 14 '20
Unfortunately not. Although it will be added in the future. First you should make a different list for gifs (unless you have another idea). If you've done it the same way I did it, the link should have a .mp4 extension (first gif from r/gifs), so
if ".mp4" in linkyou should append the link to gif list, and to make download it as .mp4 you can do as this guy with indian accent shows in this video. I'm pretty sure you can just change the extension, but can't confirm.I also can't confirm any of this will work, but I see no reason it wouldn't. I hope I helped, just play around with it!
3
Jan 15 '20 edited Jan 15 '20
You can download gifs with the
requestsmodule once you've found the source url to the gif.e.i.
import requests url = 'https://i.imgur.com/zwTFd4d.gif' gif = requests.get(url).content with open('harold.gif', 'wb') as f: f.write(gif)
2
u/ICDCFlex Jan 15 '20
what do you put for the about url and redirect uri?
1
u/glof2 Jan 15 '20
About url doesn't matter, you can put http://localhost:8080/ in redirect uri
2
u/ICDCFlex Jan 15 '20
Ok, but I’m only getting the secret key what do I put for the other one?
1
u/glof2 Jan 15 '20
You are getting it, It's above the secret key, small text. You may need to click "edit app" to see it now
21
u/Eelz_ Jan 15 '20
Awesome work here! I’ve done a little code review for you with some things I haven’t seen others say:
Add your instructions you have to the README for anyone who might stumble upon your code
Since you’re using f-strings, mention that Python 3.6 or higher is required
says.argvon its own is a useless callLine 83 isn’t quite doing what you think, it’s a pretty common mistake. You essentially have
if “.jpg”which is always true on its own so the else will never run. You really wantif “.jpg” in data or “.png” in data.Line 106 only works as you intend for Windows. If you use
pathlibyou can get better multi OS supportLine 91 maybe put the entire for loop in this context manager? You’re opening and closing the same file repeatedly when you only need to once
9
6
u/forrealbro Jan 15 '20
It’s weird when I’m scratching out notes to do this same project and then I see it posted here. ‘Tis life
2
2
u/socialistvegan Jan 15 '20 edited Jan 15 '20
Just a heads up, it looks like it's saving each image file as simply being named "images\image1.png" etc in the root folder, leaving the actual images folder empty.
Changing
filename = f"images\\image{i}.png"to
filename = f"images/image{i}.png"seemed to fix it.
1
u/glof2 Jan 15 '20
Oh, on my pc it works as intended
3
u/Username_RANDINT def get_random_int(): return "RANDINT" # TODO Jan 15 '20
Use
os.path.jointo create cross-platform paths.2
2
u/socialistvegan Jan 15 '20
Here's a screenshot to display the initial behavior, on a Mac: https://imgur.com/a/AoogitR
4
u/KenReid Jan 15 '20
Added a pull request with your instructions from here - will hopefully make it easier to use for non-programmers.
4
3
2
u/AussieMazza Jan 15 '20
Cool! This was something I was intending to write once I became more proficient with Python. I'll still do it, but it's good to have working code to compare against. :)
2
u/jess_611 Jan 15 '20
Once I get into python I want to build something similar for Facebook. I have soo many images from over the years that are “trapped” on the platform. It’s one of the only reasons I haven’t completely deleted my account, just deactivated for over a year.
1
u/NateSnakeSolidDrake Jan 15 '20
There might be a feature to download all of your profile data for offline use
1
u/jess_611 Jan 15 '20
There is, however it compresses all of your images. I’d like to preserve quality if at all possible.
2
u/Tr1pt1 Jan 15 '20
Awesome project! I made something really similar a few days ago, if you want to check it out: https://github.com/andersmatre/RedditSaver
1
2
2
u/newnewbusi Jan 15 '20
This could possibly be a good way to gather data for machine learning when it comes to identifying things.
2
2
u/thespacetimelord Jan 15 '20
I get this error:
Wrong format. The format should be:'sid.py subreddit_name sort_by'.
When I try sid.py earthporn hot
2
2
u/blabbities Redux FTW Jan 15 '20
I was gonna make one...but. I need one for uh....research on particularly profiles....and downloading all those images. If that's on the roadmap lemme know
2
u/impshum x != y % z Jan 15 '20
Wrote ages ago but... multithreaded function FTW: https://github.com/impshum/reddit-image-downloader
1
u/Ashwayne46 Jan 18 '20
I ran the run.py which was throwing 401 error with message RID no worky, A how to use reply would be informational. Thanks.
1
2
u/buleria Jan 16 '20
Just a few quick notes after a quick glance through the code, hope you find them useful:
Consider using
argparseinstead of directly playing around withsys.argv.Don't use
global, instead return the reddit instance fromprawsetup()and pass it togetimageurl()also pass the subreddit name extracted from arguments. This makes the function not rely on
sys.argvI think
requestshas ajson()function on itsResponseobjects, no need to useresponse.textif data.lower().endswith('.jpg')is slightly more bulletproof; maybe make a function that tells if a link is supported?Again, you're using a global
linksvariable instead of simply returning it fromgetimageurl()and passing on todownloadimages(). Think of your program as a set of steps that produce a set of values (like parsed commandline arguments, praw instance, list of urls etc) and pass it to the next step in your processing pipeline.
1
u/baubleglue Jan 15 '20
???
# Using sys argv to get subreddit name
sys.argvThere are at least two/three reasons not to do below code in the way it is implemented, try to guess what are they.
# Making sure imgs folder is here
try:
os.mkdir("images")
except FileExistsError:
passdiscover argparse
# Checking if user provided a subreddit name
if len(sys.argv) != 3:
print("Wrong format. The format should be:" +
"'sid.py subreddit_name sort_by'.")
quit()---
Use if __name__ == "__main__":, so somebody may import your library without executing the code. For the same reason provide an option to setup account info without using configuration file
# Main code
prawsetup()
getimageurl()
downloadimages()---
What if user has Mac or Linux? All path management in the code is sucks, use os.path calculate/ask user absolute path for output, never write to relative paths in the code.
filename = f"images\\image{i}.png" ---
You don't nned write it to a file, but if you need use os.tmpfile
with open(f"links{time_}.txt", "a") as f: About getimageurl(). don't use sys.argv in differnt places (collect all the parameters at start to some settings object and use it). Don't do action in action when it possible.
Collect all the links
for link in links: save_file(link)
1
u/o5a Jan 17 '20 edited Jan 17 '20
You can use async requests to significantly speed up mass image downloading by switching to grequests instead of requests. Here's it's usage example:
import grequests
# your url list
links = [ ... ]
# exception handler for bad requests
def exception_handler(request, exception):
print("Request failed", request.url)
#creating requests list (they are not sent yet)
rs = [grequests.get(u) for u in links]
# this will send them at the same time in a pack of size
# and we can process each response when it finishes
for r in grequests.imap(rs, size=16, exception_handler=exception_handler):
print(r.status_code, r.url)
# get file name from url
filename = r.url[r.url.rfind('/')+1:]
# save file
with open(filename, "wb") as f:
f.write(r.content)1
u/JingzOoi Jan 18 '20
Hey, I'm a bit late, but if you want to, the whole part with
try:
os.mkdir("images")
except FileExistsError:
passcan be replaced with
os.makedirs("images", exist_ok=True)if a folder of the same name exists, nothing will be changed. Essentially the same thing as yours.
0
120
u/Dr_Thrax_Still_Does Jan 14 '20
I wonder which subs this will be used on most often.......