r/learnpython • • Jan 14 '20

I've made a python program that let's you mass download images from a given subreddit!

The program/script can be found here.

Any advice would be appreciated, thanks in advance!

This is my first code over 100 lines, so it may be messy, sorry for that.

------------------------------

How to use?

Go to the link given above, and download the program. (The first page given, there's no releases or anything)

Once you got it downloaded unzip it somewhere, and open usersettings.py file with any text editor, then fill out the information.

Once that is out of the way open cmd, and download PRAW library. (pip install PRAW), then head over to the directory where sid.py is located, and type in sid.py subreddit_name sorting (Sorting can be top/hot/new/rising).

------------------------------

377 Upvotes

95 comments sorted by

120

u/Dr_Thrax_Still_Does Jan 14 '20

I wonder which subs this will be used on most often.......

60

u/glof2 Jan 14 '20

Yeahh, I wonder...

62

u/bisantium Jan 14 '20

"research"

0

u/[deleted] Jan 15 '20

[deleted]

11

u/warriordeb96 Jan 15 '20

Brought to you by Apple Inc.

3

u/hellynx Jan 15 '20

Someone there will make it 250% more efficient. May have to hold device a particular way for it to connect to desired subreddit, otherwise its your fault for holding it wrong

4

u/PianoNyan Jan 15 '20

<Jony Ive Narrating> "We designed the iKeepfap taking cues from the natural grayness... of the moon"

7

u/cjcrisos Jan 14 '20

What's the answer to this?

28

u/Revolio_ClockbergJr Jan 15 '20

The porn ones

24

u/EddyBot Der-Eddy Jan 15 '20 edited Jan 15 '20

17

u/camelglitch Jan 15 '20

Dont forget r/spaceporn

13

u/[deleted] Jan 15 '20

I will be using it on r/foodporn

2

u/sassysalmnder Jan 15 '20

Yeah porn definitely

3

u/scanguy25 Jan 15 '20

Does on work on r/gone wild Asking for a friend

1

u/FartingBob Jan 15 '20

I'm going to use it for /r/capybara because I think they are really neat.

1

u/ChokesOnYou Jan 15 '20

Aren't you happy you're finally capybara of doing this?

-7

u/Konkavstylisten Jan 15 '20

Arent the Pr0n banned on Reddit?

5

u/marcx88 Jan 15 '20

You’re thinking of tumblr.

-3

u/Konkavstylisten Jan 15 '20

Oops xD.

-3

u/nkn_ Jan 15 '20

??????????????????????????????????????????????????????????????????????????????????

24

u/xelf Elf Jan 14 '20

Can't look at the code from work, will check it out later.

Did you account for duplicate images? What about duplicate names for differing images? Or vice versa?

One thing that might be useful (if you didn't already do it) is finding a way to capture useful metadata like the poster or vote count, perhaps in the name of the file.

19

u/glof2 Jan 14 '20 edited Jan 14 '20

Didn't check for duplicates, the images are named by the program like this: image{i}.png, with i being incremented every image, so the names can't repeat.

12

u/xelf Elf Jan 14 '20

Sounds pretty cool. Looking forward to seeing it when I get home! Another possible improvement is capturing the subreddit name, either in a a folder you're putting all the images into, or as part of a hierarchical name construct.

5

u/glof2 Jan 14 '20

Oh, that's a good idea, thanks! Also how can I look for duplicates in these images? Any idea? Thank you!

5

u/xelf Elf Jan 14 '20

If you wanted to be hardcore you could compare the images, but you might find it easier just to compare the URL the image is stored at?

Oh another thought: being able to save the photos with a reverse timestamp might be nice too.

image.spacepics.2020.01.14.11.03_1.png  
image.spacepics.2020.01.14.11.03_2.png

2

u/glof2 Jan 14 '20

Well that would be some protection, but what if the image is stored at different urls? Thank you!

19

u/[deleted] Jan 14 '20

Compare their md5 ? :)

I did a mass downloader too once and I avoided duplicates like that.

8

u/lordfwahfnah Jan 15 '20

Just doesn't work of the image is recropped or compressed with jpeg....AGAIN

3

u/Flobaer Jan 15 '20

In that case they are arguably not the same image

3

u/lordfwahfnah Jan 15 '20

Don't know about you, but I don't want the same picture multiple times with decreasing quality

3

u/xelf Elf Jan 14 '20

There's only so much you can do. =) (within reason)

4

u/I_Like_Smarties_2 Jan 15 '20

Try pillow or opencv libraries

3

u/pres82 Jan 14 '20

MD5 each image and then do a lookup to check for duplicates after download.

3

u/[deleted] Jan 15 '20

There are tons of ways to do analysis to determine if the image is a duplicate. The immediate one that comes to mind is using a computer vision package such as OpenCV to analyze the pixels in the image. Though because of compression/lossy image formats I'm not positive this would work with certain types of images such as .jpg

A quick Google search leads to this response where someone handles it with searching for duplicate filenames/file sizes. If the file size is the same then they hash the image and compare hashes (I assume - didnt read the code).

2

u/j03 Jan 15 '20

There are measures like SSIM (https://en.wikipedia.org/wiki/Structural_similarity) which can be used to compare two images while not requiring pixel-perfect matches. SSIM between pairs of images (resized to be the same dimensions) with a threshold above which to discard would probably get you a long way.

26

u/[deleted] Jan 14 '20

Very nice! Good program. I'd add a few suggestions:

  1. Use requirements.txt to allow anyone to install the correct version of your python packages (e.g. praw). Google "requirements.txt python" for instructions.

  2. Use "if name main" and a shebang.

  3. Use argparse for more robust command line argument options.

  4. Try adding some Error handling! What if requests returns a 404 response? Boom!

  5. Avoid using the global keyword unnecessarily. Rather, pass things into your functions as arguments! It makes those functions more modular (e.g. reusable)

6

u/glof2 Jan 14 '20

Will do! Thank you!

8

u/Vitaman02 Jan 14 '20 edited Jan 15 '20

You could also install packages within your script, so the user doesn't need to install packages from the cmd

try:
    #  imports
except ImportError:
    import subprocess    
    import sys

    subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'PRAW'])    
finally:
    #  imports

7

u/Username_RANDINT def get_random_int(): return "RANDINT" # TODO Jan 15 '20

Don't do this! This should be done by the user, not automatically. What if the user likes to pass the --user (or other) flags? I should check, but it's possible this'll fail on Linux as it'll try to install in the system packages but doesn't have permissions.

3

u/j03 Jan 15 '20 edited Jan 15 '20

It would be much better to package your project using setuptools, and specify install_requires=['PRAW>=6.5.1'] to enforce presence of packages (along with minimum versions) rather than shelling off to pip this way.

2

u/[deleted] Jan 15 '20

No this is not a good idea

1

u/baubleglue Jan 15 '20

could also install packages within your script,

WTF? Never do that. Never decide for the user what he/she need.

0

u/[deleted] Jan 15 '20

[deleted]

1

u/baubleglue Jan 15 '20

If you do it for friends and they know about it, than it's ok, but if you share code with unknown people it isn't. You can also provide setup script, that will be more appropriate option for all cases.

1

u/Vitaman02 Jan 15 '20

I'll look into setup scripts then, thanks for the advice :)

0

u/[deleted] Jan 14 '20

[deleted]

6

u/glof2 Jan 14 '20

You should use code block, in comments It's under the three dots. Thank you!

0

u/[deleted] Jan 14 '20

[deleted]

5

u/callmelucky py34int Jan 15 '20

For a code block, put an extra 4 spaces at the start of every line.

That
Gets
You
This look

For inline code like this, surround the text with backticks `like this`

0

u/[deleted] Jan 15 '20

[deleted]

3

u/callmelucky py34int Jan 15 '20

No worries. I noticed you had edited a code block in once is written most of that comment, but figured you might have done that one a PC and not known how to do it manually. Decided to go ahead and post anyway in case it's useful to any passersby :)

2

u/DanelRahmani Jan 15 '20

I saw a :( so heres an :) hope your day is good

0

u/[deleted] Jan 15 '20

[deleted]

1

u/baubleglue Jan 15 '20

yes, add mode dependencies to your code

3

u/strkr101 Jan 14 '20

I've been working on a similar project, and I've run into issues trying to download anything that's not an image. Is your program able to download gifs?

2

u/glof2 Jan 14 '20

Unfortunately not. Although it will be added in the future. First you should make a different list for gifs (unless you have another idea). If you've done it the same way I did it, the link should have a .mp4 extension (first gif from r/gifs), so if ".mp4" in link you should append the link to gif list, and to make download it as .mp4 you can do as this guy with indian accent shows in this video. I'm pretty sure you can just change the extension, but can't confirm.

I also can't confirm any of this will work, but I see no reason it wouldn't. I hope I helped, just play around with it!

3

u/[deleted] Jan 15 '20 edited Jan 15 '20

You can download gifs with the requests module once you've found the source url to the gif.

e.i.

import requests

url = 'https://i.imgur.com/zwTFd4d.gif'
gif = requests.get(url).content
with open('harold.gif', 'wb') as f:
    f.write(gif)

2

u/ICDCFlex Jan 15 '20

what do you put for the about url and redirect uri?

1

u/glof2 Jan 15 '20

About url doesn't matter, you can put http://localhost:8080/ in redirect uri

2

u/ICDCFlex Jan 15 '20

Ok, but I’m only getting the secret key what do I put for the other one?

1

u/glof2 Jan 15 '20

You are getting it, It's above the secret key, small text. You may need to click "edit app" to see it now

21

u/Eelz_ Jan 15 '20

Awesome work here! I’ve done a little code review for you with some things I haven’t seen others say:

  • Add your instructions you have to the README for anyone who might stumble upon your code

  • Since you’re using f-strings, mention that Python 3.6 or higher is required

  • says.argv on its own is a useless call

  • Line 83 isn’t quite doing what you think, it’s a pretty common mistake. You essentially have if “.jpg” which is always true on its own so the else will never run. You really want if “.jpg” in data or “.png” in data.

  • Line 106 only works as you intend for Windows. If you use pathlib you can get better multi OS support

  • Line 91 maybe put the entire for loop in this context manager? You’re opening and closing the same file repeatedly when you only need to once

9

u/[deleted] Jan 15 '20

Oh boy, bot devs are going to LOVE this...

6

u/forrealbro Jan 15 '20

It’s weird when I’m scratching out notes to do this same project and then I see it posted here. ‘Tis life

2

u/[deleted] Jan 15 '20

Just do it anyway. Make it better in some way

2

u/socialistvegan Jan 15 '20 edited Jan 15 '20

Just a heads up, it looks like it's saving each image file as simply being named "images\image1.png" etc in the root folder, leaving the actual images folder empty.

Changing

filename = f"images\\image{i}.png"

to

filename = f"images/image{i}.png"

seemed to fix it.

1

u/glof2 Jan 15 '20

Oh, on my pc it works as intended

3

u/Username_RANDINT def get_random_int(): return "RANDINT" # TODO Jan 15 '20

Use os.path.join to create cross-platform paths.

2

u/callmelucky py34int Jan 15 '20

This is the way.

2

u/socialistvegan Jan 15 '20

Here's a screenshot to display the initial behavior, on a Mac: https://imgur.com/a/AoogitR

4

u/KenReid Jan 15 '20

Added a pull request with your instructions from here - will hopefully make it easier to use for non-programmers.

4

u/Es-Click Jan 15 '20

Gone wild for sure

3

u/dtaivp Jan 15 '20

u/glof2 you should crosspost this over in r/datahoarder they will love it

2

u/AussieMazza Jan 15 '20

Cool! This was something I was intending to write once I became more proficient with Python. I'll still do it, but it's good to have working code to compare against. :)

2

u/jess_611 Jan 15 '20

Once I get into python I want to build something similar for Facebook. I have soo many images from over the years that are “trapped” on the platform. It’s one of the only reasons I haven’t completely deleted my account, just deactivated for over a year.

1

u/NateSnakeSolidDrake Jan 15 '20

There might be a feature to download all of your profile data for offline use

1

u/jess_611 Jan 15 '20

There is, however it compresses all of your images. I’d like to preserve quality if at all possible.

2

u/Tr1pt1 Jan 15 '20

Awesome project! I made something really similar a few days ago, if you want to check it out: https://github.com/andersmatre/RedditSaver

1

u/Ashwayne46 Jan 18 '20

The file came as .exe so it works only on windows?

2

u/[deleted] Jan 15 '20

[removed] — view removed comment

1

u/Tr1pt1 Jan 15 '20

fixed it, should work now, my bad!

2

u/newnewbusi Jan 15 '20

This could possibly be a good way to gather data for machine learning when it comes to identifying things.

2

u/claymore666 Jan 15 '20

Can it be m@de to do webms?

2

u/thespacetimelord Jan 15 '20

I get this error:

Wrong format. The format should be:'sid.py subreddit_name sort_by'.

When I try sid.py earthporn hot

2

u/AaronRoh Jan 15 '20

Thanks.

Can it run by python any version 3.x ?

2

u/blabbities Redux FTW Jan 15 '20

I was gonna make one...but. I need one for uh....research on particularly profiles....and downloading all those images. If that's on the roadmap lemme know

2

u/impshum x != y % z Jan 15 '20

Wrote ages ago but... multithreaded function FTW: https://github.com/impshum/reddit-image-downloader

1

u/Ashwayne46 Jan 18 '20

I ran the run.py which was throwing 401 error with message RID no worky, A how to use reply would be informational. Thanks.

1

u/impshum x != y % z Jan 21 '20

It's just the rid() function at the moment.

Rurk larder!

2

u/buleria Jan 16 '20

Just a few quick notes after a quick glance through the code, hope you find them useful:

  • Consider using argparse instead of directly playing around with sys.argv.

  • Don't use global, instead return the reddit instance from prawsetup() and pass it to getimageurl()

  • also pass the subreddit name extracted from arguments. This makes the function not rely on sys.argv

  • I think requests has a json() function on its Response objects, no need to use response.text

  • if data.lower().endswith('.jpg') is slightly more bulletproof; maybe make a function that tells if a link is supported?

  • Again, you're using a global links variable instead of simply returning it from getimageurl() and passing on to downloadimages(). Think of your program as a set of steps that produce a set of values (like parsed commandline arguments, praw instance, list of urls etc) and pass it to the next step in your processing pipeline.

1

u/baubleglue Jan 15 '20

???

# Using sys argv to get subreddit name
sys.argv

There are at least two/three reasons not to do below code in the way it is implemented, try to guess what are they.

# Making sure imgs folder is here
try:
    os.mkdir("images")
except FileExistsError:
    pass

discover argparse

# Checking if user provided a subreddit name
if len(sys.argv) != 3:
    print("Wrong format. The format should be:" +
          "'sid.py subreddit_name sort_by'.")
    quit()

---

Use if __name__ == "__main__":, so somebody may import your library without executing the code. For the same reason provide an option to setup account info without using configuration file

# Main code
prawsetup()
getimageurl()
downloadimages()

---

What if user has Mac or Linux? All path management in the code is sucks, use os.path calculate/ask user absolute path for output, never write to relative paths in the code.

filename = f"images\\image{i}.png" 

---

You don't nned write it to a file, but if you need use os.tmpfile

  with open(f"links{time_}.txt", "a") as f: 

About getimageurl(). don't use sys.argv in differnt places (collect all the parameters at start to some settings object and use it). Don't do action in action when it possible.

  1. Collect all the links

  2. for link in links: save_file(link)

1

u/o5a Jan 17 '20 edited Jan 17 '20

You can use async requests to significantly speed up mass image downloading by switching to grequests instead of requests. Here's it's usage example:

import grequests

# your url list
links = [ ... ]

# exception handler for bad requests
def exception_handler(request, exception):
    print("Request failed", request.url)

#creating requests list (they are not sent yet)
rs = [grequests.get(u) for u in links]

# this will send them at the same time in a pack of size
# and we can process each response when it finishes
for r in grequests.imap(rs, size=16, exception_handler=exception_handler):
    print(r.status_code, r.url)
    # get file name from url
    filename = r.url[r.url.rfind('/')+1:]
    # save file
    with open(filename, "wb") as f:
        f.write(r.content)

1

u/JingzOoi Jan 18 '20

Hey, I'm a bit late, but if you want to, the whole part with

try:
    os.mkdir("images")
except FileExistsError:
    pass

can be replaced with

os.makedirs("images", exist_ok=True)

if a folder of the same name exists, nothing will be changed. Essentially the same thing as yours.

0

u/[deleted] Jan 15 '20

[deleted]

4

u/Voidskiz Jan 15 '20

It's good to come up with your own solutions, especially when still learning!