r/learnpython • u/bextie • Nov 27 '19
Black Friday hack with python
So I just finished my python script to monitor the price of my target gadget this holiday. It’s going to be heavily discounted soon but it will go to the fastest finger.
I’ve written a python script with BeautifulSoup. I’ve also set email and SMS alerts in the script 😂😂
Was just wondering if there’s any other person doing this for Black Friday and Fastest Finger deals.
Update: Long overdue but just thought I should add that I got the phone.
26
u/expressly_ephemeral Nov 27 '19
Hey, why don’t you refactor into selenium and beautiful soup, teach your script your credit card info and let it buy it for you when it detects the drop?
20
u/BruceJi 📦 Nov 27 '19
I wonder if that's not the best idea in case something goes wrong? But if you get the email, you can probably just buy it online with your phone within a few minutes anyway.
16
u/azur08 Nov 27 '19 edited Nov 27 '19
in case something goes wrong
Like buying everything Amazon has to offer over and over again!
31
9
u/BruceJi 📦 Nov 27 '19
"Merry Christmas, here are 20 pop-up tents, for all that... popping up?... you do? Eheh..."
2
u/expressly_ephemeral Nov 27 '19
Hah! I know a lady who gave her grandkid the tablet to play with and he ordered 20 chromebooks. Kids and scripts, man. Fuckin' dangerous.
10
u/expressly_ephemeral Nov 27 '19
I acknowledge... tricky to test, tricky to validate. Cooler, though.
3
u/BruceJi 📦 Nov 27 '19
It really would be!
3
u/expressly_ephemeral Nov 27 '19
I guess I think you could use the email as a backstop, though. You get the email saying the price happened and then a follow up saying “I bought it” or “your code stinks, get out your credit card.”
I mean.. what if OP is in meeting when the trigger fires? What if his wife’s water has just broken while she’s driving her real estate clients around? He won’t be able close the deal!
1
u/dBASSa Nov 27 '19
Theres tons of sneaker bots that do this already. They are heavily linked with fraud and abuse so limited sale items block orders this way often times.
1
u/expressly_ephemeral Nov 27 '19
How would a vendor be able to block a bespoke script running from a host on OPs personal network?
0
Nov 27 '19
[deleted]
1
u/expressly_ephemeral Nov 27 '19
OP should know better than to put his CC info into his github repo, right? Maybe I’m being naive?
2
u/bextie Nov 27 '19
None of my sensitive details are there.
1
u/MonkeyNin Nov 27 '19
You can simplify your exceptions, because a failed find call will return
Nonewe can take sourceclass Foo(): def price(self): try: productPrice = self.soup.find(id="priceblock_ourprice").get_text() return productPrice except AttributeError: try: productPrice = self.soup.find(id="priceblock_dealprice").get_text() return productPrice except AttributeError: passWe can test for
Nonewhen there are no matchesclass Foo(): def price(self): element_price_price = self.soup.find(id="priceblock_ourprice") if not element_price_price: print('Price not found!') return 0 # or `None` depending on your context return product_price.get_text()0
Nov 27 '19
[deleted]
1
u/bextie Nov 27 '19
Oh I haven’t thought about that. Is it worth it? I feel that’s for ‘bigger’ projects.
0
Nov 27 '19
[deleted]
1
u/bextie Nov 27 '19
Thank you! I’m definitely going to do that.
That’s a nice project you’ve got by the way.
0
Nov 27 '19
[deleted]
1
u/expressly_ephemeral Nov 27 '19
Sure, but then you’d have to be canny enough to make the following realizations: I shouldn’t share this with anybody. I should figure a way to make sure the eventual connection is ssl (via selenium). This should be hosted on my private network only unless there’s a way to make this kind of thing reasonably secure. And, finally, there’s probably a way to make this kind of thing secure and I should figure out how people publish code like this.
I have my personal login credentials hardcoded into some scripts, but the run solely on a machine that lives in a closet in my house, and I make sure not to push them to a public repo.
0
Nov 27 '19
[deleted]
1
u/expressly_ephemeral Nov 27 '19 edited Nov 27 '19
You know, I’ve done the threat modeling. If somebody breaks my OpenVPN instance and compromises my router, then they ssh into the server that runs these scripts (a server which only responds to logins with my private rsa token), or worse finds an exploit by which they can get a reverse shell... then I’ve got much bigger worries than whether they can harvest the username password I use to check my kid’s missing homework list.
Edit: I’m saying, you’re right. I wouldn’t do it with my login credentials for my bank,
0
Nov 27 '19
[deleted]
2
u/expressly_ephemeral Nov 27 '19
There's a fine line between good threat modeling and gross negligence!
2
u/expressly_ephemeral Nov 27 '19
VPN, i didn't think of that.
Yeah, I mean, again, in this use case all the application has to do is make GET and POST calls. Outbound only. Server wouldn't be visible from the outside world even WITHOUT the vpn.
4
2
u/PM_ME_BOOTY_PICS_ Nov 27 '19
I made a web scraper so I could get the item specs to use for my reports at work!
1
u/bextie Nov 27 '19
That sounds cool! Beautiful to see how coding makes life easier.
1
u/PM_ME_BOOTY_PICS_ Nov 27 '19
It really does but I have some bugs to squash. Some reason certain specs for certain items aren't being pulled so I have to go trouble shoot that.
1
93
u/Den4200 Nov 27 '19
I made a script that is very similar to yours. It tracks prices of items on Amazon and saves a graph of each item’s prices in a png. It’ll send you an email and/or text when an item goes below your target price.
Check it out! This is going to be so useful with black friday and cyber monday coming up: https://www.github.com/Den4200/amazon-price-tracker
35
u/corbin_cassady Nov 27 '19
Sounds like a self-hosted camelcamcelcamel! I like the premise a lot.
3
u/trowawayatwork Nov 27 '19
yes its a great start. would be great to add push notifications to it rather than email as its quicker feedback on your phone. i have set up something similar to track and then send a push notification ot a telegram group
a simpler thing would be to just monitor camelizer, as theres no point in reinventing the wheel (although good practice for beginners), and then send that on to your medium of choice
2
u/Den4200 Nov 27 '19
This supports sending text messages to your phone!
Hmm I didn’t think of scraping camelcamelcamel instead of Amazon, but what’s done is done.
15
u/domac Nov 27 '19
It's often super helpful to add Screenshots to the repo. just wanted to point that out.
thanks for the awesome work!
3
u/flying_mechanic Nov 27 '19
The keepa extension for chromium browsers does the same but in page, it's a nice addition. Might be worth checking out
2
Nov 27 '19
I have made the same program but in a much lighter format. Lovely to see another approach.
1
u/wrgsRay wrgsray Nov 27 '19
How do you get around Amazon’s protection against scraping? For a newbie like me, it’s painful to scrape Amazon
1
u/Den4200 Nov 27 '19
They don't really have much protection against scraping. As long as your scraper doesn't check for the price too quickly, you're fine (about 15 minutes between each check is good).
Amazon won't let you scrape without any headers btw.
Also, this little trick helps out. Look at line 28 and 29 intracker.py1
u/TeblowTime Nov 28 '19
Hey, I keep hitting the Invalid URL when I try to add a new link. I successfully installed all the required libraries from requirements.txt, so it's not that. Any ideas? I'm a beginner with python, so it's likely something I am or am not doing. Do the URLs need to be in specific formats?
1
u/Den4200 Nov 28 '19
Did you fill in the settings?
And nope, the script will auto shorten the links.
1
u/TeblowTime Nov 28 '19
Welp, I'm an idiot lol. I did fill in the settings info, but did not save the file. Thanks!
0
Nov 27 '19
[deleted]
2
u/Den4200 Nov 27 '19
Yeah haha. I usually leave bigger projects with GPL and smaller ones with MIT. Nice project btw!
8
Nov 27 '19
I wrote something that checked for Buy Now button on amazon when the Nintendo Switch first came out and was always out of stock.
Week after it released my script paid off
8
u/yodatrust Nov 27 '19
How do you make your script running 24/7?
7
u/B166er_ Nov 27 '19
Run the script and don’t turn off your computer... Or use a service like heroku/pythonanywhere
6
u/Uadsmnckrljvikm Nov 27 '19
Run it on a server, could be for example a Raspberry pi or one from AWS.
1
u/yodatrust Nov 27 '19
The Raspberry Pi solution sounds interesting. Got any useful docs/links for that?
2
u/artificial_neuron Nov 27 '19 edited Nov 27 '19
You put everything in a loop that never ends. You don't need a special piece of hardware, just something that will run the script and you'll never turn off.
Eg.
while(true) Do the desired activity wait for x time
7
u/RedBlaze4 Nov 27 '19
How do you send the SMS ?
2
Nov 27 '19
My provider (France) offers URL format API for free to use with my account. A bot has to connect to a URL with my logins and a message. And that is it
1
u/RedBlaze4 Nov 27 '19
Tu es chez quel opérateur ?
2
Nov 27 '19
"Free".
go to your options and enable SMS notifications. Then, call this url
https://smsapi.free-mobile.fr/sendmsg?user=99999999&pass=cle_identification&msg=Hello%20World%20!
Reponses are :
200 : SMS sent.
400 : missing required args.
402 : too many SMSs in a short time.
403 : Service unenabled or incorrect user/pass.
500 : Server error. Retry later.
0
7
Nov 27 '19
I considered doing something similar for a website I like but then realized the website had a function all ready to do what I wanted. I'm not advanced or clever enough to do something like that yet.
1
u/BakingSota Nov 27 '19 edited Nov 28 '19
You’re definitely clever enough.
Check out these libraries:
re - for regex magic
requests - for html magic
And this chapter in AtBS: https://automatetheboringstuff.com/chapter11/
1
u/demesm Nov 27 '19
Is it possible to have this go through the entire purchase process automatically as well? Always wondered if you could snipe v deals like that
1
u/bextie Nov 27 '19
It is. Someone mentioned using selenium to control the browser to automatically purchase the item but of course, you have to be more careful with that so you don’t get into an infinite loop of buying stuff.
1
u/artificial_neuron Nov 27 '19
You can using Selenium. It allows you to use a website as if you're a user clicking on things - it makes life easier when JS is involved.
I would have to watch it perform the purchase 10s of times before i would trust it to do things by itself without supervision. Imagine how you would feel if it emptied your bank account and bought 100s of the same item.
1
u/BaboJango Nov 27 '19
Lol “hack” but I tried this myself and the email function failed because I hadn’t set up “less secure access” setting turned off. It then worked once I turned it on. Hopefully this helps someone figure it out faster than it took me!
1
u/bextie Nov 27 '19
I won’t advise that you use that method. You can use the other method I mentioned (activating the 2FA and generating an app password). I recommend it because it’s very secure.
But yeah, both methods work fine for this purpose.
1
u/BaboJango Nov 27 '19
Is there extra code that you would need to create to enter in the app password? What does that look like?
1
u/bextie Nov 27 '19
No extra code. You need to generate it from your google account. Search ‘app password ‘ on google.
1
1
u/FirmPiezoelectricity Nov 28 '19
I was just messing around with this and noticed that the price reduced down to what you wanted it at. Hope you get it! Thanks for sharing the program. I am trying to learn Beautiful Soup.
1
2
1
u/TotesMessenger Nov 27 '19
1
1
u/SrHombrerobalo Nov 27 '19
I haven’t read your code but as a safety measure, try to have any password or personal data on a .ini external file
1
Nov 27 '19
No, but only because I can't figure out how to build a script that eliminates Black Friday from the fabric of society.
1
u/LeSplooch Nov 27 '19
I know the subject of this subreddit is Python but if you want to be the first to get your deal, then some other dev using another language (such as Rust) might be faster to get his notification. Be tactical, haha!
1
2
Nov 27 '19
[removed] — view removed comment
15
Nov 27 '19
They’re not the same. Beautiful soup is a parser, extracting data from html. Selenium automates a browser to perform actions.
1
u/pulsarrex Nov 27 '19
Can you eli5 what a 'parser' is?
3
u/Conrad_noble Noble1988 Nov 27 '19
I think it means it translates html into user friendly readable text
1
u/OMGpancakeable Nov 27 '19
Not quite “translate”. Parsing: resolve (a sentence) into its component parts and describe their syntactic roles.
1
u/rrjamal Nov 27 '19
It splits data in a way you want. An HTML doc can be a shit ton of words. You'd parse it down to split and retrieve only the bits you care about
3
Nov 27 '19
[removed] — view removed comment
8
u/TheAmazingJames Nov 27 '19
Selenium is way way slower and memory heavy. Selenium should only be used if you need JavaScript to render/work.
2
u/Mathew_Berrys_Cock Nov 27 '19
Would more ram help this problem?
2
u/TheAmazingJames Nov 27 '19
Selenium’s usually using chromium in the background so it’s unlikely to max out your RAM unless you’re trying to run on a barebones setup, but it will be using significantly more than a requests/beautifulsoup setup and that could be costly if you’re running it in the cloud. The main reason not to though is speed. Typically I’d expect requests to run 5-10x faster than selenium. Selenium also requires more maintenance in terms of chromedriver updates and is harder to deploy on others’ systems. I’m not saying Selenium isn’t awesome, but it’s for specific use cases where you’re either browser testing or need JavaScript.
7
u/adriangc Nov 27 '19
I’m more curious to understand why people choose BS over Scrapy.
2
2
u/GrandRub Nov 27 '19
i am a python newbie and found scrapy yesterday and i ask myself the same question.. whats the pros and cons of scrapy?
1
May 22 '20
[removed] — view removed comment
1
u/AutoModerator May 22 '20
Your comment in /r/learnpython was automatically removed because you used a URL shortener.
URL shorteners are not permitted in /r/learnpython as they impair our ability to enforce link blacklists.
Please re-post your comment using direct, full-length URL's only.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
63
u/17_jku Nov 27 '19
I have not (just starting to tinker with Python...I'm a VB, SQL, SAS developer), but would love to see your approach. Of course, you would need to change the item URL so none of us snatch it up before you. Hahaha.