r/pythontips • u/AceJohnson_ • Nov 08 '19
Best Py web scraping tutorial
I'm on a mission to hack a web scraper for my personal use. The tutorials I try to follow using a Jupyter Notebook have all led to a dead end. Could the community lead me toward a tutorial on a good webscraping tutorial. Looking for more mid level not beginner tutorial.
I'm an /r/bigseo person looking to scrape <p> tags of my website and competitors to gain some NLP insight into how we can improve our web content. Really looking for the best webscraPy to target <p> tags to NLP content for SEO.
4
u/116ill Nov 08 '19
Not an expert.. but I web scrape occasionally. Thus, JIC this helps check out these fairly popular web scraping tools/libraries:
Selenium
BeautifulSoup
Scrapy
2
u/sleepyleperchaun Nov 08 '19
I liked Corey Shafer. If you know nothing about python I'd definitely recommend learning the basics first cause it's not quite as simple as learning a single tool, you will need to know how to use those tools for each unique situation. But if you are decent at python already, then Corey Shafer is pretty good. Al Swigert anything is pretty dope too as I'm sure you've read around here (dude is a 'local' legend (I love you Al)).
2
u/APIglue Nov 08 '19
If you need to run JavaScript use selenium. Otherwise just use wget.
Beautiful soup is great for Parsing the downloaded files.
Throttle the downloading.
2
Nov 08 '19
Free - https://www.youtube.com/playlist?list=PL5tcWHG-UPH1fnJw-BvBiiiPUPm1LUKsm
Best - Web Scraping with Python: Collecting Data from the Modern Web https://www.amazon.com/dp/1491910291/
1
6
u/quadras_music Nov 08 '19
Use requests and beautifulsoup:
Import requests Import bs4
Req = requests.get(URL) Soup = Bs4.beautifulsoup(req.content, ‘xlmx’) P_tags = soup.find_all(‘p’)
Print(p_tags)
(I’m on mobile so convert to lower!)