r/Python • u/alexkidd1914 • Nov 14 '13
webscraping: Selenium vs conventional tools (urllib2, scrapy, requests, etc)
I need to webscrape a ton of content. I know some Python but I've never webscraped before. Most tutorials/blogs I've found recommend one or more of the following packages: urllib2, scrapy, mechanize, or requests. A few, however, recommend Selenium (e.g.: http://thiagomarzagao.wordpress.com/2013/11/12/webscraping-with-selenium-part-1/), which apparently is an entirely different approach to webscraping (from what I understand it sort of "simulates" a regular browser session). So, when should we use one or the other? What are the gotchas? Any other tutorials out there you could recommend?
7
Upvotes
11
u/westurner Nov 15 '13 edited Nov 15 '13
https://en.wikipedia.org/wiki/Web_scraping
https://en.wikipedia.org/wiki/Robots.txt_protocol
http://docs.python.org/2/library/robotparser.html
https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers
http://lxml.de/index.html#documentation (libxml2 + libxslt)
http://www.crummy.com/software/BeautifulSoup/bs4/doc/
http://html5lib.readthedocs.org/en/latest/
http://pythonhosted.org/pyquery/
http://stackoverflow.com/questions/8692/how-to-use-xpath-in-python
https://en.wikipedia.org/wiki/Selenium_%28software%29
https://en.wikipedia.org/wiki/Selenium_%28software%29#Selenium_WebDriver
http://www.seleniumhq.org/projects/ide/
http://stackoverflow.com/questions/18094238/what-is-the-page-object-pattern-in-selenium-webdriver
http://casperjs.org/
http://phantomjs.org/
https://github.com/ariya/phantomjs/wiki/Headless-Testing
https://github.com/detro/ghostdriver
http://slimerjs.org/
http://doc.scrapy.org/ (Twisted + lxml)
http://doc.scrapy.org/en/0.20/topics/architecture.html
http://doc.scrapy.org/en/0.20/topics/firefox.html
http://doc.scrapy.org/en/0.20/topics/firebug.html
http://docs.python-guide.org/en/latest/scenarios/scrape/ (requests + lxml)
http://redd.it/1c6866