r/learnpython • • Dec 23 '19

What are some creative small programs that you have coded using Python?

I am a beginner at coding with Python, so I am curious about learning what small programs or projects you all have created. I am hoping to develop my skills using Python, so I figured looking at actual code for various programs would greatly help.

Please include a short description of what the program does and the code itself in the comments below. Thank you!

348 Upvotes

224 comments sorted by

View all comments

42

u/Sumat2222 Dec 23 '19 edited Dec 23 '19

I'm currently looking for jobs as a Data Analyst, which was why I started to learn Python in the first place. I managed to make a bot with Selenium that can automatically apply to jobs on LinkedIn for you. You can provide it your (email), (password), (job), (area) so it can Apply automatically with your previously stored settings.

Pretty cool the things you can achieve with Selenium!

7

u/szirith Dec 23 '19

Pretty cool the things you can achieve with Selenium!

Is there a reason you chose selenium over beautifulsoup?

21

u/Voxxey Dec 23 '19 edited Dec 23 '19

Selenium is web automation with the ability to webscrape.

Beautiful soup is easy web scraping.

7

u/[deleted] Dec 23 '19

Can you even use beautifulsoup for handling JavaScript?

I wasn't aware it had such a functionality.

4

u/szirith Dec 23 '19

I don't know! I'm hazy on the details and trying to understand the design choices being made.

4

u/[deleted] Dec 23 '19 edited Dec 23 '19

From my understanding, Selenium is useful for when you need to navigate a webpage to retrieve information form it.

If you have a static page, then bs4 should do the job.

For example, recently I designed a simple program to navigate a website and download all PDFs. The problem was that the website had a disclaimer page and a cookie consent banner which "hid" the main page. So passing the URL to beautifulsoup wouldn't have found anything.

I had to use Selenium to click the "I agree" button and then close the cookie consent form which then loaded the actual page. From there there was an infinite scroll page (much like Facebook, Reddit etc) so you would only have the full collection of PDFs if you scrolled to the bottom. Again, using bs4 I would have only retrieved the first 20-30 pdfs that were visible without scrolling.

I ended up using Selenium so much for that little project that I actually learnt to love it. It's my default go to now.

Here's the simple program if you are interested:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
import pandas as pd
pd.set_option('display.width',1920)
pd.set_option('display.max_columns',20)

# Driver Setup
download_dir = r'xxx\'
chrome_options = webdriver.ChromeOptions(); 
chrome_options.add_experimental_option("prefs", {"download.prompt_for_download": False,"plugins.always_open_pdf_externally": True}) 
prefs = {'download.default_directory' : download_dir}
driver = webdriver.Chrome(executable_path=r'xxx',options=chrome_options)

# Webpage setup
id = "xxx" 
id_name = "xxx" 
agree_ID = 'xxx' 
banner_close = 'xxx' 
list_elements = "xxx" 
fund_group_list = 'xxx' 
go_ID = "xxx" 
factsheets = 'xxx' 
KIIDs = 'xxx'

#Grab URL
driver.get('xxx')

#Navigate through disclaimer and cookie accept
def get_passed_disclaimer():
    cookie_accept = 'cookieBanner-close'
    agree_checkbox = 'form-row-checkbox' 
    agree_button_ID = 'xxx' 
    try:         
        driver.find_element(By.CLASS_NAME,cookie_accept).click() 
    except: 
        print('No cookie alert detected')
    time.sleep(0.5)
    driver.find_element(By.CLASS_NAME,agree_checkbox).click()
    time.sleep(0.5)
    driver.find_element(By.ID,agree_button_ID).click()


get_passed_disclaimer()


#Infinite scroll
SCROLL_PAUSE_TIME = 2.0
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
     driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight") 
    if new_height == last_height: 
        break 
    last_height = new_height

#Grab pdfs and tabulate output
fund_name, document_name, download_link = 'lit-c1', 'lit-c2', 'lit-c4'
links = [link.get_attribute('href') for link in driver.find_elements(By.CSS_SELECTOR,f'.{download_link} a')] 
fund_list = [funds.get_attribute('innerHTML') for funds in driver.find_elements(By.CLASS_NAME,fund_name)] 
doc_list = [docs.get_attribute('innerHTML') for docs in driver.find_elements(By.CLASS_NAME,document_name)]

#Add top row for links to ensure all columns are equal length
links.insert(0,'Links')

#Create dataframe for output
df = pd.DataFrame({'Fund_Names': fund_list, 'Document_Name': doc_list, 'Link_Ref': links})
df.to_csv('xxx.CSV')

# download all PDFs
for rows in df['Link_Ref']:
    try: 
        driver.get(rows) 
    except: 
        print("Bad Link")

1

u/Sumat2222 Dec 23 '19

I faced some issues with BS4 and with all the constant parsing, it ended up taking longer. So I decided to go with Selenium, so if you don't want Phantom.js to do everything automatically BTS you could just use Chrome and see it work in front of you.

3

u/dontanswerme Dec 23 '19

Can this be done through LinkedIn API?

1

u/Sumat2222 Dec 23 '19

It can be done through LinkedIn API. There were some people suggesting me to go that route but I haven't worked with APIs before and Selenium just looked a lot easy to work with.

2

u/SimpleCanadianFella Dec 23 '19

That sounds awesome! Can you share the code? Is love to add to that

5

u/Sumat2222 Dec 23 '19

Ah yes, that'd be great! I'll post a link on this sub soon. I'm almost done with it. Just have been looking at all the different scenarios while applying to a job that could potentially ruin the flow.

2

u/SweetSoursop Dec 23 '19

Conversely, I made one that searches for candidates on LinkedIn.

But don't tell them that :)

0

u/scottishbee Dec 23 '19

Does it alert you to any non-standard and/or optional questions before applying?

As someone who's hired analysts at five companies I always have included simple questions (eg "what's a technical read you'd recommend?" or "what's the toughest question you've been dying to work on?") just to gauge level of interest and give us a starting topic for a call (resume review comes after the first call). Anyone who doesn't bother answering them automatically gets rejected.

2

u/Sumat2222 Dec 23 '19

I know what you mean and no it does not yet. Just worked on it as a little side project. It's only putting code and links for Selenium so it got a bit repetitive and I lost motivation. Will add that to it.