r/Python • • Nov 28 '20

Intermediate Showcase "Sillynium" - Auto generate working Python Selenium scripts by drawing coloured rectangles around web elements on a page that you want to automate

TLDR: 60 sec video here:https://streamable.com/1y17lt

TLDW: When I want to automate actions on a site, such as logins or forms, or scraping pages with JavaScript, I manually inspect each web element I need and begin writing a script.

I'm lazy though and had the idea to automate the creation of these automation/testing scripts. And thus "Sillynium" was born.

Sillynium allows you to draw boxes around the elements you require on a page and automatically gets the element and its properties for you. Different coloured rectangles tell the program you want to do different things. Eg: Red means "Input text", Green means "Click button".

When you've finished drawing boxes, a working python selenium script is generated. If you run that script, it does exactly what you told it to do! All this without you writing any code, and losing time.

Working 60 second video link of program in action: https://streamable.com/1y17lt

I made this project over the course of a day as a quick and dirty proof of concept. Its very flawed in its current iteration (taking a screenshot of webpage), so V2 I'm trying to figure out how to draw directly to the browser. Critique it and rip it to shreds. I'm aware of browser recorders btw so I know those exist, this is my twist on automation

Edit: I was not prepared for the amount of interest in this little proof of concept! So thank you all very much for your time you’ve made my day :)

In the interest of transparency and to answer your questions about the code and my plans:

I originally thought about monetisation (web extension etc) after finishing this POC (**sigh i know I’m a greedy bastard) but now I’m in 2 minds about it!

On the one hand I want to make it free for personal use, on the other hand I’d like to potentially charge for commercial use (once its finished).

Final edit 2 The community has spoken, I WILL BE RELEASING THIS VERSION AS OPEN SOURCE! I did some light refactoring, and have added it to a repo which will be shared in a follow up. It shows the POC and the community is welcome to build on the original idea :)

I will create a follow up post with links to repo once I’ve created it, this will be my first project on GitHub so try be nice ;) pretty nervous about sharing my code :D

Thank you all for your patience, interest, and awesome discussions

1.5k Upvotes

171 comments sorted by

View all comments

Show parent comments

0

u/[deleted] Nov 28 '20

[deleted]

1

u/prokid1911 Nov 28 '20

I'll suggest using Selenium Webdriver. Much easier to scrape with it

0

u/[deleted] Nov 28 '20

[deleted]

1

u/prokid1911 Nov 28 '20

It is much simple. You navigate through pages via clicks, fill in form elements with send_keys and then use xpath to scrape what you want. Just navigate to the page you wanna scrape.

It's much easier to scrape aspx pages through Selenium.

1

u/warxhead Nov 28 '20

The biggest issue I run into is speed there. If I can use requests to login to authentication without all those damn Ajax calls I'll use it in a heartbeat with beautiful soup. Bs4 is lightning fast and so easy to find elements. I have not once had any trouble with it.

1

u/prokid1911 Nov 28 '20

I have had difficulty with pagination when scraping aspx websites. Then I came accross Selenium. It handles all the things with ease and you can also run it in headless mode.

You can scrape in any way you like. It's totally your call.

2

u/warxhead Nov 28 '20

Yes, in your use case that seems like selenium is the best option. I use it also daily for my work script to retrieve from web pages that requests doesn't like to authenticate to.

Headless is fast - but still not as fast as requests. Sending a post form through requests has always been faster than using selenium to enter each field.