r/Python • u/[deleted] • Nov 28 '20
Intermediate Showcase "Sillynium" - Auto generate working Python Selenium scripts by drawing coloured rectangles around web elements on a page that you want to automate
TLDR: 60 sec video here:https://streamable.com/1y17lt
TLDW: When I want to automate actions on a site, such as logins or forms, or scraping pages with JavaScript, I manually inspect each web element I need and begin writing a script.
I'm lazy though and had the idea to automate the creation of these automation/testing scripts. And thus "Sillynium" was born.
Sillynium allows you to draw boxes around the elements you require on a page and automatically gets the element and its properties for you. Different coloured rectangles tell the program you want to do different things. Eg: Red means "Input text", Green means "Click button".
When you've finished drawing boxes, a working python selenium script is generated. If you run that script, it does exactly what you told it to do! All this without you writing any code, and losing time.
Working 60 second video link of program in action: https://streamable.com/1y17lt
I made this project over the course of a day as a quick and dirty proof of concept. Its very flawed in its current iteration (taking a screenshot of webpage), so V2 I'm trying to figure out how to draw directly to the browser. Critique it and rip it to shreds. I'm aware of browser recorders btw so I know those exist, this is my twist on automation
Edit: I was not prepared for the amount of interest in this little proof of concept! So thank you all very much for your time you’ve made my day :)
In the interest of transparency and to answer your questions about the code and my plans:
I originally thought about monetisation (web extension etc) after finishing this POC (**sigh i know I’m a greedy bastard) but now I’m in 2 minds about it!
On the one hand I want to make it free for personal use, on the other hand I’d like to potentially charge for commercial use (once its finished).
Final edit 2 The community has spoken, I WILL BE RELEASING THIS VERSION AS OPEN SOURCE! I did some light refactoring, and have added it to a repo which will be shared in a follow up. It shows the POC and the community is welcome to build on the original idea :)
I will create a follow up post with links to repo once I’ve created it, this will be my first project on GitHub so try be nice ;) pretty nervous about sharing my code :D
Thank you all for your patience, interest, and awesome discussions
95
u/kingbrady12 Nov 28 '20
That is some crazy sht. Wow! Amazing man
15
Nov 28 '20
Cheers guys really motivating hearing the community feedback on what was just going to be a small test proof of concept!
Now i just need to learn how to use GitHub 😂🥺
10
52
u/NoblySP Nov 28 '20
Is there a GitHub repo for your project?
28
Nov 28 '20 edited Nov 29 '20
Not at the moment! 2 reasons, 1st: i’m still a baby at Python (9 months) and haven’t yet used GitHub for a project!
2nd: to be completely transparent with everyone, I originally thought about monetisation after finishing this POC (**sigh i know I’m a greedy bastard) but now I’m in 2 minds about it!
On the one hand I want to make it free for personal use, on the other hand I’d like to potentially charge for commercial use (once its finished).
Since this idea is quite fresh I’m still ironing everything out before making that (big) decision. I welcome discussion though it would help alot in figuring out the way forward
Edit: open source coming in follow up post!
24
u/LividPhysics Nov 28 '20
I think open source would be the way to go. If you want more help with improving your project there's no better way than getting some guy who has been doing Python for 15 years to contribute and learning from them.
16
Nov 28 '20
Im leaning towards that being the way to go, as i love this community 😍😍 just have a bit of reading to do before making that dive !
15
2
15
Nov 28 '20
I'm gonna be the other guy and say you should totally consider a license with royalties for companies who use this and themselves monetize; it's essentially money you've earned them!
Also I could absolutely see this being a paid chrome extension, or a web platform for mass automation for dummies!!
Anyway, great stuff, I'm jealous.
4
Nov 28 '20
Thanks mate!! Appreciate the alternative view! One way I’ve thought if is license based but free for a single user (so individuals like us can use) but charged for each additional user of an organisation! The heads still spinning on this one though 😇
5
Nov 29 '20
You built something, it is up to you what to do with it. My 2 cents is to make it free for personal use and not free for commercial use.
3
7
u/Oimmuk Nov 28 '20
Instead of making money off of it, use it as one of the examples of your creative ability in your portfolio. A project you lead/maintain that gets alot of attention looks good on a resume. That is the fruit of your labor. Bring others in on your project and make it a robust and feature rich stand alone application that allows non coders to automate thier own tasks.
3
Nov 28 '20
Thanks Oimmuk! I’m trying to figure out a way to juggle it all, clarity will come in time just need to process it all :) my grand plan is to open this up as a browser extension pure task automation tool for non coders 🙂
1
3
9
u/WishIWasOnACatamaran Nov 28 '20
I want this repo so bad rn
1
4
1
26
u/ES-Alexander Nov 28 '20
Nice!
Might be worth using cv2.selectROI to have some visual feedback while drawing each rectangle :-)
4
25
u/infinitejest69 Nov 28 '20
Looks good. I use selenium in java a lot this might finally give me reason to play with it in python. Would be cool to generate page object models just picking elements you want.
2
24
u/SharksPreedateTrees Nov 28 '20
Amazing idea, but the devil is in the details. I noticed your script has s tendency to choose the wrong element (either the parent or the child of the desired element). Your project will suffer for the same reason other recording software(e.g. katalon) fails to generate selenium scripts. Great effort though
6
Nov 28 '20
Good point (and eye)! This is just the early POC, and you’re probably spot on there! But hoping to iron out the kinks in V2 ;)
8
u/SharksPreedateTrees Nov 28 '20
Just make sure to have realistic expectations. You are attempting to trivialize an entire field of Software Engineering(really QA engineering). Even the original selenium devs tried to make something like this work (selenium IDE) and it fails for 95% of use cases
6
Nov 28 '20
I understand where you’re coming from 100% Sharks, and I’m really hoping something great comes of it! The way I see it is if it has a similar outcome as software like Selenium IDE thats a good first step, and where I fail others can improve on the idea (or hopefully it inspires new ideas in others). Failing upwards :)
4
159
u/garlic_bread_thief Nov 28 '20
My man just automated automation.
39
u/Aryamaan27 Nov 28 '20
Let's automate that
49
6
u/dethb0y Nov 29 '20
you could probably write an image recognition program that identified areas on a page by "sight"....
1
u/dparks71 Nov 29 '20
Can't wait for webcam eye tracking, we know you thought about buying this, so we slipped it in your cart.
1
u/dethb0y Nov 29 '20
you joke, but i could legit see a role for an application that worked as follows:
A bunch of items are shown on screen (say, shirts)
An eye-tracker gauges which one you looked at the longest or lingered over
it suggests that item as the one you really want
I bet for some people, that would be really nice. There's many items (snacks, clothes, etc) that are fundamentally similar in price, function etc and the choice is just between those like items, and that would help narrow down the one you really wanted vs. the internal debate of which to choose
1
u/dparks71 Nov 29 '20
Legit role for the application doesn't necessarily mean I want it, or would be okay with them implementing it without being upfront about it. Same reason I don't have Amazon/google devices in my house, they're waaaayyyy too desperate to give them away for free for me to trust them.
9
u/Deceitful_Sloth Nov 28 '20
This looks really good, can you link your codebase? I really want to see how this works! Keep up the good work.
2
0
Nov 28 '20
[deleted]
1
Nov 28 '20
Hey guys/girls thanks for the interest! See my edit :) feel free to critique me on that decision too please:)
0
6
u/gargar070402 Nov 28 '20
Might be a silly question, but what does this do that Selenium IDE doesn't? As far as I'm concerned, Selenium IDE already supports recording actions done on a browser.
1
Nov 28 '20
Great question! I personally have never used a browser recorder (Katalon, Selenium IDE) so I can’t answer too much on that. But from having a very brief look at them I think mostly its just a different way of doing things.
One point however, is that my program can also scrape alongside recording your actions which i don’t think the other programs can do :) so if you have a URL with a form you want to automate but also scrape page content at the same time, you can do that! Others feel free to correct me
5
u/PovertyNomad Nov 28 '20
This is AMAZING! I'm amazed that you only have 9 months Python experience! Do you mind sharing how you learned Python? I'm about 4 months in now and nowhere near this level, although I've spend the last ~2 months learning HTML, CSS and Django...
Any good resources you could recommend?
3
Nov 28 '20
Thanks heaps means alot
I started with the hardcover book “automate the boring stuff” by Al Sweigart (shout out to u/AlSweigart loved your book and getting your new one too!) and was amazed how quickly that opens up what you can do with Python. Naturally one of the chapters discusses Selenium and web automation. I spent almost every day on this book, highlighters in hand (im old school)
Well i finished that book and straight away purchased Python Crash Course by Eric Matthes which I’m working through now and almost finished, which cemented the learnings and taught new things (OOP/Classes)
But biggest of all is my job. Im very fortunate in the fact I managed to convince my boss that I can perform tasks at work in Python, and so I spend about 4 hours everyday working with Python developing tools for work. I think thats the big advantage I’ve had.
I took up a few Selenium automation jobs from UpWork too, and my issues with those actually inspired the idea :)
I also get home and if my partners at work I try building small projects
13
u/MrDeebus Nov 28 '20
The video looks super nice, but how am I supposed to rip it to shreds without a link to the code? Come on man, I need my Saturday afternoon dose of false superiority!
2
Nov 28 '20
Thanks my first ever! Good old windows photo editor 😅 please see my edit re the code! Feel free to critique me on that too btw
2
u/MrDeebus Nov 29 '20
Haha no yeah I meant more the content of the video, it's to the point, brevity is not very common.
As for your edit... well, good luck, bold approach if you ask me -- especially in the python ecosystem. I can't fault you for wanting to monetize your effort, but closed-source (which goes beyond proprietary) software is a pretty drastic move, mostly reserved for in-house or SaaS stuff these days... but hey, obviously your call! Your code is your code.
I will offer critique on something else though: I saw elsewhere that you "need to learn how to use GitHub", and based on that I will assume you aren't familiar with git itself (because once you have that, basics of github is like 10-15 clicks and 50 keystrokes, for username/password/projectname). I really, very strongly advise you to get that out of the way first, no matter what you decide to do with the project. It's (thankfully) fairly ubiquitous by now, and version control is an indispensable tool. Think of it like gloves of a forester, where code is the chainsaw, and software engineering the forestry discipline (maintaining a forest to reap its produce in the long run).
And of course, apologies if I went too far in my assumption and your inexperience is limited to GitHub the product. In which case, learning it is like 15 minutes of reading at most, just do it :)
1
Nov 29 '20
Thanks appreciate it i try to be succinct! Yeah i do know and use git and you’re right i need to learn to work with github, probably is a 10 minute job I’ve just put it off :)
1
0
4
4
u/SweetSoursop Nov 28 '20
Dude, you could absolutely topple UIPath, Automation Anywhere, BluePrism and the others, mad respect!
1
1
3
u/EversonElias Nov 28 '20
Great idea! It would have saved me a ton of time and joy of life, hahaha. Keep the great work and let us know about the progress.
2
3
3
u/Frankenstien456 Nov 28 '20
If you ever decide to go through with making the extension, you can use the pyjs (http://pyjs.org/) python to javascript compiler.
2
3
3
3
Nov 29 '20
That’s amazing! You definitely ought to monetize this! I’d pay in a heartbeat
3
2
2
u/def_hass Nov 28 '20
Amazing work man! Can you give a short overview of how this works or link a repo where we can look? Will this work with pages where the elements have no IDs?
3
Nov 28 '20
I’d love to tell you how it works! No repo atm (see my edit) but heres how it works:
It grabs a screenshot of the page/URL you give it. It then launches an editor for you to draw the rectangles (with number key-mapping for different colours). The rectangle coords and colours are then used as a reference. Then the rectangle coords are compared against the site pixel coords and basically does an “inspect element” and grabs every possible detail about the element to use if no ID (ID/CLASS/SELECTOR/XPATH)
Then those elements are stored with their attributes, then the code is generated (written/appended) to a .py file with all the elements and the actions (with actions based off colour) !!!!! PHEW 😅
1
2
2
2
2
2
u/davidpofo Nov 28 '20
Do you have plans to make this open source? I would really benefit from this every day and would gladly contribute to improve it.
1
1
2
u/neisor Nov 28 '20
Nice project! Will try it out for sure.
Btw, what color theme are you using? Thanks
2
Nov 28 '20
Thanks! I think its called Synthwave 84, really cool theme right? Has a “glow” option too which makes the text look like a neon sign
2
2
2
u/alexandros87 Nov 28 '20
That sound off in the distance is 1000 automation engineers gulping nervously.
Seriously though this looks incredibly cool!
2
2
u/ergelshplerf Nov 28 '20
I'm lazy though and had the idea to automate the creation of these automation/testing scripts. And thus "Sillynium" was born.
Why not Selazium?
2
Nov 28 '20
Didn’t think of that one! Did think of Sleazium though because i felt dirty automating automation
2
2
u/ravepeacefully Nov 28 '20
What’s better about this than scrapy?
2
Nov 28 '20
Hey rave I haven’t used scrapy personally, though i’ve heard the learning curve can be initially quite steep and requires additional envs and installs. I guess this simplifies processes alot, and since we are visual creatures it works with that!
2
u/ravepeacefully Nov 28 '20
Ahh ok very cool but very similar to scrapy, you should check out theirs and use their ideas. Their founder is doing quite well in the entrepreneurial world
2
2
u/jobar86 Nov 28 '20
Formal QA test engineer here, if someday you want help testing it, and maybe find some defects 😁
2
1
2
u/AndresArc123 Nov 29 '20
It is a good idea.
if the element has no id, how do you map it? Absolute xpath?
Have you thought about how to handle multiple windows or iframes? dynamic elements such as those generated by react?
2
Nov 29 '20
Hey thanks for the interest! I have a priority sequence to handle that. So if there no ID, ill look for a name, if theres no name, ..... all the way down the list to relative and absolute XPaths :)
Havent yet implemented that functionality but would be doable for the iframes for sure
Can’t yet say about dynamic elements but ill keep you posted with V2/3
2
u/Shakaka88 Nov 29 '20
You could always do something similar to Gyazo, free users can get like X free uses or only button clicking for free and text input paid, etc. something like those ideas may help you make the best of both worlds.
1
2
2
u/mortenb123 Nov 29 '20
Impressive, There are software doing this out there like https://smartbear.com/product/testcomplete/features/, but it is costly and windows only.
I'm using selenium in headless mode inside containers, but currently my most time-consuming task is to nest out Reacts :shadow virtual dom, this is code being executed when you click of move the cursor to the object and outside of webdriver. You then manually need to dig up and execute and use css-selectors to interact with the objects. My record is 8 levels of :shadow. and the object to execute may be different from level to level.
Any tool that automates this I'm willing to pay a for.
2
Nov 29 '20
😮 i havent had too much interaction with shadow doms but I’ll try see if its a possibility!
2
u/rmpr_uname_is_taken Dec 03 '20
Reminds me a project of my own atbswp
1
Dec 03 '20
So cool! As soon as i saw that acronym i knew what it would be ;) very clever, i had an idea to extend this to offline use as well! But I’m more comfortable working with the web :) keep an eye out for my follow up post! I’ve made massive ground, Would be keen to chat!
1
1
u/theantidrug Nov 28 '20
I’m sure you will get better and more critiques if you post the code.
2
Nov 28 '20
Good point antidrug, but see my thoughts/decision in the edit and feel free to critique:) cheers
2
2
u/fraud_93 Nov 28 '20
Thank you for this. It's not lazy, sometimes we have a hard time selecting shit selenium tries to fuck up.
2
2
u/gooeydumpling Nov 29 '20
Automate-ception, the name is counterintuitive tho, this is not silly at all but reallllly smart
2
2
u/f_leejy Nov 29 '20
This is pretty cool! I think it’s already a feature tho in selenium Selenium IDE
2
u/just4fun50 Nov 29 '20
So I’ve always had a certain dislike of Selenium. It’s clunky. Usually prefer the requests route if possible. I’ll be interested in if this truly eases using Selenium. Will be trying it out soon!
1
u/manimal80 Dec 02 '20
that is great!
how do you draw the rectangles ? can it be done only with python and selenium?
0
0
0
Nov 28 '20
[deleted]
1
Nov 28 '20
Great question! I haven’t coded anything to handle shadow dom, but think it’s possible to add eventually (read about it on stackoverflow)
0
Nov 28 '20
[deleted]
1
Nov 28 '20
Depends on the site, an easy way to debug is to not run in headless mode so you can see the elements you’re interacting with
0
Nov 29 '20
[deleted]
1
Nov 29 '20
Need to be specific with what exact parts you’re struggling with and what exactly you’re trying to accomplish/ pseudocode
1
u/prokid1911 Nov 28 '20
How are you doing it currently?
0
Nov 28 '20
[deleted]
1
u/prokid1911 Nov 28 '20
I'll suggest using Selenium Webdriver. Much easier to scrape with it
0
Nov 28 '20
[deleted]
1
u/prokid1911 Nov 28 '20
It is much simple. You navigate through pages via clicks, fill in form elements with send_keys and then use xpath to scrape what you want. Just navigate to the page you wanna scrape.
It's much easier to scrape aspx pages through Selenium.
1
u/warxhead Nov 28 '20
The biggest issue I run into is speed there. If I can use requests to login to authentication without all those damn Ajax calls I'll use it in a heartbeat with beautiful soup. Bs4 is lightning fast and so easy to find elements. I have not once had any trouble with it.
1
u/prokid1911 Nov 28 '20
I have had difficulty with pagination when scraping aspx websites. Then I came accross Selenium. It handles all the things with ease and you can also run it in headless mode.
You can scrape in any way you like. It's totally your call.
2
u/warxhead Nov 28 '20
Yes, in your use case that seems like selenium is the best option. I use it also daily for my work script to retrieve from web pages that requests doesn't like to authenticate to.
Headless is fast - but still not as fast as requests. Sending a post form through requests has always been faster than using selenium to enter each field.
1
u/jawsem27 Nov 28 '20
If you are trying to use selenium with python, search for katalon recorder. It can record the actions you take in your browser and generate code in multiple languages that does the same thing in multiple languages.
If i am using selenium to automate browser action i usually start with katalon and then move to Python.
1
u/warxhead Nov 28 '20
You're welcome to PM me and explain what you are trying to do and I can give you some pseudo code. I use requests /Bs4 on a daily basis
2
Nov 28 '20
[removed] — view removed comment
2
1
Nov 28 '20
I dont have any other accounts to share it on maybe try diff browser or device? 😅 cheers! Fyi on the open source check my edit!
3
•
u/Im__Joseph Python Discord Staff Nov 28 '20
Hey there!
Going by the responses to this post it seems a lot of people are impressed with your project, myself included!
We won't be removing this post, but I'd like to draw your attention to rule 5 of the subreddit which concerns showcase flairs.
If possible it would be really great to include some form of source code on a platform like GitHub, this allows projects posted here to be educational for all.
We may enforce rule 5 automatically at some point in the future, but for now we'll keep this up since it's generated some nice discussion.
Cheers,
Joe