r/SeleniumPython • • Mar 27 '25

Selenium Projects

1 Upvotes

I had created my first project in selenium with the help of python .I want to make sure if it's working correct or not .How can i post it here


r/SeleniumPython • • Feb 19 '25

Google flagged my web driver

1 Upvotes

Hey guys I accidentally opened like 8 drivers and afterwards I couldn’t use any websites on it. Facebook, Google, indeed. None of them worked. I tried using a vpn and that didn’t do anything either.

What should I do to solve this.


r/SeleniumPython • • Feb 11 '25

Best Approach for Automating Interactions on a Static Web Page with a Public API?

1 Upvotes

Hello,

I'm using Selenium to extract data from interactions on a static web page. The page itself relies on JavaScript to call several APIs. I use Selenium to automate interactions and test the "system", but I also need to allow the automated interactions via a public API.

Currently, I have an API (using lambda functions) that triggers Selenium with custom interactions. However, this setup feels a bit heavy. I managed to put everything into a FastAPI server, but I’m wondering if there’s a more efficient or lightweight alternative than combining Selenium and FastAPI.

Is there an existing tool better suited for this use case? Or any better way to use these tools and that you would be aware of?

Any insights would be greatly appreciated. Thanks!


r/SeleniumPython • • Feb 10 '25

Any tool suggestions for test automation reports and to check logs

1 Upvotes

Using selenium and python, Gitlab for CICD, test execution reports are generating in confluence page in my project. Feeling difficult to look at the fail and pass % of tests reports in confluence and test logs in jira test plan and it's time taking for me to go to testplan and 2 testcases are running in each job. If 80 testcases in parallel execution 40 jobs will run so, logs in jira would be like 40 zip folder will get created and need to check gitlab for in which job it failed and to check the job number on zipfloder in jira to check the failure logs.

Could anyone suggest if there is any tool we can integrate and to check the exec results and logs for failure at one place. It would be helpful for me and if any tool is used and storage is a concern. If the data stored on application. Could be thankful if anyone brief on this and help with detailed information.


r/SeleniumPython • • Feb 04 '25

Python Looking for the name of a Selenium Python docker project supporting HEADFUL

1 Upvotes

Hello.

Is there a project that runs Selenium in a docker container, HEADFUL, and to which you can not only run tests but you can also connect to via a browser and "observe" the running browser tests?

I found something like this months ago, thought "hey that's damn cool but I don't need it", then today it's suddenly work relevant and of course I can't Google it back.

So the features I remembered were

  • Containered tests (or a containered service that does the work, like a farm)

  • the container must run the browser tests in a "headful" manner (NOT headless), and could record video of the browser activity

  • Supported Python.

  • I want to say this project used Selenium. But I could be incorrect on this point (that's fine, I just want to find that project again, more than caring about the tech).


r/SeleniumPython • • Jan 17 '25

Help Xpath not handling all the data

1 Upvotes

I have been work on code to scrape insta post comments and like data's But i have this probelm with the comment It dont collect all the comment it skip the comments that have multi mention like (@anas @mack )and if there text at the end too can any one help me with the right xpath To make it collect all the deferent comments


r/SeleniumPython • • Jan 15 '25

Shadow Root is Killing Me In The Face!

1 Upvotes

Trying to download a PDF, and just getting humiliated.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
import time


# Setup Chrome driver
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

# Replace with the actual URL of the page
driver.get("https://historyofenglishpodcast.com/wp-content/uploads/2022/01/HOE-Transcript-Episode001b.pdf")
time.sleep(5)
download = driver.execute_script('return document.querySelector("#viewer").shadowRoot.querySelector("#toolbar").shadowRoot.querySelector("#downloads")')
time.sleep(5)

download_button = download.find_element(By.CSS_SELECTOR, "#download")                                 
download_button.click()
print("Download button clicked!")
time.sleep(5)  # give it time to download or take action

driver.quit() # close browser

Any clues would be greatly appreciated

Ted


r/SeleniumPython • • Dec 19 '24

any other alternatives to selenium wire?

1 Upvotes

i’m running a scraping tool via python that extracts network response from requests that return 403 errors. i started using selenium wire and i got it to work, but the main issue is the memory increasing more and more the longer i run it.

i’ve tried everything in order for it to not increase in memory usage, but ive had no success with it.

i’m wondering if anyone has had this problem and found a solution to access these requests without memory increasing over time. or if anyone has found another solution.

i’ve tried playwright and seleniumbase, but i didn’t have success with those.

thank you.


r/SeleniumPython • • Dec 17 '24

Help Selenium IE driver

1 Upvotes

Hi, there. Does someone has a functional project with Selenium IE Driver? I tried to do a simply auto login but can't use the IE driver successfully. I followed the selenium docs, Microsoft example and other sources. Can someone give a help about how to set up de IE driver successfully for a useful implementation?


r/SeleniumPython • • Dec 08 '24

unable to install selenium in colab in chrome.

1 Upvotes

Hello everyone, I am unable to install selenium in colab. I followed step by step procedures given on YouTube video as well used chatgpt & colab assistant. But I couldn't install it successfully. Is some directory changed in source code or what's the issue? Can you please help me?


r/SeleniumPython • • Nov 18 '24

Selenium/Python

1 Upvotes

Hello,

Is there somebody who can help me with the following:

I've created a .side file with the Selenium IDE extension within Google Chrome

Now I would like to schedule this script, so it will be performed every x period.

I'm looking into selenium/python/chromedriver, but I'm not getting there

Who can help me?

Thank you!


r/SeleniumPython • • Nov 08 '24

Can Selenium click acuarding to string content

1 Upvotes

Hi, my scrapper gonna be linked to an LLM, so the scrapper gonna send the data to LLM and LLM uses the scraped data to tell the Scraper where it should click and then scrape again.

The question is, how should it be done? Can I tell the LLM to choose string of the right options? Or another part should be returned from the output?


r/SeleniumPython • • Nov 03 '24

Running Selenium in Docker Container (on Proxmox) - Need Help Finishing

1 Upvotes

Hi there,

I'm a newbie so please bear with me. I've setup a container on Proxmox and have installed docker and Selenium Standalone Chrome (in the console it's coming back as having installed the repository for this).

I can access the Selenium Grid (at http://192.168.1.126:4444/ rather than localhost) and I can see it on the grid.

However, I can't access http://192.168.1.126:4444/wd/hub from a browser (says it can't find the handler) and when I try the following command it says 'too many arguments.

from selenium import webdriver

Any idea where to go next?

What I'm trying to do is to use Home Assistant (running on a separate VM in Proxmox) to pull the data from the following Python Script using the UKBinCollection integration found here (https://github.com/robbrad/UKBinCollectionData)

python collect_data.py ForestOfDeanDistrictCouncil https://community.fdean.gov.uk/s/waste-collection-enquiry -s -p "XXXX XXX" -n XX -w http://192.168.1.126:444/

Where I would enter the information at p as my postcode; n being the house number and w being the remote selenium URL.


r/SeleniumPython • • Nov 01 '24

Looping through a CSV to fill a website form out multiple times

1 Upvotes

I'm wrote this script to fill out a form using a CSV for the required fields. It fills it out once and clicks the submit button, but then it closes the browser and the script stops. I originally had the open browser part outside of the function but even within the loop, the script stops after one input.

import csv
import os
import chromedriver_binary
from selenium import webdriver
import selenium
import time
import dotenv
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from dotenv import load_dotenv, find_dotenv

load_dotenv('C:/path/credentials.env')
user=os.environ.get("USER")
pass=os.environ.get("PASS")


# Use Google Chrome as the browser to add entries
driver = webdriver.Chrome()
website="https://URL"


#Create function
def LoginToSite(): 
    with open('C:/path/clients.csv', mode='r', newline='') as csv_file:
        csv_reader = csv.DictReader(csv_file)
        for row in csv_reader:
            driver.get(website)
            time.sleep(2.5)
            UserField=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div[2]/div/div[2]/div/div[1]/div/input")
            UserField.send_keys(user)
            PassField=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div[2]/div/div[2]/div/div[2]/div/input")
            PassField.send_keys(pass)
            driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div[2]/div/div[2]/div/div[3]/button").click()
            time.sleep(15)
            #Sets variables from CSV file
            ClientName=row['Name']
            ClientDomain=row['Domain']
            ClientAddress=row['StreetAddress']
            ClientCity=row['City']
            ClientCountry=row['Country']
            ClientProvinceState=row['ProvinceState']
            ClientUserCount=row['UserCount']
            #ClientManaged=row['Managed']
            #ClientAccountType=row['AccountType']
            #ClientPermission=row['Permission']
            ClientNotes=row['Notes']
            print(f'Name {ClientName} Domain {ClientDomain} Country {ClientCountry}')
            #Fills form out
            CompanyName=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[3]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-input/lightning-primitive-input-simple/div[1]/div/input")
            CompanyName.send_keys(ClientName)
            DomainName=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[3]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-input/lightning-primitive-input-simple/div[1]/div/input")
            DomainName.send_keys(ClientDomain)
            Address=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[4]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-input/lightning-primitive-input-simple/div[1]/div/input")
            Address.send_keys(ClientAddress)
            City=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[4]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-input/lightning-primitive-input-simple/div[1]/div/input")
            City.send_keys(ClientCity)
            Country=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[4]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-lwc-field/div/flowruntime-dependent-picklists/lightning-select/div[1]/div/select")
            countryselect=Select(Country)
            countryselect.select_by_visible_text(ClientCountry)
            ProvinceState=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[4]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-lwc-field/div/flowruntime-dependent-picklists/lightning-select[2]/div[1]/div/select")
            provincestateselect=Select(ProvinceState)
            provincestateselect.select_by_visible_text(ClientProvinceState)
            UserCount=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[5]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-input/lightning-primitive-input-simple/div[1]/div/input")
            UserCount.send_keys(ClientUserCount)
            driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[5]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[1]/flowruntime-lwc-field/div/flowruntime-radio-button-input-lwc/fieldset/div/span[1]/label").click()
            driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[5]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[2]/flowruntime-lwc-field/div/flowruntime-radio-button-input-lwc/fieldset/div/span[1]/label").click()
            Notes=driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-lwc-body/div/flowruntime-list-container/div/flowruntime-base-section/div/flowruntime-screen-field[6]/flowruntime-lwc-field/div/flowruntime-flow-screen-input/flowruntime-input-wrapper2/div/lightning-textarea/div/textarea")
            Notes.send_keys(ClientNotes)
            time.sleep(10)
            #clicks Next button
            driver.find_element(By.XPATH, "/html/body/div[3]/div[2]/div/div/div[2]/div/div[3]/div/div/div/article/div/flowruntime-flow/flowruntime-navigation-bar/footer/div[2]/lightning-button/button").click()
#closes browser
#driver.quit()


LoginToSite()

r/SeleniumPython • • Oct 22 '24

How to Use new ChromeDriver

1 Upvotes

Howdy All, I have been maintaining code I wrote in 2020 and using it fairly successfully. Time for a major overhaul do to updated needs. I am ashamed to say that when chrome changed their driver delivery methods a year or more ago that I couldnt really get my head around it. I found a work around to do the worst, and install an older version of chrome for the moments I needed this code.

Hoping to figure out how to manage the new driver system and do a long needed overhaul of the scripts in general. Sorry I am not really even sure how to ask this question, but appreciate any help.

I am on a mac as well.

TIA.

-J


r/SeleniumPython • • Oct 17 '24

System specifications

1 Upvotes

What is the System specifications needed for running Selenium WebDriver in a Docker container


r/SeleniumPython • • Oct 13 '24

Getting data from table loading dynamically

1 Upvotes

I have a website from which I want to take some data. There is a table that lists the first 50 rows (scrollbar represents the totality of rows), and as you scroll down it loads the rows in batches of 50 rows and unloads the previous 50 (confirmed by watching the HTML, which always has only 50 ‘tr’ elements).

I have tried sending keys but does not work as once the focused line reaches the 50th, this returns to the beginning of the current “batch”.

Any ideas on how to do it? By now I am stopping after every batch, scroll manually to next batch (easy to miss batches) and continue, but I have pages of more than 20 batches.


r/SeleniumPython • • Sep 27 '24

Help Issues with chromedriver on linux, "no chrome binary at usr/bin/google-chrome"

1 Upvotes

I am trying to run some tests with selenium but for some reason it is giving me the error described in the title, even though google-chrome is definitely in /usr/bin. Both Chrome and chromedriver are the latest versions. Any ideas why this might be happening?


r/SeleniumPython • • Sep 24 '24

Help Unable to access "#text" element?

1 Upvotes

Hello, I'm new to web scraping and selenium and wanted to web scrape this website:
https://rarediseases.info.nih.gov/diseases/21052/10q223q233-microduplication-syndrome

one of the texts I want to grab is the disease summary, which seems to be the child of the element denoted by this XPATH: "/html/body/app-root/app-disease-about/disease-at-a-glance/div/div/div[2]/div[1]/app-disease-at-a-glance-summary/div/div[1]/div/div"

the line of code I'm trying to run to grab it is:

driver.find_element(By.XPATH, "/html/body/app-root/app-disease-about/disease-at-a-glance/div/div/div[2]/div[1]/app-disease-at-a-glance-summary/div/div[1]/div/div").text

However, whenever my code runs, it returns an empty ' ' string
I've tried adding "//*" at the end of the XPATH as it seems like the text is actually stored as a child element, but I get a "no such element" exception. I've looked into CSS selectors but seem to run into the same issues. I've looked everywhere and couldn't find a solution or explanation, but I also recognize my experience with HTML and web scraping is limited. Any suggestions and help are greatly appreciated!


r/SeleniumPython • • Sep 23 '24

Stuck at timeoutexception error

1 Upvotes

Hi guys, I am trying to extract some data from a webpage. The data is on a table and there are 87 rows, some rows are not visible and require scrolling down on the table.

I have written this code:

time.sleep(60)

rows = WebDriverWait(driver, 60).until(EC.presence_of_all_elements_located((By.XPATH, "//div[@id='PSBUses']//div[contains(@class, 'x-grid3-row')]")))

I increased the waiting time to 150, didn’t work. Still receiving timeoutexception error. Does anybody have any suggestion to this? I am sooo stuck


r/SeleniumPython • • Sep 21 '24

Automated Import of Holdings to Google Finance from Excel

Thumbnail reddit.com
1 Upvotes

r/SeleniumPython • • Sep 19 '24

Send keys without specifying element Selenium webdriver

1 Upvotes

so i want to access the username and password without without specifying element and login https://practicetestautomation.com/practice-test-login/ how to do that?


r/SeleniumPython • • Sep 19 '24

Help Using selenium to login to reddit

1 Upvotes

Hi Guys,
Im new to webscraping and was trying to login to reddit via selenium.
Im able to enter the login details , but Im not able to select the login button to continue, I've tried using xpaths , css selectors and it looks like theres something called DOM that might interfere with the process.

iv tried using css selectors to get around it , but iv been stuck at this for a while, Any help with this would be awesome and a lifesaver!!


r/SeleniumPython • • Sep 17 '24

Selenium uses a ton of internet data in conjunction with Google Drive upload

1 Upvotes

Hi there,

I am writing a program in Pyhon with Selenium on Mac OS that downloads .pdf files from a website and uploads the .pdfs to a Google Drive folder. The pdfs are only a few pages and average at around 300-400kb of data, and I'm downloading at most 50 .pdf files. There are .tmp.drivedownload folders that take up a ton of data in my downloads, with files inside that look like this, e.g. ".com.google.Chrome.AzphV3". These files range from 1-4gb and also populate in my Google Drive, filling up my limited 15gb of storage.

This has caused huge spikes in my internet data usage. When I started this a few days ago, I went through almost all of my data. Here is a photo of my daily usage from my Internet Provider:

When investigating further, most of my data usage is under the "Other" category. It can not be located or traced.

My code is long, but this is the function I wrote to move my .pdf from my downloads folder into my Google Drive folder:

def move_file_to_manifest_folder(manifest_dir,j):
    
    downloads_dir = '/Users/stepdoe/Downloads/'
    time.sleep(3)
    
    # Here I'm searching in my downloads folder for the last .pdf downloaded, then I  am moving that file into my Google Driver folder with os.replace 
    files = list(filter(os.path.isfile, glob.glob(downloads_dir + "*.pdf")))
    files.sort(key=lambda x: os.path.getmtime(x))

    filename = files[-1] # after I sort by time with os.path.getmtime, I take the last file in my list, which corresponds to my most recent file downloaded.
    filename = filename.split('/')
    filename = filename[-1] 
    print(f'filename[-1]: {filename}')
    filename = str(j).zfill(2) + '_' + filename # naming convension for what I want my file to be called in my Google Drive
    newpath = f'{manifest_dir}/{filename}'
    print(f'newpath: {newpath}')
    os.replace(files[-1],newpath)

I am asking for solutions to prevent these huge spikes in data download and uploads. I would expect my daily increase in usage would increase by 2-3gb (at the most 5gb), not in the order of magnitude of 100-500gb. Any help on this would be great, as my internet bill will skyrocket without it.