r/SeleniumPython • • Jan 28 '25

How to overcome browser verification by friendlycaptcha with selenium python?

2 Upvotes

When i start chrome with selenium python I cannot solve friendlycaptcha. in the demo page https://friendlycaptcha.com/#demo "Verification failed: Browser check failed, try a different browser".

It's used in a webpage login I need to access; I'm fine with logging manually, but I need to overcome this verification.


r/SeleniumPython • • Jan 02 '25

Selenium Is it worth to learn Selenium python? After spending 2 years in manual trying to move to the automation

3 Upvotes

r/SeleniumPython • • Sep 27 '24

Help Issues with chromedriver on linux, "no chrome binary at usr/bin/google-chrome"

1 Upvotes

I am trying to run some tests with selenium but for some reason it is giving me the error described in the title, even though google-chrome is definitely in /usr/bin. Both Chrome and chromedriver are the latest versions. Any ideas why this might be happening?


r/SeleniumPython • • Jan 21 '25

Help Studying Python Selenium for 2weeks.

3 Upvotes

Hello everyone,

Background: former software engineer with 2 years and 5 months of experienced.

I am from Philippines and currently unemployed. So, what I actually do in my free time is to upskill/learn something new. I've been studying python selenium for the past 2 weeks. I know the waits, locators, xpaths, find_element and etc. Basically, the basics. I also succeed with the captcha resolver and applied the rotating user-agent, proxy and stealth for the antibot detection. Currently I am planning to add some more like webrtc/canvas fingerprinting. My main goal here is to learn more that's why I am looking for someone/dev buddy to collaborate with your projects. My only purpose is to improve/sharpen my skills when it comes to python selenium.

I would appreciate any project recommendations that could help me sharpen my skills.

Thank you!


r/SeleniumPython • • Jul 29 '24

Webdriver manager in selenium applications

1 Upvotes

It is interesting that I have been using Selenium for automaton processes for some time. For a project, I preferred to use webdriver_selenium to access the drivers for the convenience of myself and those who intend to use my program, and I expected to be able to use the driver in offline mode once the program works in online mode. But it seems that its mechanism is different. Does anyone have experience in this field to familiarize me with the mechanism of webdriver_manager? Is there a way to use its features in offline mode?


r/SeleniumPython • • Aug 20 '24

Help Needed an urgent help/suggestion towards python - selenium code.

1 Upvotes

Hi everyone, i am seeking anyone who has experience in selenium python for a code review as i am facing few errors and needed a suggestion towards my approach of test setup. DM me or comment below as well we can connect. I would really appreciate. 🥹🙏🏻


r/SeleniumPython • • Nov 22 '24

Is web crawler wrong in Brazil in this case?

2 Upvotes

I made a web crawler to get Negative Certificate which is public data but on the website there was a CAPTCHA. Have I committed any infraction? As it is public data, I understood that it would be allowed, but then I felt nervous. It is not copyright content. My fear is exposing my company to some risk.


r/SeleniumPython • • Dec 19 '24

any other alternatives to selenium wire?

1 Upvotes

i’m running a scraping tool via python that extracts network response from requests that return 403 errors. i started using selenium wire and i got it to work, but the main issue is the memory increasing more and more the longer i run it.

i’ve tried everything in order for it to not increase in memory usage, but ive had no success with it.

i’m wondering if anyone has had this problem and found a solution to access these requests without memory increasing over time. or if anyone has found another solution.

i’ve tried playwright and seleniumbase, but i didn’t have success with those.

thank you.


r/SeleniumPython • • Feb 19 '25

Google flagged my web driver

1 Upvotes

Hey guys I accidentally opened like 8 drivers and afterwards I couldn’t use any websites on it. Facebook, Google, indeed. None of them worked. I tried using a vpn and that didn’t do anything either.

What should I do to solve this.


r/SeleniumPython • • Oct 22 '24

How to Use new ChromeDriver

1 Upvotes

Howdy All, I have been maintaining code I wrote in 2020 and using it fairly successfully. Time for a major overhaul do to updated needs. I am ashamed to say that when chrome changed their driver delivery methods a year or more ago that I couldnt really get my head around it. I found a work around to do the worst, and install an older version of chrome for the moments I needed this code.

Hoping to figure out how to manage the new driver system and do a long needed overhaul of the scripts in general. Sorry I am not really even sure how to ask this question, but appreciate any help.

I am on a mac as well.

TIA.

-J


r/SeleniumPython • • Sep 08 '24

Element visibility problem when Webscraping with Selenium

2 Upvotes

Hi,

I'm a student who's writing webscraping code using Selenium on python for the first time. I have limited knowledge of the library and very basic knowledge of web components too. My aim is navigating different webpages on the platform and simulating user actions to perform the typically manual file extractions , then uploading the extracts on python for transformation (As part of an intended python ETL pipeline).

However, I noticed some extract buttons are included in drop down lists which are not always visible as HTML/CSS elements. I'm seeing that there are attributes such as aria-hidden and aria-live attributes. And I suspect javascript is involved as well.

Any advice on how to deal with the situation? (and, if the request is not against the sub rules, is someone willing provide some help or guidance in private ?)


r/SeleniumPython • • Dec 17 '24

Help Selenium IE driver

1 Upvotes

Hi, there. Does someone has a functional project with Selenium IE Driver? I tried to do a simply auto login but can't use the IE driver successfully. I followed the selenium docs, Microsoft example and other sources. Can someone give a help about how to set up de IE driver successfully for a useful implementation?


r/SeleniumPython • • Dec 07 '24

Selenium Scraping stock historical data prices with Selenium

2 Upvotes

Hi all, with Selenium I am trying to import the historical prices of AAPL (Apple) which are published on Yahoo Finance (link). But doesn't seem to be a way to do this. Yahoo apparently obfuscated the page's code to make it impossible to scrape.

If I analyze the code of the page with the Firefox Inspector I can see the DOM elements to select with a selector, such as:

selettore = driver.find_elements(By.CLASS_NAME,"yf-j5d1ld")

but this finds nothing. I guess this happens because the page code is loaded by a javascript after some time. For this I also used WebDriverWait:

selettore = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'yf-j5d1ld'))) 

But the item is not found. I don't know what else to do. I know there are libraries like "yfinance" to get that data, but I would like to be able to do it myself using Selenium. Does anyone know how this can be done or has anyone managed to import that data successfully?


r/SeleniumPython • • Aug 26 '24

Help

1 Upvotes

I need a help i am trying to do code but when i run it my browser closes instantly even for a simple code I tried time.sleep but didn’t worked as well yt tutorials but nothing worked


r/SeleniumPython • • Oct 17 '24

System specifications

1 Upvotes

What is the System specifications needed for running Selenium WebDriver in a Docker container


r/SeleniumPython • • Feb 10 '25

Any tool suggestions for test automation reports and to check logs

1 Upvotes

Using selenium and python, Gitlab for CICD, test execution reports are generating in confluence page in my project. Feeling difficult to look at the fail and pass % of tests reports in confluence and test logs in jira test plan and it's time taking for me to go to testplan and 2 testcases are running in each job. If 80 testcases in parallel execution 40 jobs will run so, logs in jira would be like 40 zip folder will get created and need to check gitlab for in which job it failed and to check the job number on zipfloder in jira to check the failure logs.

Could anyone suggest if there is any tool we can integrate and to check the exec results and logs for failure at one place. It would be helpful for me and if any tool is used and storage is a concern. If the data stored on application. Could be thankful if anyone brief on this and help with detailed information.


r/SeleniumPython • • Jan 15 '25

Shadow Root is Killing Me In The Face!

1 Upvotes

Trying to download a PDF, and just getting humiliated.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
import time


# Setup Chrome driver
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

# Replace with the actual URL of the page
driver.get("https://historyofenglishpodcast.com/wp-content/uploads/2022/01/HOE-Transcript-Episode001b.pdf")
time.sleep(5)
download = driver.execute_script('return document.querySelector("#viewer").shadowRoot.querySelector("#toolbar").shadowRoot.querySelector("#downloads")')
time.sleep(5)

download_button = download.find_element(By.CSS_SELECTOR, "#download")                                 
download_button.click()
print("Download button clicked!")
time.sleep(5)  # give it time to download or take action

driver.quit() # close browser

Any clues would be greatly appreciated

Ted


r/SeleniumPython • • Aug 15 '24

Help collecting information from website when mousing over item

1 Upvotes

I'm trying to collect information from the website: quicksell.store. When you hover over an item on the right, it displays information about that item. I'm trying to collect this information for each item using selenium, but I can't figure out how. If anyone knows how to go about doing this I would really appreciate the help.


r/SeleniumPython • • Sep 19 '24

Help Using selenium to login to reddit

1 Upvotes

Hi Guys,
Im new to webscraping and was trying to login to reddit via selenium.
Im able to enter the login details , but Im not able to select the login button to continue, I've tried using xpaths , css selectors and it looks like theres something called DOM that might interfere with the process.

iv tried using css selectors to get around it , but iv been stuck at this for a while, Any help with this would be awesome and a lifesaver!!


r/SeleniumPython • • Sep 19 '24

Send keys without specifying element Selenium webdriver

1 Upvotes

so i want to access the username and password without without specifying element and login https://practicetestautomation.com/practice-test-login/ how to do that?


r/SeleniumPython • • Sep 17 '24

Selenium uses a ton of internet data in conjunction with Google Drive upload

1 Upvotes

Hi there,

I am writing a program in Pyhon with Selenium on Mac OS that downloads .pdf files from a website and uploads the .pdfs to a Google Drive folder. The pdfs are only a few pages and average at around 300-400kb of data, and I'm downloading at most 50 .pdf files. There are .tmp.drivedownload folders that take up a ton of data in my downloads, with files inside that look like this, e.g. ".com.google.Chrome.AzphV3". These files range from 1-4gb and also populate in my Google Drive, filling up my limited 15gb of storage.

This has caused huge spikes in my internet data usage. When I started this a few days ago, I went through almost all of my data. Here is a photo of my daily usage from my Internet Provider:

When investigating further, most of my data usage is under the "Other" category. It can not be located or traced.

My code is long, but this is the function I wrote to move my .pdf from my downloads folder into my Google Drive folder:

def move_file_to_manifest_folder(manifest_dir,j):
    
    downloads_dir = '/Users/stepdoe/Downloads/'
    time.sleep(3)
    
    # Here I'm searching in my downloads folder for the last .pdf downloaded, then I  am moving that file into my Google Driver folder with os.replace 
    files = list(filter(os.path.isfile, glob.glob(downloads_dir + "*.pdf")))
    files.sort(key=lambda x: os.path.getmtime(x))

    filename = files[-1] # after I sort by time with os.path.getmtime, I take the last file in my list, which corresponds to my most recent file downloaded.
    filename = filename.split('/')
    filename = filename[-1] 
    print(f'filename[-1]: {filename}')
    filename = str(j).zfill(2) + '_' + filename # naming convension for what I want my file to be called in my Google Drive
    newpath = f'{manifest_dir}/{filename}'
    print(f'newpath: {newpath}')
    os.replace(files[-1],newpath)

I am asking for solutions to prevent these huge spikes in data download and uploads. I would expect my daily increase in usage would increase by 2-3gb (at the most 5gb), not in the order of magnitude of 100-500gb. Any help on this would be great, as my internet bill will skyrocket without it.


r/SeleniumPython • • Sep 23 '24

Stuck at timeoutexception error

1 Upvotes

Hi guys, I am trying to extract some data from a webpage. The data is on a table and there are 87 rows, some rows are not visible and require scrolling down on the table.

I have written this code:

time.sleep(60)

rows = WebDriverWait(driver, 60).until(EC.presence_of_all_elements_located((By.XPATH, "//div[@id='PSBUses']//div[contains(@class, 'x-grid3-row')]")))

I increased the waiting time to 150, didn’t work. Still receiving timeoutexception error. Does anybody have any suggestion to this? I am sooo stuck


r/SeleniumPython • • Nov 18 '24

Selenium/Python

1 Upvotes

Hello,

Is there somebody who can help me with the following:

I've created a .side file with the Selenium IDE extension within Google Chrome

Now I would like to schedule this script, so it will be performed every x period.

I'm looking into selenium/python/chromedriver, but I'm not getting there

Who can help me?

Thank you!


r/SeleniumPython • • Feb 04 '25

Python Looking for the name of a Selenium Python docker project supporting HEADFUL

1 Upvotes

Hello.

Is there a project that runs Selenium in a docker container, HEADFUL, and to which you can not only run tests but you can also connect to via a browser and "observe" the running browser tests?

I found something like this months ago, thought "hey that's damn cool but I don't need it", then today it's suddenly work relevant and of course I can't Google it back.

So the features I remembered were

  • Containered tests (or a containered service that does the work, like a farm)

  • the container must run the browser tests in a "headful" manner (NOT headless), and could record video of the browser activity

  • Supported Python.

  • I want to say this project used Selenium. But I could be incorrect on this point (that's fine, I just want to find that project again, more than caring about the tech).