Hey all - I'm going a bit crazy here and not sure why this is suddenly happening.

Background: I have a number of Python (3.12) programs which use Selenium (4.19) to perform business process tasks on external websites for which there are no APIs that I can interact with directly. Essentially, these 'bots' perform a huge amount of repetitive/data entry work which would otherwise require a LOT of human labor.

These programs have been running without issue for months. Nothing has changed in regards to the Python environment. The code specified chrome version 121, and selenium manager makes sure the proper versions of Chrome and ChromeDriver are used. All of a sudden, this past Friday evening (Oct 25, approx 6PM EDT), I started seeing every program throw an exception ("selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools") within a few moments of execution:

Traceback (most recent call last):
  File "C:\Users\automatedproc\Desktop\Compiled Bots\python\loop_so_entry\loop_status_check.py", line 134, in <module>    browser.find_element(By.CSS_SELECTOR,("input[aria-label='Filter LID']")).send_keys(str(wt["lid"][i]))
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 741, in find_element return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 347, in execute self.error_handler.check_response(response)
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
  (failed to check if window was closed: disconnected: not connected to DevTools)
  (Session info: chrome=121.0.6167.184)

The exception is thrown at different points from different individual programs, but it's always within a few moments (30-60 seconds) after the browser launches and automated control of the browser begins.

It APPEARS as if the browser is crashing, leaving no browser for the WebDriver to control. Again, this started happening suddenly this past Friday evening, with zero changes of any kind to anything on the machine this runs on.

Since then, I updated to the current version of Selenium (4.25) and have been able to get things working again, but ONLY when Selenium is controlling the installed Chrome browser (currently 130). If any version OTHER than the installed Chrome version is specified - for example 128 or 129 - then the browser crashes shortly after launch. When I specify the same version as the installed version of Chrome, then everything works without issue, as it always had.

This is fine for now, since the current version of selenium/selenium manager support Chrome 130 (the current stable release) but I'm concerned about what will happen when the browser updates itself, which is why I have always specified a specific Chrome version and had selenium manager deal with the chromedriver and chrome versions so they match.

I am NOT a developer - I'm very much an IT generalist - so perhaps this is just completely over my head and there's something simple I'm not understanding? But this behavior is happening on multiple machines (all with Python 3.12 and Selenium 4.25), one of which I just set up with a fresh install of Windows 11 earlier today.

Out of desperation, I wrote some quick minimal code to test behavior:

from selenium import webdriver
import time
import os
from datetime import datetime
import logging

baseFilePath = "C:/users/username/desktop/"
botName = "testing"
computerName = os.environ.get("COMPUTERNAME")
logFilePathAndName =  baseFilePath + datetime.now().strftime("%Y%m%d") + "-" + botName + "-" + computerName + ".txt"
logging.basicConfig(filename=logFilePathAndName,level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
logging.info("Started execution")

def initWebdriver(dlPath:str= None, chromeVer:str="129"):
    browserOpts = webdriver.ChromeOptions()
    browserOpts.browser_version = chromeVer
    browserPrefs = {"credentials_enable_service":False,"profile.password_manager_enabled": False}
    if dlPath:
        browserPrefs["download.default_directory"]= dlPath.replace("/","\\")
        browserPrefs["download.prompt_for_download"]= False
        logging.info("Setting chrome download path to: %s",dlPath.replace("/","\\"))
    browserOpts.add_experimental_option("excludeSwitches", ["enable-automation","enable-logging"])
    browserOpts.add_experimental_option("prefs", browserPrefs)
    browserOpts.add_argument("--disable-single-click-autofill")
    browserOpts.add_argument("--window-size=1300,1000")
    #browserOpts.add_argument("--disable-search-engine-choice-screen")
    #browserOpts.add_argument("--disable-gpu")
    #browserOpts.add_argument("--enable-logging")
    #browserOpts.add_argument("--v=1")
    #browserOpts.add_argument("--no-sandbox")
    browserOpts.set_capability("goog:loggingPrefs", {"performance": "ALL"})
    browser = webdriver.Chrome(options=browserOpts)
    try:
        logging.info("Chrome broswer version: %s", browser.capabilities["browserVersion"])
    except:
        pass
    return browser

try:
    browser = initWebdriver(chromeVer="129")
    while True:
        browser.get("http://apple.com")
        time.sleep(2)
        browser.get("http://google.com")
        time.sleep(2)
        browser.get("http://microsoft.com")
        time.sleep(2)
except Exception as e:
    logging.error("UNHANDLED MAIN ROUTINE occurred", exc_info=True)

Below is the entries to the logfile produced by the code above. You can see that despite this code containing just some simple "get" commands, the exception is thrown shortly after Chrome is launched. This can be reproduces on my personal machine (W11P), a W10P VM which JUST runs various 'bots', and a W11P machine I just set up with a fresh install of Win 11 earlier today - same results.

28-Oct-24 17:04:01 - INFO - Started execution
28-Oct-24 17:04:01 - DEBUG - Selenium Manager binary found at: C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\common\windows\selenium-manager.exe
28-Oct-24 17:04:01 - DEBUG - Executing process: C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\common\windows\selenium-manager.exe --browser chrome --browser-version 129 --debug --language-binding python --output json
28-Oct-24 17:04:03 - DEBUG - Sending stats to Plausible: Props { browser: "chrome", browser_version: "129", os: "windows", arch: "amd64", lang: "python", selenium_version: "4.25" }
28-Oct-24 17:04:03 - DEBUG - chromedriver not found in PATH
28-Oct-24 17:04:03 - DEBUG - chrome detected at C:\Program Files\Google\Chrome\Application\chrome.exe
28-Oct-24 17:04:03 - DEBUG - Running command: wmic datafile where name='C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe' get Version /value
28-Oct-24 17:04:03 - DEBUG - Output: "\r\r\n\r\r\nVersion=130.0.6723.70\r\r\n\r\r\n\r\r\n\r"
28-Oct-24 17:04:03 - DEBUG - Detected browser: chrome 130.0.6723.70
28-Oct-24 17:04:03 - DEBUG - Discovered chrome version (130) different to specified browser version (129)
28-Oct-24 17:04:03 - DEBUG - Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
28-Oct-24 17:04:03 - DEBUG - Required browser: chrome 129.0.6668.100
28-Oct-24 17:04:03 - DEBUG - chrome 129.0.6668.100 already exists
28-Oct-24 17:04:03 - DEBUG - chrome 129.0.6668.100 is available at C:\Users\username\.cache\selenium\chrome\win64\129.0.6668.100\chrome.exe
28-Oct-24 17:04:03 - DEBUG - Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
28-Oct-24 17:04:03 - DEBUG - Required driver: chromedriver 129.0.6668.100
28-Oct-24 17:04:03 - DEBUG - chromedriver 129.0.6668.100 already in the cache
28-Oct-24 17:04:03 - DEBUG - Driver path: C:\Users\username\.cache\selenium\chromedriver\win64\129.0.6668.100\chromedriver.exe
28-Oct-24 17:04:03 - DEBUG - Browser path: C:\Users\username\.cache\selenium\chrome\win64\129.0.6668.100\chrome.exe
28-Oct-24 17:04:03 - DEBUG - Started executable: `C:\Users\username\.cache\selenium\chromedriver\win64\129.0.6668.100\chromedriver.exe` in a child process with pid: 1980 using 0 to output -3
28-Oct-24 17:04:03 - DEBUG - POST http://localhost:57572/session {'capabilities': {'firstMatch': [{}], 'alwaysMatch': {'browserName': 'chrome', 'pageLoadStrategy': <PageLoadStrategy.normal: 'normal'>, 'browserVersion': None, 'goog:loggingPrefs': {'performance': 'ALL'}, 'goog:chromeOptions': {'excludeSwitches': ['enable-automation', 'enable-logging'], 'prefs': {'credentials_enable_service': False, 'profile.password_manager_enabled': False}, 'extensions': [], 'binary': 'C:\\Users\\username\\.cache\\selenium\\chrome\\win64\\129.0.6668.100\\chrome.exe', 'args': ['--disable-single-click-autofill', '--window-size=1300,1000']}}}}
28-Oct-24 17:04:03 - DEBUG - Starting new HTTP connection (1): localhost:57572
28-Oct-24 17:04:04 - DEBUG - http://localhost:57572 "POST /session HTTP/1.1" 200 0
28-Oct-24 17:04:04 - DEBUG - Remote response: status=200 | data={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"129.0.6668.100","chrome":{"chromedriverVersion":"129.0.6668.100 (cf58cba358d31ce285c1970a79a9411d0fb381a5-refs/branch-heads/6668@{#1704})","userDataDir":"C:\\Users\\username\\AppData\\Local\\Temp\\scoped_dir1980_1934999438"},"fedcm:accounts":true,"goog:chromeOptions":{"debuggerAddress":"localhost:57600"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"windows","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"ab1a8c304a979d2230b97c4a5621db00"}} | headers=HTTPHeaderDict({'Content-Length': '885', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:04 - DEBUG - Finished Request
28-Oct-24 17:04:04 - INFO - Chrome broswer version: 129.0.6668.100
28-Oct-24 17:04:04 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://apple.com'}
28-Oct-24 17:04:05 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:05 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:05 - DEBUG - Finished Request
28-Oct-24 17:04:07 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://google.com'}
28-Oct-24 17:04:08 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:08 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:08 - DEBUG - Finished Request
28-Oct-24 17:04:10 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://microsoft.com'}
28-Oct-24 17:04:13 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:13 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:13 - DEBUG - Finished Request
28-Oct-24 17:04:15 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://apple.com'}
28-Oct-24 17:04:16 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:16 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:16 - DEBUG - Finished Request
28-Oct-24 17:04:18 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://google.com'}
28-Oct-24 17:04:19 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:19 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:19 - DEBUG - Finished Request
28-Oct-24 17:04:21 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://microsoft.com'}
28-Oct-24 17:04:22 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:22 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:22 - DEBUG - Finished Request
28-Oct-24 17:04:24 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://apple.com'}
28-Oct-24 17:04:25 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:25 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:25 - DEBUG - Finished Request
28-Oct-24 17:04:27 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://google.com'}
28-Oct-24 17:04:27 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:27 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:27 - DEBUG - Finished Request
28-Oct-24 17:04:29 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://microsoft.com'}
28-Oct-24 17:04:31 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:31 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:31 - DEBUG - Finished Request
28-Oct-24 17:04:33 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://apple.com'}
28-Oct-24 17:04:34 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:34 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:34 - DEBUG - Finished Request
28-Oct-24 17:04:36 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://google.com'}
28-Oct-24 17:04:36 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:36 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:36 - DEBUG - Finished Request
28-Oct-24 17:04:38 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://microsoft.com'}
28-Oct-24 17:04:40 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:40 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:40 - DEBUG - Finished Request
28-Oct-24 17:04:42 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://apple.com'}
28-Oct-24 17:04:43 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:43 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:43 - DEBUG - Finished Request
28-Oct-24 17:04:45 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://google.com'}
28-Oct-24 17:04:46 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 200 0
28-Oct-24 17:04:46 - DEBUG - Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:46 - DEBUG - Finished Request
28-Oct-24 17:04:48 - DEBUG - POST http://localhost:57572/session/ab1a8c304a979d2230b97c4a5621db00/url {'url': 'http://microsoft.com'}
28-Oct-24 17:04:48 - DEBUG - http://localhost:57572 "POST /session/ab1a8c304a979d2230b97c4a5621db00/url HTTP/1.1" 500 0
28-Oct-24 17:04:48 - DEBUG - Remote response: status=500 | data={"value":{"error":"disconnected","message":"disconnected: not connected to DevTools\n  (failed to check if window was closed: disconnected: not connected to DevTools)\n  (Session info: chrome=129.0.6668.100)","stacktrace":"\tGetHandleVerifier [0x00007FF6582DB095+29557]\n\t(No symbol) [0x00007FF65824FA50]\n\t(No symbol) [0x00007FF65810B56A]\n\t(No symbol) [0x00007FF6580F2BAC]\n\t(No symbol) [0x00007FF6580F2A70]\n\t(No symbol) [0x00007FF65810DF31]\n\t(No symbol) [0x00007FF6581A7E49]\n\t(No symbol) [0x00007FF658186F33]\n\t(No symbol) [0x00007FF65815116F]\n\t(No symbol) [0x00007FF6581522D1]\n\tGetHandleVerifier [0x00007FF65860C96D+3378253]\n\tGetHandleVerifier [0x00007FF658658497+3688311]\n\tGetHandleVerifier [0x00007FF65864D1CB+3642539]\n\tGetHandleVerifier [0x00007FF65839A6B6+813462]\n\t(No symbol) [0x00007FF65825AB5F]\n\t(No symbol) [0x00007FF658256B74]\n\t(No symbol) [0x00007FF658256D10]\n\t(No symbol) [0x00007FF658245C1F]\n\tBaseThreadInitThunk [0x00007FFC1695257D+29]\n\tRtlUserThreadStart [0x00007FFC188AAF08+40]\n"}} | headers=HTTPHeaderDict({'Content-Length': '1034', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
28-Oct-24 17:04:48 - DEBUG - Finished Request
28-Oct-24 17:04:48 - ERROR - UNHANDLED MAIN ROUTINE occurred
Traceback (most recent call last):
  File "c:\Users\username\Dropbox\Company\PythonProjects\loop_so_entry\testing-browser.py", line 46, in <module>
    browser.get("http://microsoft.com")
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 363, in get
    self.execute(Command.GET, {"url": url})
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 354, in execute
    self.error_handler.check_response(response)
  File "C:\Program Files\Python312\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
  (failed to check if window was closed: disconnected: not connected to DevTools)
  (Session info: chrome=129.0.6668.100)
Stacktrace:
GetHandleVerifier [0x00007FF6582DB095+29557]
(No symbol) [0x00007FF65824FA50]
(No symbol) [0x00007FF65810B56A]
(No symbol) [0x00007FF6580F2BAC]
(No symbol) [0x00007FF6580F2A70]
(No symbol) [0x00007FF65810DF31]
(No symbol) [0x00007FF6581A7E49]
(No symbol) [0x00007FF658186F33]
(No symbol) [0x00007FF65815116F]
(No symbol) [0x00007FF6581522D1]
GetHandleVerifier [0x00007FF65860C96D+3378253]
GetHandleVerifier [0x00007FF658658497+3688311]
GetHandleVerifier [0x00007FF65864D1CB+3642539]
GetHandleVerifier [0x00007FF65839A6B6+813462]
(No symbol) [0x00007FF65825AB5F]
(No symbol) [0x00007FF658256B74]
(No symbol) [0x00007FF658256D10]
(No symbol) [0x00007FF658245C1F]
BaseThreadInitThunk [0x00007FFC1695257D+29]
RtlUserThreadStart [0x00007FFC188AAF08+40]

I'm sure there's a lot of details I haven't addressed, so if there's any questions, please let me know. My brain is sort of fried after spending the day Googling and trying all sorts of different things... Help?!?

I've some python code that scrapes data from the web using selenium with chrome webdriver. This worked nicely till 2 months or so ago. Suddenly it started slowing down my computer in the extreme. The problem occurs with different projects, so it doesn't seem to be the code as the code of some of these projects has not changed in this time period. When I look at in windows task manager it shows that the CPU at 100% and a huge number chrome background processes.

Anybody know what may cause this sudden change and what may be done to solve it?

Hello everyone , I am using Selenium Python to scrape a website https://www.whed.net/results_institutions.php
and This website contains data for every country(list of institutions) and one need to click on link of every instituion and scrape name , location and www associated with that institution.
Now I have tried to use selenium for automating the task and I am doing mostly fine other than being unable to close the dialog box.

This is my sample code. Can somebody Explain me how to do it.

service = Service("C:/Selenium_drivers/chromedriver-win64/chromedriver.exe")

driver = webdriver.Chrome(service=service)

driver.get(url)

country = 'Afghanistan'

institues = []
cities = []
wwws = []

drop_down = Select(driver.find_element(By.XPATH, '//select'))
drop_down.select_by_visible_text(country)

all_institute = driver.find_element(By.XPATH, "//input[@id='membre2']")
if not all_institute.is_selected():
    all_institute.click()
    
button = driver.find_element(By.XPATH, "//input[@type='button']")

button.click()


results_per_page = Select(driver.find_element(By.XPATH, "//select[@name='nbr_ref_pge']"))
results_per_page.select_by_visible_text('100')


total_results = int(driver.find_element(By.XPATH, "//p[@class='infos']").text.split()[0])

max_iter = total_results//100 + 1
iterations = 0

go_on = True

while go_on:
    iterations += 1
    
    institutions = driver.find_elements(By.XPATH, "//li[contains(@class, 'clearfix plus')]")
    
    
    for institue in institutions:

            link = institute.find_element(By.XPATH, ".//h3/a")
            link.click()
            
            time.sleep(2)
            
            pop_up = driver.find_element(By.XPATH, "//iframe[starts-with(@id, 'fancybox-frame')]")
            
            driver.switch_to_frame(pop_up)

    #             main_window = driver.current_window_handle  # Store the handle of the main window
    #             popup_window = None

    #             for window_handle in driver.window_handles:
    #                 if window_handle != main_window:
    #                     popup_window = window_handle

            # Switch to the popup window
    #             driver.switch_to.window(popup_window)

            institue = driver.find_element(By.XPATH, "//div[@class='detail_right']/div[1]").text

            city = driver.find_element(By.XPATH, "//span[@class='libelle' and text() = 'City:']/following-sibling::span[@class='contenu']").text

            www = driver.find_element(By.XPATH, "//span[@class='libelle' and text() = 'WWW:']/following-sibling::span[@class='contenu']").get_attribute("title")

            institues.append(institute)

            cities.append(city)

            wwws.append(www)

            close_button = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@title='Close']")))
            close_button.click()

#             driver.switch_to.window(main_window)

#             driver.switch_to.window(main_window)

    if iterations >= max_iter:
        go_on =False
        break
        
    time.sleep(2)
    
    next_page = driver.find_elements(By.XPATH, "//a[@title='Next page' ]")[0]
    next_page.click()

Does anybody know how to get rid of these?

Hello everyone. I’m having an issue with simple code with selenium-python.

import time from selenium import webdriver from selenium.webdriver.common.by import By driver = webdrvier.Chrome()

And that’s it, an empty chrome page automatically opens and in terminal there is error message: USB: usb_device_handle_win_cc 1045 Failed to read descriptor from node connection: A device attached to the system is not functioning.

Please help me out to solve the issue. 🙏🏻🙏🏻🙏🏻 OS windows 11. I’m new to selenium as well.

I'm attempting to modify this projecthttps://github.com/oborys/Selenium_automation_Adding_Cisco_Meraki_VPN_exclusion_rules to the current Meraki dashboard setup. I'm running into some issues with the 'add' button on the 'Local internet breakout' box.

I've tried several methods for the locator with various errors.

time.sleep(3)

#button = driver.find_element(By.TAG_NAME, "//a[@class='btn btn-default']")

#button = driver.find_element(By.TAG_NAME, "fa fa-plus plus")

#button = driver.find_element(By.TAG_NAME, 'add')

#button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "btn btn-default")))

#button = WebDriverWait(driver, 10).until(By.CSS_SELECTOR, "a[class='btn btn-default']")

button = WebDriverWait(driver, 10).until(By.XPATH, "//a[@class='btn btn-default']")

time.sleep(1)
button.click()

Using selectors hub I've confirmed the XPATH is //a[@class='btn btn-default'] but when I debug I get the error'str' object is not callable. I've appended this to the below line but it never actually clicks it.

button = driver.find_element(By.XPATH, "//a[@class='btn btn-default']").click()

I haven't been able to locate the actual ID to try that. Inspecting the element I see that the below section contains the "Add"<a class="btn btn-default" data-toggle="dropdown" data-domplate="href" href="#rule\\\\\\\\\\\\\\_menu\\\\\\\\\\\\\\_vpn\\\\\\\\\\\\\\_exclusion\\\\\\\\\\\\\\_shaper">Add<i class="fa fa-plus plus"></i></a>

If I try by tag name I get the errorMessage: no such element: Unable to locate element: {"method":"tag name","selector":"//a[@class='btn btn-default']"}

Message: no such element: Unable to locate element: {"method":"tag name","selector":"fa fa-plus plus"}

Message: no such element: Unable to locate element: {"method":"tag name","selector":"add"}

If there is a better method for this I'm open to it but I didn't have much luck using autogui and I'd like this to be able to work from multiple pcs so using selenium would be preferred.

I'm currently working on a project that requires automating the solving of reCAPTCHA dialogs. I need to solve a large number of them quickly and concurrently.

Initially, I used a script that converted the audio CAPTCHA into text, but it had several drawbacks: it was slow and could only handle one CAPTCHA at a time. Since my project requires concurrency, this approach was inadequate.

I later switched to using a browser extension called NopeCHA, which resolved the concurrency issue. However, it introduces a new problem—the extension needs to be pre-configured, which is cumbersome and not scalable.

I'm now looking for a more efficient solution—one that can solve reCAPTCHA dialogs rapidly and concurrently, without requiring manual setup or browser-based extensions.

Has anyone developed a method or tool that addresses this?

Hello everyone,

Background: former software engineer with 2 years and 5 months of experienced.

I am from Philippines and currently unemployed. So, what I actually do in my free time is to upskill/learn something new. I've been studying python selenium for the past 2 weeks. I know the waits, locators, xpaths, find_element and etc. Basically, the basics. I also succeed with the captcha resolver and applied the rotating user-agent, proxy and stealth for the antibot detection. Currently I am planning to add some more like webrtc/canvas fingerprinting. My main goal here is to learn more that's why I am looking for someone/dev buddy to collaborate with your projects. My only purpose is to improve/sharpen my skills when it comes to python selenium.

I would appreciate any project recommendations that could help me sharpen my skills.

Thank you!

I’m trying to run the “chromedriver” command on python (selenium). I have the chrome.exe in a folder that I can successfully path on Python, but I can’t fire the “chromedriver” command. I have it blocked as it gets considered suspicious/malware and can’t find a way to disable the block on it. Hopefully someone can help me ✌🏼

Hey everyone,

Trying to build a project and want to click on the Instagram post then collect the username and put it into a csv. Any insights on how I can do that ?

🚀 Exciting News! 🚀 The wait is over! BrowserBruter is now public and available for download, the world's first advanced browser-based automated web application penetration testing tool!

After in development for over a year, it is now officially released!

👉 Proof Of Concept - https://net-square.com/browserbruter/WhyWeNeedBrowserBruter/ 👉 Live Demonstration - https://youtube.com/playlist?list=PL1qH_bg_l1aMNDpCYSMXg83o-56vLdPS7&si=LtQxvbLDKWhiCsEC 📖 Explore the documentation: https://net-square.com/browserbruter/ 📥 Download now: https://github.com/netsquare/BrowserBruter/releases/tag/v2024.4-BrowserBruter

📈 BrowserBruter revolutionizes web application security testing by attacking web applications through controlled browsers, injecting malicious payloads into input fields. It automates the process of sending payloads to web application input fields in the browser and sending them to the server.

Highlighted Features:

• 🔐 Bypass Encrypted HTTP Traffic: Fuzz web application forms even when the HTTP body is encrypted, because it will fuzz web application before encryption takes place.

• 🤖 Bypass Captchas: Allows the pentester to manually perform human interactions to bypass captchas and proceed with payload insertions.

• 🖥️ Fuzz Front-Ends without HTTP Traffic: Can fuzz front-end elements even when there is no HTTP traffic.

• 🔗 Simplified Session Management: Removes the burden of session management, CSRF handling, and other micro-management tasks when using HTTP proxy tools, because these are managed by browser it self which is controlled by Browser Bruter.

📗 After fuzzing, BrowserBruter generates a comprehensive report that includes all the data and results of the penetration test, along with HTTP traffic. This report can be viewed using The Report Explorer tool, which comes bundled with BrowserBruter.

Handcrafted in India 🇮🇳

Behind the Scenes: The Backstory of BrowserBruter

🥷 As a penetration tester working on web application security VAPT projects, I faced a common challenge: the encryption of HTTP traffic was hindering my ability to fuzz input fields using traditional tools.

⚙️ Available tools like BurpSuite, SQLMap, etc. operate by modifying HTTP requests and responses. However, when encryption is implemented (not ssl, when the http request body's data is encrypted), the HTTP traffic becomes opaque to these tools, making it impossible to inject payloads into the web application's input fields.

💡 This limitation sparked an innovative idea: what if we could bypass the encryption and fuzz the web application at the browser layer instead of the HTTP layer? This approach would allow us to interact with the web application as if we were a user, bypassing the need to break the encryption of HTTP traffic.

The result? BrowserBruter, the world's first advanced browser-based automated web application penetration testing tool! By controlling browsers and injecting payloads into input fields, BrowserBruter bypasses encryption and automates the process of sending payloads to web application input fields in the browser.

This project is licensed under the GNU General Public License v3.0

For some reason, Selenium is unable to scrape web pages from some domain, there must be some server side filtering going on, that prevents a simulated browser from accessing specific domains. Did anyone ever have a similar issue? How did you resolve?

Hello, Reddit community,

I am new to learning selenium automation with Python, If someone is doing any project, I am willing to help them without any fee or money involved I want to learn and excel in automation.

Please reach out to me and provide me with your contact details I will contact you ASAP

I'm currently facing a challenge with deploying my Flask server application, which utilizes Selenium for web scraping, into a production environment. While I've successfully implemented Selenium for local development using ChromeWebDriver, I'm unsure about the best practices for Dockerizing it and deploying it in a production setting.

Here's a bit of background: I've built a Flask server that scrapes data from X's tweets ( formerly known as twitter ) using Selenium. However, as I prepare to deploy this application into production, I realize that I need guidance on how to effectively containerize it with Docker and manage the Selenium instances.

1. How can I Dockerize my Flask application along with Selenium dependencies to ensure seamless deployment in a production environment?

2. What are the best practices for managing Selenium instances within Docker containers?

3. Are there any specific configurations or optimizations I should consider for running Selenium in a production environment?

I'd appreciate any resources, blogs, or YouTube videos that provide insights into running Selenium in production environments with Flask servers. Whether it's documentation, tutorials, or personal experiences, any guidance would be helpful

from flask import Flask, render_template, request, jsonify
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import json
from time import sleep

app = Flask(__name__)


# Set the path to your chromedriver executable
chromedriver_path = '/path/to/chromedriver'

@app.route('/scrape', methods=['POST'])
def scrape():
    if request.method == 'POST':
        url = request.json.get('url')
        browser = webdriver.Chrome()

        # Initialize Chrome driver  
        try:
            # Navigate to the provided URL
            browser.get(url)
            # print(1)
            # Extract title and any other data you need
            tweet =  browser.find_element(By.XPATH, '//article[@data-testid="tweet"]')
            # print(2)
            element = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.CLASS_NAME,'css-9pa8cd')))
            img = tweet.find_element(By.XPATH,'//img[@class="css-9pa8cd"]').get_attribute("src")
            # print(3)
            # print(img,"sujal")
            # print(4)
            # print(5)
            user_name_container = tweet.find_element(By.XPATH, '//a[@class="css-175oi2r r-1wbh5a2 r-dnmrzs r-1ny4l3l r-1loqt21"]')
            # print(6)
            user_name = user_name_container.get_attribute("href")[20:]
            # print(8)
            name_container = tweet.find_elements(By.XPATH, '//span[@class="css-1qaijid r-bcqeeo r-qvutc0 r-poiln3"]')
            name = name_container[6].text
            tweet_body = name_container[8].text
            time = tweet.find_element(By.TAG_NAME, 'time').text
            
            # print(time)
            # print(user_name)
            # Add more scraping logic as needed

            

            # Return the scraped data as JSON
            return jsonify({
                'user_name':user_name,
                'name':name,
                'tweet_body':tweet_body,
                'time':time,
                'img':img     
                })
        except Exception as e:
            # Handle any errors that may occur during scraping
            return jsonify({'error': str(e)})
        finally:
            # Make sure to close the driver even if an exception occurs
            pass

if __name__ == '__main__':
    app.run(debug=True)

I can find innumerable tutorials; I don't want those.

I want to go to a web-based API reference for the Python selenium module (or whatever passes for that), where I can lookup, e.g., webdriver.Remote() and see every possible argument that can be passed to that method.

Can anyone point me in the correct direction?

Many thanks!

Why when i login in my instagram with selenium the "Not Now' button doesn't work

error : selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:{"method":"xpath","selector":"//*[@id="mount_0_0_LY"]/div/div/div[2]/div/div/div/div[1]/div[1]/div[1]/div/']

Ever wondered about the mystery of how some websites hide their content behind scripts, JSON, or paywalls?

This is one of my Own Experience
I've noticed that some sites don’t render the full article directly in the DOM. Often, the content is embedded in JSON inside <script> tags — meant for search engines or social

Here’s a small demo using Python Selenium that extracts article content from structured data:
What’s happening here:

1.We check if a paywall element exists.

2.Then we grab the JSON that contains the full article content.

3.Finally, we render it nicely in the browser console or page.

Sometimes the full article is sitting right there, quietly waiting to be discovered … if you know where to look .

Hello everyone, I need some help because I'm trying to run selenium in headless mode on my proxmox server but it's not working. Here are the specifics:

- i set up a proxmox container with ubuntu-24.10-standard_24.10-1_amd64.tar.zst

- i created inside a venv for python, installed selenium and the necessary packages and chrome.

- I created a test_selenium.py script like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless=new")  # use new headless mode if available
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("window-size=1920x1080")

# Install Chromedriver and initialize the service
driver_path = ChromeDriverManager().install()
print(f"Chromedriver path: {driver_path}")
service = Service(driver_path)

# Create the webdriver instance
driver = webdriver.Chrome(service=service, options=chrome_options)

# Navigate to a webpage
driver.get("https://www.example.com")
print("Page Title:", driver.title)

# Close the browser
driver.quit()

Now when I run the script I obtain:

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6395a5333ffa <unknown>
#1 0x6395a4df2970 <unknown>
#2 0x6395a4e318e8 <unknown>
#3 0x6395a4e2cdca <unknown>
#4 0x6395a4e2818f <unknown>
#5 0x6395a4e78bd9 <unknown>
#6 0x6395a4e78106 <unknown>
#7 0x6395a4e6a063 <unknown>
...#19 0x7565c6b33a4c <unknown>

Then I added to the code:

chrome_options.add_argument("--remote-debugging-port=9222")

And I obtain the error:

selenium.common.exceptions.SessionNotCreatedException: Message: session not created from chrome not reachable

What am I doing wrong? can someone help me?

I can't get to match the versions of chromedriver and chrome browser

last version of chromedriver is .88

last version of google chrome is .89 ( it updated automatically so it broke my script)

yes, google provide older versions of chrome, but doesnt give me an install file, it gives me a zip with several files ( as if it were installed, sort of- sorry, im newbie) , and I dont know what to do with that

could someone help ? thanks!

edit: IDK what I did, it just started working. After that, it broke again and mismatched the versions.

then, deleting C:\Users\MyUser\.wdm FIXED IT

When i start chrome with selenium python I cannot solve friendlycaptcha. in the demo page https://friendlycaptcha.com/#demo "Verification failed: Browser check failed, try a different browser".

It's used in a webpage login I need to access; I'm fine with logging manually, but I need to overcome this verification.

Hi all, with Selenium I am trying to import the historical prices of AAPL (Apple) which are published on Yahoo Finance (link). But doesn't seem to be a way to do this. Yahoo apparently obfuscated the page's code to make it impossible to scrape.

If I analyze the code of the page with the Firefox Inspector I can see the DOM elements to select with a selector, such as:

selettore = driver.find_elements(By.CLASS_NAME,"yf-j5d1ld")

but this finds nothing. I guess this happens because the page code is loaded by a javascript after some time. For this I also used WebDriverWait:

selettore = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'yf-j5d1ld'))) 

But the item is not found. I don't know what else to do. I know there are libraries like "yfinance" to get that data, but I would like to be able to do it myself using Selenium. Does anyone know how this can be done or has anyone managed to import that data successfully?