r/SeleniumPython • • Jun 22 '24

How to locate web elements for pyautogui using selenium.

2 Upvotes

I am using element.locate & element .size to get location & size of element i want to click using pyautogui. Now problem is selenium gives coordinates in webpage whereas pyautogui calculate that coordinate on monitor's screen. So both coordinate are mismaching. If I find coordnate manually by . position () method in pyautogui than it works. But I first want to locate element via selenium and want to pass location of that element in pyautogui. Please help.


r/SeleniumPython • • Mar 30 '23

Working with IE on browserstack ( Python + Selenium)

1 Upvotes

Trying to automate a usecase in Browserstack with selenium and python
The below error was seen when trying to click on elements
WebDriverException: Message: actions

Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:13:22.693Z'

System info: host: '150-129-0-169', ip: '150.129.0.169', os.name: 'windows', os.arch: 'x86', os.version: '10.0', java.version: '1.8.0_181'

Driver info: driver.version: unknown

Stacktrace:

at org.openqa.selenium.remote.http.AbstractHttpCommandCodec.encode (AbstractHttpCommandCodec.java:218)

at org.openqa.selenium.remote.http.AbstractHttpCommandCodec.encode (AbstractHttpCommandCodec.java:117)

at org.openqa.selenium.remote.server.ProtocolConverter.handle (ProtocolConverter.java:83)

at org.openqa.selenium.remote.server.RemoteSession.execute (RemoteSession.java:127)

at org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$3 (WebDriverServlet.java:250)

at java.util.concurrent.Executors$RunnableAdapter.call (None:-1)

at java.util.concurrent.FutureTask.run (None:-1)

at java.util.concurrent.ThreadPoolExecutor.runWorker (None:-1)

at java.util.concurrent.ThreadPoolExecutor$Worker.run (None:-1)

at java.lang.Thread.run (None:-1)

Upon further search
Found the below guide
https://www.browserstack.com/guide/setproperty-in-selenium
// Defining System Property for the IEDriver

System.setProperty("webdriver.ie.driver", "D:IE Driver ServerIEDriverServer.exe");

// Instantiate a IEDriver class.

WebDriver driver=new InternetExplorerDriver();
However, unable to find an equivalent to System.setProperty in Python. Tried the below options

https://learn.microsoft.com/en-us/microsoft-edge/webdriver-chromium/ie-mode?tabs=python

desired_cap = webdriver.IeOptions()
desired_cap.attach_to_edge_chrome = True
desired_cap.edge_executable_path = "IEDriverServer.exe"
driver = webdriver.Remote(command_executor=self.BROWSERSTACK_URL, desired_capabilities=desired_cap)

{'browserName': 'IE', 'platformName': 'windows', 'pageLoadStrategy': 'normal', 'bstack:options': {'os': 'Windows', 'osVersion': '10',......, browserVersion': 'latest', 'se:ieOptions': {'ie.edgechromium': True, 'ie.edgepath': 'IEDriverServer.exe'}}

Initialized browserstack web driver for IE

The same error still persists.
Could someone please share their insights on how to work with IE


r/SeleniumPython • • Jun 13 '24

Selenium Disable/Overwrite Javascript window.close function

1 Upvotes

Currently doing some scraping with Selenium and I am struggling with a page that seems to be fighting me:

  • The page opens a link in a new tab

  • I want to use driver.window_handles to iterate through the tabs to find the one I am looking for. However, this is somehow detected by the page and it closes the tab that I want to scrape as soon as I call window_handles :(

  • So I tried to disable the JavaScript closing function, using driver.execute_script() with window.close = function() { }; and close = function() { };. But this does not suppress the closing of the tab :(

I am really surprised as to what is going on here and how to bypass this.

Using selenium~=4.16.0. Also tried with undetected-chromedriver~=3.5.5.

Edit: just realized that disabling of the close function is of course not working. The call to close is in the new window/tab and I don't think I can override it from the old tab :(


r/SeleniumPython • • Apr 29 '24

Selenium iteration issue

2 Upvotes

Hey Everyone, I wrote a program that iterates through each airbnb listing, gets data and moves on the next listing. The problem is it just loops one listings. any ideas?

def fetch_images_from_carousel(driver):
images = set()
print("Fetching images from Carousel:")
try:
print("Loading images")
WebDriverWait(driver, 10).until(
EC.visibility_of_all_elements_located((By.CSS_SELECTOR,
".itu7ddv.atm_e2_idpfg4.atm_vy_idpfg4.atm_mk_stnw88.atm_e2_1osqo2v__1lzdix4.atm_vy_1osqo2v__1lzdix4.i1cqnm0r.atm_jp_pyzg9w.atm_jr_nyqth1.i1de1kle.atm_vh_yfq0k3.dir.dir-ltr"))
)
initial_images = driver.find_elements(By.CSS_SELECTOR,
".itu7ddv.atm_e2_idpfg4.atm_vy_idpfg4.atm_mk_stnw88.atm_e2_1osqo2v__1lzdix4.atm_vy_1osqo2v__1lzdix4.i1cqnm0r.atm_jp_pyzg9w.atm_jr_nyqth1.i1de1kle.atm_vh_yfq0k3.dir.dir-ltr")
for img in initial_images:
src = img.get_attribute('src')
if src and isinstance(src, str) and src not in images: # Ensure src is a string and not previously added
images.add(src)
print("Added image:", src)

# Handle the carousel navigation using JavaScript to click the button
try:
while True:
next_button = driver.find_element(By.CSS_SELECTOR, ".ms83rji.atm_fg_esu3gu.dir.dir-ltr")
driver.execute_script("arguments[0].click();", next_button) # Click the button using JavaScript
time.sleep(3) # Adjust the delay to allow the next set of images to load (e.g., 3 seconds)
new_images = fetch_current_visible_images(driver)
print(f"Fetched {len(new_images)} new images after clicking the button.")
print("New Images:", new_images)
if set(new_images) == images:
logging.info("No new images loaded. Exiting carousel navigation.")
break
except TimeoutException:
logging.info("Timed out waiting for carousel elements.")
except Exception as e:
logging.error(f"Error navigating carousel: {str(e)}")

return images

heres a portion of the code that fails. The main is

def scrape_airbnb(url, state_name):
table_name = f"{state_name.replace(' ', '_')}"
print("Starting Scrape For: ", table_name)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
try:
driver.get(url)

# Find listings one by one to avoid stale element references
while True:
listings = WebDriverWait(driver, 10).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".dir.dir-ltr"))
)

for listing in listings:
try:
# Perform scraping operations for each listing
images = fetch_images_from_carousel(driver)
price = listing.find_element(By.CSS_SELECTOR, "._1jo4hgw").text
beds = listing.find_element(By.CSS_SELECTOR,
".a8jt5op.atm_3f_idpfg4.atm_7h_hxbz6r.atm_7i_ysn8ba.atm_e2_t94yts"
".atm_ks_zryt35.atm_l8_idpfg4.atm_mk_stnw88.atm_vv_1q9ccgz.atm_vy_t94yts.dir"
".dir-ltr").text
ratings = listing.find_element(By.CSS_SELECTOR,
".t1a9j9y7.atm_da_1ko3t4y.atm_dm_kb7nvz.atm_fg_h9n0ih.dir.dir-ltr").text

# Click on the listing to view more details if needed
listing.click()

# Wait for the new page to load
WebDriverWait(driver, 10).until(EC.url_changes(url))

listing_id = driver.current_url.split('/')[-1] # Extract listing ID from URL
insert_or_update_listing_data(table_name, listing_id, images, price, beds, ratings)

# Go back to the previous page to continue scraping other listings
driver.back()
except NoSuchElementException:
continue
except StaleElementReferenceException:
continue
# Check if there's a next page
try:
next_page_button = driver.find_element(By.CSS_SELECTOR, "._1u3e89e")
next_page_button.click()
except NoSuchElementException:
break
except Exception as e:
logging.error(f"Error processing a listing: {e}")

finally:
driver.quit()


r/SeleniumPython • • Dec 29 '22

Selenium problems

1 Upvotes

I need help... I am using a python program to send parameters to the Spain e-administration site to get an appointment for a procedure. But the page is blocking my session: The requested URL was rejected. Please consult with your administrator.

Your support ID is: <1417176221212705327608>. .

I was reading some solutions, and I saw that using sleep.time() I could simulate the writing of a human so that the page doesn't crash. But it didn't work. Can anyone think of something else?


r/SeleniumPython • • Jan 24 '24

ActionChains send_keys can’t handle emojis

2 Upvotes

I made a bot that logs into my facebook daily, checks to see whose birthday it is via the events page, and submits a happy birthday message via the form field. Facebook seems like it’s always changing, so this bot has changed a lot over the years. Most recently, once the birthday form fields are identified by XPATH, an exception occurs indicating that send_keys can’t type inside the form (can’t interact with the element). I eventually got this to work with ActionChains, but the send_keys method it uses can’t seem to handle emojis. Is there another way to click inside the form and send the keys?


r/SeleniumPython • • Apr 12 '24

Help Unable to play this video in Selenium, works in regular browser

1 Upvotes

Unable to play this video in Selenium, works in regular browser

https://hd2watch .tv/watch-series/greys-anatomy-hd-39545/1428686

player is just all black


r/SeleniumPython • • Jul 15 '23

Firefox browser closes automatically without specifying close() and quit() methods in the automation script. How to fix this issue in selenium python...

1 Upvotes

r/SeleniumPython • • Jul 27 '23

How to click the like button on a YouTube video. I understand that I can do it if I Know the name of the button, but I can not find it in the HTML. Some one know how to find the name or tag of the button?

1 Upvotes

r/SeleniumPython • • Aug 11 '23

Help An issue in running Automated test scripts with Python through PyCharm on Apple MacBook Pro M2 (silicon) in the latest Google Chrome Browser.

1 Upvotes

On macOS Ventura (version 13.4.1) with an M2 chip with amr64 architecture, I`m trying to run tests by using PyCharm 2023 Community edition, with selenium robot framework 6.11, Python 3.11, and up to date web driver manager installed with Chrome browser (version 15.0.5790.170 (Official Build) (arm64) ) and compatible version of chrome web driver(amr64).
After running a test getting the following error :
OSError: [Errno 86] Bad CPU type in executable: '/Users/igormelnyk/PycharmProjects/python/selenium_mini_course/venv/lib/python3.11/site-packages/selenium/webdriver/common/macos/selenium-manager'
Can anybody Clarify? Thanks.
Spent about 3 hours navigating GPT, checked all versions for compatibility, Official source for Chromium and Selenium, retyped scripts in Python, reactivated the virtual environment, pip all the libraires and....... Still nothing....


r/SeleniumPython • • Jul 13 '23

Xpath Advantageous Landscape of Integrating IoT and ERP

Thumbnail
testorigen.com
1 Upvotes

r/SeleniumPython • • Mar 20 '24

Script stops...

1 Upvotes

Hello everyone, I'm new to selenium and also not so deep into the coding at all so I'm quite happy to got so far :D

So I want to automate a Process where I need to klick a Button, this Button get's only active after a few seconds. This is because of a script that got implemented to see if everything is okay to proceed. I placed the script at the End here.

My Problem is now, that I can't click the button with selenium.

I tried these options:

and none of them worked, I think it's because the element is part of the script and not a normal button in the source code. Does anyone here have an idea how to solve this problem?

<div class="navbar navbar-expand-md navbar-light action-bar"> <div id="action-toolbar" name="action-toolbar"> </div><script>kendo.syncReady(function(){jQuery("#action-toolbar").kendoToolBar({"items":[{"enable":false,"id":"save-button","text":"\u003ci class=\u0027icon-floppy-disk\u0027\u003e\u003c/i\u003eSpeichern","type":"button","attributes":{"class":"k-primary save-button","data-activewhendirty":"","role":"button","aria-disabled":"true","disabled":"disabled","tabindex":"-1"}},{"enable":false,"id":"reset-button","text":"\u003ci class=\u0027icon-undo\u0027\u003e\u003c/i\u003eVerwerfen","type":"button","attributes":{"class":"reset-button","data-activeif":"ContractChangeDetails.resettable","role":"button","aria-disabled":"true","disabled":"disabled","tabindex":"-1"}},{"enable":false,"id":"totalValueCalculationButton","text":"\u003ci class=\u0027icon-calculator3\u0027\u003e\u003c/i\u003eGesamtwertberechnung","type":"button","attributes":{"data-id":"1008429","role":"button","data-activeif":"ContractChangeDetails.calculatable","disabled":"disabled"},"click":ContractChangeDetails.totalValueCalculationClickHandler},{"enable":false,"id":"setContractPrice","text":"\u003ci class=\u0027icon-coin-euro\u0027\u003e\u003c/i\u003eVertragspreis setzen","type":"button","attributes":{"data-id":"1008429","role":"button","data-activeif":"ContractChangeDetails.calculated","disabled":"disabled"},"click":ContractChangeDetails.setContractPriceClickHandler},{"enable":false,"id":"transferToContract","text":"\u003ci class=\u0027icon-add-to-list\u0027\u003e\u003c/i\u003eSofort in den Vertrag überführen","type":"button","attributes":{"data-id":"1008429","data-activeif":"ContractChangeDetails.transferable","role":"button","disabled":"disabled"},"click":ContractChangeDetails.transferToContractClickHandler},{"enable":false,"id":"discardContractChange","text":"\u003cspan class=\u0027k-icon k-i-x-outline\u0027\u003e\u003c/span\u003eVertragsänderung verwerfen","type":"button","attributes":{"data-id":"1008429","data-activeif":"ContractChangeDetails.discardable","role":"button","disabled":"disabled"},"click":ContractChangeDetails.discardClickHandler},{"enable":false,"id":"createKonkalProcess","text":"\u003ci class=\u0027icon-link\u0027\u003e\u003c/i\u003eVorgang anlegen","type":"button","attributes":{"data-id":"1008429","data-activeif":"ContractChangeDetails.createProcessIsAllowed","role":"button","disabled":"disabled"},"click":ContractChangeDetails.konkalCreateProcessClickHandler},{"enable":true,"id":"export-button","text":"\u003ci class=\u0027icon-download\u0027\u003e\u003c/i\u003eExportieren \u003ci class=\u0027icon-arrow-down22\u0027\u003e\u003c/i\u003e","type":"button","attributes":{"data-activeif":"ContractChangeDetails.exportable","role":"button"}},{"enable":false,"id":"import-button","template":jQuery('#fileUploadDialogTemplate').html(),"text":"File","type":"button","attributes":{"data-id":"1008429","data-contextmenu":"false"}}]});});</script> </div>

P.s.: it's an intranet website, so I can't share the link with you.


r/SeleniumPython • • Aug 17 '23

Help Using chrome 115 but no chromedriver 115

1 Upvotes

How do I instruct selenium to use the beta version of chrome? (not chromedriver - it is finding that - but chrome itself). This is using selenium and python on linux (ubuntu 22.04).

I'm trying to use chrome 116 (which is /bin/google-chrome-beta on my system). I've already installed chromedriver 116, but when it launches it still attempts to drive the older version of chrome (115), not the beta version of chrome (116), which only results in a fatal error indicating that chromedriver 116 cannot be used with chrome 115.

I tried to replace /bin/google-chrome with /bin/google-chrome-beta but that breaks it completely.

There's a selenium chrome option for path to the executable, but that is apparently deprecated. (I didn't try it)


r/SeleniumPython • • Aug 04 '23

How to click on chrome’s toolbar buttons?

1 Upvotes

Big goal is I would like to make program that able to scrape the URL of the websites from my read later list on chrome.

I figure the first step is to get selenium to click on the 3 dot menu button on chromes toolbar. But this is part of the browser and not the webpage and when I try to right click to inspect it for its elements nothing happens.

How do I get selenium to interact with the browsers toolbar buttons?


r/SeleniumPython • • Nov 15 '23

How do I count the number of times a looping Selenium script has been run in the Script itself?

2 Upvotes

As the title says guys...

Im looping my Selenium script 200 times with the cmd "times".

Is there another command I can place at the end of my code that will count and show me live how often the script has looped already?

Thanks for your help!


r/SeleniumPython • • Mar 17 '24

Selenium Need help running code

1 Upvotes

Hey, I’m completely new to programming and wanted some help on running this code:

https://github.com/LordFruitSnacks/SatCenterSearch

This tool would help greatly as I am a highschooler that needs to take the SAT in June but since all the spots near me were full I keep checking the website everyday to find test centers that might’ve opened up but even when they do, they fill up too quickly for me to sign up so this tool helps check for open ones so I don’t have to stay on the website 24/7. Any help or redirects to somewhere I can ask for help would be awesome, thanks!


r/SeleniumPython • • Apr 25 '23

Best Platform to Learn Python Programming Language?

Thumbnail self.pythontips
1 Upvotes

r/SeleniumPython • • Jun 19 '23

Microsoft Encryption

1 Upvotes

Hello guys,

I am trying to automate a process, with a excel file sended with Microsoft Encryption. At end of process after a I check my account. Show up a page without XPath, and a just need click on a button. Somebody know how to deal with pages with autoid?


r/SeleniumPython • • Aug 23 '23

Help Selenium not working anymore

1 Upvotes

Selenium (or chrome webdriver) stopped working.

At first it was version error so I upgraded selenium, updated chrome Version 116.0.5845.111, and got a new webdriver Version 116.0.5845.96. versions are not exact, is that a problem? there is no webdriver with the same version otherwise.

Also the error went away but I think there is a problem with finding the path since it never gets passed

driver = webdriver.Chrome(path)

Assuming im getting the path correct, are there any concerns I should have?

Thanks in advanced,


r/SeleniumPython • • Feb 15 '23

Scraping interval Question

2 Upvotes

I'm looking to scrape a website once an hour just for basic data. Is this too often that could trigger a captcha or would it be case by case depending on the website?


r/SeleniumPython • • Apr 30 '24

Selenium Help opening new window / tab from clicking a form submit

1 Upvotes

Hi, hoping someone can help me as I just spent over and hour bashing my dead on my desk.

The search_button below works it's finding the proper button I need on the page. But I would like it to click the button and open in a new tab.

search_button = driver.find_element(By.CSS_SELECTOR, "#edit-submit")
driver.execute_script(f"window.open({search_button});")

I get the following error. It seems I'm not passing the proper structure to the script?

File "C:\Python\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.JavascriptException: Message: javascript error: Unexpected token '<' (Session info: chrome=124.0.6367.61)


r/SeleniumPython • • Apr 15 '23

Selenium Using selenium to extract data from chrome to pandas dataframe.

1 Upvotes

Hello,

I am new to coding, I get the gist behind it but I get lost when it comes to debugging. I keep arriving at a InvalidArguementException when using the driver.get() function. There are multiple variables and lists involved which I will show in the actual code below.

But to fully understand my intention is to scrape Linkedin by using google to filter my search results and have the chromewebdriver access and log each link google provides into a pandas dataframe.

The code where the issue arises (below):

Jobdata = []

Lnks = []

for x in range(0,20,10):

driver.get(f'https://www.google.com/search?q=site%3Alinkedin.com%2F+AND+%22managing+director+%40+Morgan+Stanley%22+AND+%22New+York+City+Metropolitan+Area%22&rlz=1C1ONGR_enUS1012US1012&ei=7totZJioCr6dptQPi4qR4Ag&ved=0ahUKEwiY1_KOy5P-AhW-jokEHQtFBIwQ4dUDCBA&oq=site%3Alinkedin.com%2F+AND+%22managing+director+%40+Morgan+Stanley%22+AND+%22New+York+City+Metropolitan+Area%22&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQDEoECEEYAFAAWABgAGgAcAB4AIABAIgBAJIBAJgBAA&sclient=gws-wiz-serp')

sleep(randint(3,5))

linkedin_urls = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='yuRUbf']")))]

Lnks.append(linkedin_urls)

for x in Lnks: for i in x:

        # get the profile URL
        print(i)
        driver.get(i)
        sleep(randint(3,5))

Please let me know of any suggestions or if more info is required. Again, I am a complete noob at coding so some technical aspects may go over my head. Thank you.


r/SeleniumPython • • Apr 02 '23

Help I think I figured this out, now I just need to make it run on autopilot every 2 minutes.

1 Upvotes

How do I set it up on a schedule?

Selenium IDE

I basically needed a macro or automation to login to a site, check for open tickets and then click on each one to close.

I figured out how to do it one time. Now I want it to run and check for open tickets every x minutes.

Thanks in advance.

Edited for clarity


r/SeleniumPython • • Dec 19 '23

Login with "hidden" button - not working

1 Upvotes

Hello,

i am trying to log into https://shop.vfb.de/konto/#hide-registration using python and selenium.

Parsing mail and password is working fine...

How ever i can not manage to click the "Anmelden" Button because there is no ID or something i can search for...(maybe additional info: the button is changing the cullor if you put the mouse on it)

This is how the html is looking:

Anyone having a suggestion what I can do? Is there a working solution to work with such "hidden" buttons?

Thank you so much!!