r/SeleniumPython • • Nov 22 '24

Is web crawler wrong in Brazil in this case?

2 Upvotes

I made a web crawler to get Negative Certificate which is public data but on the website there was a CAPTCHA. Have I committed any infraction? As it is public data, I understood that it would be allowed, but then I felt nervous. It is not copyright content. My fear is exposing my company to some risk.


r/SeleniumPython • • Oct 07 '24

Help How Can I Deploy a Selenium Web Driver App That Extracts Tables from Images?

2 Upvotes

Hey everyone! I’ve built a web driver application using Selenium that scrapes a webpage, captures a full-page screenshot, and extracts tables from the image using OpenCV. It then processes this data further to return results. The app is built using Flask for the API. Now, I want to deploy this application, and I’m wondering about the best options for deployment.

Here’s a rough overview of the tech stack:

Selenium for scraping and screenshots. Flask to serve the API. OpenCV for image processing. It extracts tabular data from a webpage screenshot. Any suggestions or best practices for deploying this type of app? Thanks!


r/SeleniumPython • • Sep 08 '24

Element visibility problem when Webscraping with Selenium

2 Upvotes

Hi,

I'm a student who's writing webscraping code using Selenium on python for the first time. I have limited knowledge of the library and very basic knowledge of web components too. My aim is navigating different webpages on the platform and simulating user actions to perform the typically manual file extractions , then uploading the extracts on python for transformation (As part of an intended python ETL pipeline).

However, I noticed some extract buttons are included in drop down lists which are not always visible as HTML/CSS elements. I'm seeing that there are attributes such as aria-hidden and aria-live attributes. And I suspect javascript is involved as well.

Any advice on how to deal with the situation? (and, if the request is not against the sub rules, is someone willing provide some help or guidance in private ?)


r/SeleniumPython • • Sep 02 '24

Help How do I get rid of this annoying Popup (without having to remove Teams from my machine)

2 Upvotes

It doesn't let me proceed or click any other button without manually closing the popup which I cannot do when I run the script with the headless argument. Any help would be much appreciated!


r/SeleniumPython • • Aug 13 '24

Help can someone help me and tell me why my code is throwing syntax errors? thanks

2 Upvotes

import os
import time
import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.binary_location = '/Applications/Opera GX.app/Contents/MacOS/Opera'
driver_path = 'file_path/operadriver_mac64/operadriver'
service = Service(executable_path=driver_path)
driver = webdriver.Chrome(service=service, options=chrome_options)


r/SeleniumPython • • Aug 03 '24

AttributeError: module 'selenium.webdriver' has no attribute 'PhantomJS'

Thumbnail
gallery
2 Upvotes

r/SeleniumPython • • Jul 29 '24

Selenium Unable book exam slots which are sold instantly

2 Upvotes

I'm working on a Selenium project to book exam slots for the Goethe exam. The booking opens at a specific date and time, and I'm struggling to secure a slot. My current script refreshes the page to detect when the booking button appears and then attempts to click it immediately. However, even if I click the button the moment it appears, the slots are already full.

Has anyone tackled a similar issue? Are there more effective methods with Selenium to achieve this without continuously refreshing the page? Any advice or alternative approaches ?


r/SeleniumPython • • Jun 30 '24

can you give me repo selenium python to refer ??

2 Upvotes

r/SeleniumPython • • Jun 22 '24

How to locate web elements for pyautogui using selenium.

2 Upvotes

I am using element.locate & element .size to get location & size of element i want to click using pyautogui. Now problem is selenium gives coordinates in webpage whereas pyautogui calculate that coordinate on monitor's screen. So both coordinate are mismaching. If I find coordnate manually by . position () method in pyautogui than it works. But I first want to locate element via selenium and want to pass location of that element in pyautogui. Please help.


r/SeleniumPython • • May 31 '24

Using Selenium to read discord messages and get triggered?

2 Upvotes

Is there any way to use a selenium chromedriver to read the discord server chat and print it in the cmd?


r/SeleniumPython • • May 23 '24

Help automation with persistent browser in a loop?

2 Upvotes

Is it possible to keep the browser open during a loop . For example I have my script set up as a loop for function calling…let’s say the first run open browser is called…. Instead of it closing after opening it I want the script to loop back around and call another function eg. scroll down function.. I want all of this to happen on the same browser window. my issue is every time I’m able to get the browser to open up on the first run, but then as soon as I reached the beginning of the loop again it closes the browser that was open and reopens a new one instead of resuming on the same Ro I reached the beginning of the loop again it closes the browser that was open and reopens a new one instead of resuming on the same browser for the next action.


r/SeleniumPython • • Apr 29 '24

Selenium iteration issue

2 Upvotes

Hey Everyone, I wrote a program that iterates through each airbnb listing, gets data and moves on the next listing. The problem is it just loops one listings. any ideas?

def fetch_images_from_carousel(driver):
images = set()
print("Fetching images from Carousel:")
try:
print("Loading images")
WebDriverWait(driver, 10).until(
EC.visibility_of_all_elements_located((By.CSS_SELECTOR,
".itu7ddv.atm_e2_idpfg4.atm_vy_idpfg4.atm_mk_stnw88.atm_e2_1osqo2v__1lzdix4.atm_vy_1osqo2v__1lzdix4.i1cqnm0r.atm_jp_pyzg9w.atm_jr_nyqth1.i1de1kle.atm_vh_yfq0k3.dir.dir-ltr"))
)
initial_images = driver.find_elements(By.CSS_SELECTOR,
".itu7ddv.atm_e2_idpfg4.atm_vy_idpfg4.atm_mk_stnw88.atm_e2_1osqo2v__1lzdix4.atm_vy_1osqo2v__1lzdix4.i1cqnm0r.atm_jp_pyzg9w.atm_jr_nyqth1.i1de1kle.atm_vh_yfq0k3.dir.dir-ltr")
for img in initial_images:
src = img.get_attribute('src')
if src and isinstance(src, str) and src not in images: # Ensure src is a string and not previously added
images.add(src)
print("Added image:", src)

# Handle the carousel navigation using JavaScript to click the button
try:
while True:
next_button = driver.find_element(By.CSS_SELECTOR, ".ms83rji.atm_fg_esu3gu.dir.dir-ltr")
driver.execute_script("arguments[0].click();", next_button) # Click the button using JavaScript
time.sleep(3) # Adjust the delay to allow the next set of images to load (e.g., 3 seconds)
new_images = fetch_current_visible_images(driver)
print(f"Fetched {len(new_images)} new images after clicking the button.")
print("New Images:", new_images)
if set(new_images) == images:
logging.info("No new images loaded. Exiting carousel navigation.")
break
except TimeoutException:
logging.info("Timed out waiting for carousel elements.")
except Exception as e:
logging.error(f"Error navigating carousel: {str(e)}")

return images

heres a portion of the code that fails. The main is

def scrape_airbnb(url, state_name):
table_name = f"{state_name.replace(' ', '_')}"
print("Starting Scrape For: ", table_name)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
try:
driver.get(url)

# Find listings one by one to avoid stale element references
while True:
listings = WebDriverWait(driver, 10).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".dir.dir-ltr"))
)

for listing in listings:
try:
# Perform scraping operations for each listing
images = fetch_images_from_carousel(driver)
price = listing.find_element(By.CSS_SELECTOR, "._1jo4hgw").text
beds = listing.find_element(By.CSS_SELECTOR,
".a8jt5op.atm_3f_idpfg4.atm_7h_hxbz6r.atm_7i_ysn8ba.atm_e2_t94yts"
".atm_ks_zryt35.atm_l8_idpfg4.atm_mk_stnw88.atm_vv_1q9ccgz.atm_vy_t94yts.dir"
".dir-ltr").text
ratings = listing.find_element(By.CSS_SELECTOR,
".t1a9j9y7.atm_da_1ko3t4y.atm_dm_kb7nvz.atm_fg_h9n0ih.dir.dir-ltr").text

# Click on the listing to view more details if needed
listing.click()

# Wait for the new page to load
WebDriverWait(driver, 10).until(EC.url_changes(url))

listing_id = driver.current_url.split('/')[-1] # Extract listing ID from URL
insert_or_update_listing_data(table_name, listing_id, images, price, beds, ratings)

# Go back to the previous page to continue scraping other listings
driver.back()
except NoSuchElementException:
continue
except StaleElementReferenceException:
continue
# Check if there's a next page
try:
next_page_button = driver.find_element(By.CSS_SELECTOR, "._1u3e89e")
next_page_button.click()
except NoSuchElementException:
break
except Exception as e:
logging.error(f"Error processing a listing: {e}")

finally:
driver.quit()


r/SeleniumPython • • Mar 25 '24

How to do properly loop in Seleniumbase?

2 Upvotes

Hello!

I am struggling with doing driver loops using BaseCase on SeleniumBase.

It doesnt work to do self.open() and go to another page for example

How to loop drivers in seleniumbase? (pytest)

Thank you!


r/SeleniumPython • • Mar 16 '24

Help can't get playlists from youtube music through invalid html

2 Upvotes

I'm trying to save playlists from youtube music (not the music files, just a list which songs are in the playlist),

but it looks like I get invalid html, so selenium or other like html-simple-dom can't parse it.

A simple find_element never works, no meter which filter.

The same code, just on another website works without issues.

Are there some hacks how I could get it working?

Is there something known with youtube (music)?

Thanks!

EDIT: could solve it with BeautifulSoup


r/SeleniumPython • • Feb 13 '24

Run Selenium on Windows and MacOS devices without installing software?

2 Upvotes

Is it possible to run a Selenium script on a device without installing software or browser extension?

The aim is to screen share with the user of the remote device. They will be running as a standard user without the privileges required to install software, they will also not be allowed to install anything else such as browser extensions. The script should open a browser and go to a URL. The mouse should then make a series of clicks on the application. Ideally this would also be recorded within the script itself and accessible from the cloud server, worst case is it would be saved locally.

Perhaps Selenium isn’t the answer, any suggestions are more than welcome.

I have been able to use BrowserStack to record a script and then run on my device, but I had to install software on my device to do so.


r/SeleniumPython • • Jan 24 '24

ActionChains send_keys can’t handle emojis

2 Upvotes

I made a bot that logs into my facebook daily, checks to see whose birthday it is via the events page, and submits a happy birthday message via the form field. Facebook seems like it’s always changing, so this bot has changed a lot over the years. Most recently, once the birthday form fields are identified by XPATH, an exception occurs indicating that send_keys can’t type inside the form (can’t interact with the element). I eventually got this to work with ActionChains, but the send_keys method it uses can’t seem to handle emojis. Is there another way to click inside the form and send the keys?


r/SeleniumPython • • Dec 08 '23

Python Selenium Tutorial #13 - Proxies Explained: How to Use Them Effectively

Thumbnail
youtube.com
2 Upvotes

r/SeleniumPython • • Nov 15 '23

How do I count the number of times a looping Selenium script has been run in the Script itself?

2 Upvotes

As the title says guys...

Im looping my Selenium script 200 times with the cmd "times".

Is there another command I can place at the end of my code that will count and show me live how often the script has looped already?

Thanks for your help!


r/SeleniumPython • • Sep 05 '23

Basic Question about Selenium, Python, and Webscraping

2 Upvotes

I am new to web scraping and Python. I am trying to scrape the sector performance chart on https://digital.fidelity.com/prgw/digital/research/sector . I know how to use the basics of selenium. When I use a chrome driver to find an element by xpath, for example,

driver.find_element(By.XPATH, value = '//*[@id="market-sector-performance-table"]/tbody/tr[1]/td[2]') 

to get the S&P 500 performance for 1 month (-1.59%), it says could not find element with that xpath. On some websites that I scrape, this works, on others like the Fidelity site, it doesn't. Why is this? Does it have anything to do with JavaScript or that the website is dynamical? What is the work-around to get the elements I need in this case?

Similarly, I have tried using BeautifulSoup to get the data, and I get empty lists with no data, or errors saying elements could not be found.

How specifically would I scrape the Fidelity chart with Python? Specific code would be very helpful.


r/SeleniumPython • • Aug 02 '23

Python and selenium help

Post image
2 Upvotes

What is wrong with this code?


r/SeleniumPython • • Jul 29 '23

Selenium Chrome Webdriver out of date

Post image
2 Upvotes

My program gets a error "there is no such driver by URL https://chromedriver.storage.googleapis.com/LATEST_RELEASE_115.0.5790". The driver is out of date on this link. I get The original file but on my code I use a reference to it. Someone who do It like me know How did now?


r/SeleniumPython • • Jul 17 '23

Help Tips & Best Practices To Know Before Automating Your Testing Work

Thumbnail
testorigen.com
2 Upvotes

r/SeleniumPython • • Jun 27 '23

Help What are the Selenium suite components?

2 Upvotes

Cypress and selenium are so similar, I am not sure which platform to learn. Selenium also has different components.

https://www.instagram.com/reel/Ct_FNStLNqQ/?utm_source=ig_web_copy_link&igshid=MzRlODBiNWFlZA==

which platform to learn.

PS. I've been trying to switch profiles.


r/SeleniumPython • • Jun 05 '23

Webdriver Exception

Post image
2 Upvotes