r/SeleniumPython • • Sep 17 '24

Selenium uses a ton of internet data in conjunction with Google Drive upload

Hi there,

I am writing a program in Pyhon with Selenium on Mac OS that downloads .pdf files from a website and uploads the .pdfs to a Google Drive folder. The pdfs are only a few pages and average at around 300-400kb of data, and I'm downloading at most 50 .pdf files. There are .tmp.drivedownload folders that take up a ton of data in my downloads, with files inside that look like this, e.g. ".com.google.Chrome.AzphV3". These files range from 1-4gb and also populate in my Google Drive, filling up my limited 15gb of storage.

This has caused huge spikes in my internet data usage. When I started this a few days ago, I went through almost all of my data. Here is a photo of my daily usage from my Internet Provider:

When investigating further, most of my data usage is under the "Other" category. It can not be located or traced.

My code is long, but this is the function I wrote to move my .pdf from my downloads folder into my Google Drive folder:

def move_file_to_manifest_folder(manifest_dir,j):
    
    downloads_dir = '/Users/stepdoe/Downloads/'
    time.sleep(3)
    
    # Here I'm searching in my downloads folder for the last .pdf downloaded, then I  am moving that file into my Google Driver folder with os.replace 
    files = list(filter(os.path.isfile, glob.glob(downloads_dir + "*.pdf")))
    files.sort(key=lambda x: os.path.getmtime(x))

    filename = files[-1] # after I sort by time with os.path.getmtime, I take the last file in my list, which corresponds to my most recent file downloaded.
    filename = filename.split('/')
    filename = filename[-1] 
    print(f'filename[-1]: {filename}')
    filename = str(j).zfill(2) + '_' + filename # naming convension for what I want my file to be called in my Google Drive
    newpath = f'{manifest_dir}/{filename}'
    print(f'newpath: {newpath}')
    os.replace(files[-1],newpath)

I am asking for solutions to prevent these huge spikes in data download and uploads. I would expect my daily increase in usage would increase by 2-3gb (at the most 5gb), not in the order of magnitude of 100-500gb. Any help on this would be great, as my internet bill will skyrocket without it.

1 Upvotes

2 comments sorted by

1

u/ixlr8t Sep 18 '24

Would clearing the cache on the browser help.

Something like

driver.delete_all_cookies()

1

u/ixlr8t Sep 18 '24

Maybe this as well

driver.delete_all_cookies() # Clears all cookies

For cache, you could try navigating to a URL to clear it (works in some browsers):

driver.get('chrome://settings/clearBrowserData') # Chrome specific