r/pythontips • • Aug 29 '22

Python3_Specific 0 experience, send help!

34 Upvotes

Hello fellas,

Just started my engineering studies and turns out they expect you to already know some basic coding «This is not supposed to be a introduction to coding class». Which is wierd since i’ve never touched it in school Prior to this.

Anyways, not wanna half ass my studies i yet again turn to Reddit for advice. Instead of looking up everything on Google spending hours and hours trying to filter out What is actually useful i figured you guys got some gold nuggets up your sleeves. Does anyone here have any advice for an absolute beginner? I’ve considered codecademy for instance. Again, i cant even type in 1 input thats valid atm. We’re using Python through spyder in class for what its worth.


r/pythontips • • Aug 05 '22

Meta About learning Python

34 Upvotes

Hi, I am learning Python on my own, can someone learning on their own work as a Python programmer? Or is there much difference between going to programming class?

If you can give me some advice about programming or studying I would appreciate it.


r/pythontips • • Mar 31 '22

Data_Science Don’t know where to start when it comes to understanding python. Need help with my approach

34 Upvotes

I do not have a programming background but am looking to get into Data Science and become a Data Scientist. I am taking a course but I am falling behind due to my lack of programming knowledge.

I am aware that I have basic understanding of syntax such as when to use .head(), .tail(), print() but I have trouble understanding:

  1. When to use curly brackets, square brackets, parenthesis, etc. and when to use multiple brackets

i.e wts[((wts[‘diet’]==1) - how do I know when to use which brackets and how do I know when I need multiple sets of brackets?

2. How to structure the code. I typically don’t know where to start. Is there a specific order that code needs to go in? (I know the data frame goes first) But if a question asks me to convert a column, filter the data and create a scatter plot I’m not sure what order to put everything in.


r/pythontips • • Jan 20 '22

Data_Science Guide to Applied Data Science with Python

34 Upvotes

Define what Data Science actually is and see a visual preview of it using the Pandas Library

Applied Data Science


r/pythontips • • Nov 24 '21

Syntax Learn how to work with Python's magic methods

34 Upvotes

Writing nice code and interfaces using the core features of python is something that we should all do. If you want to learn how to do that using magic methods, this article might be for you https://towardsdatascience.com/how-to-write-awesome-python-classes-f2e1f05e51a9?sk=48efd3274df1115ee48024a787b363dc Enjoy reading.


r/pythontips • • May 27 '21

Python2_Specific 🐍 What's new in Flask 2.0

34 Upvotes

Hey Devs! Anyone here uses Flask? Do you know what's new in Flask 2.0? I'd like to share with you a very interesting article about  Flask 2.0. Read more about the Tutorial here.


r/pythontips • • Feb 11 '21

Module Is there a way to mass download articles as pdfs/epubs from links on a webpage?

34 Upvotes

I'm reading a an epic fantasy book series (Malazan Book Of The Fallen). It has a reread from tor.com with chapter by chapter discussion articles.

Here: https://www.tor.com/series/malazan-reread-of-the-fallen/

I want to be able to download all the links from the page as either pdfs or epubs so I can read them offline on my kindle/phone.

Is it possible to do it?


r/pythontips • • May 02 '20

Long_video python free course

34 Upvotes

r/pythontips • • Jan 03 '20

Making a Little Database is easy with TinyMongo!

34 Upvotes

I love MongoDB, and PyMongo, but a full MongoDB replica set server cluster is overkill for a lot of cases. In those instances you can achieve all of the same things with TinyMongo. It's like SQLite, but for NoSQL!

In my latest Intermediate Python tutorial video I show how easy it is to add persistence to your application with TinyMongo:

https://youtu.be/mx-c87-os3A

If you missed the previous videos in the series you can watch them all in this playlist: https://www.youtube.com/playlist?list=PLMD4SGMb7qkVt4drPdDxImhhGw3ZNuzbB


r/pythontips • • Dec 05 '19

Learn How to Download Files with Scrapy

34 Upvotes

LONG POST ALERT - I have created a blog out of this post at https://coderecode.com/download-files-scrapy/ and a free video course at https://coderecode.com/download-files-mini-course on the same topic

---------------------------------------------------------------------------------

In this tutorial, we will learn how to download files with Scrapy. It may look daunting at first but is actually easy with its Crawl spider. This tutorial will walk you through all the steps. The site that I have chosen for this tutorial is www.nirsoft.net. This site has a lot of small utilities and tools that have been lifesaver many times and has been my favorite for many years. I used Wireless Network Watcher to identify who is connected to my wifi and eventually take measures to secure it for example. I thought it would be a good idea to have all utilities downloaded from this site. The perfect solution to this use case is web scraping where I can talk about crawlers and downloading files.

Prerequisite

This tutorial shows how to download files with scrapy. Therefore, it assumes that you are familiar with the concept of web scraping and the basics of Python. If you don’t know what web scraping, you will get a general idea from this tutorial.

I assume that you have at least working knowledge of Python though. This tutorial also assumes that you have at the very least, have played around with Scrapy.

Install Scrapy and Create Scrapy Project

If you want to download files with scrapy, the first step is to install Scrapy. Scrapy is the single most powerful framework for all kinds of web scraping needs. All other tools like BeautifulSoup4, Selenium, and Splash integrate nicely with u/ a rule of thumb, install it in a virtual environment. If you are not familiar with virtual environments, they are like virtual machines. Instead of a different operating system, they have their own packages installed. I am just going to install it at the user level.

pip install scrapy

Create a directory where you want to run this project and create a new Scrapy project

md nirsoft
cd nirsoft
scrapy startproject zipfiles

The output would be something like this.

New Scrapy project 'zipfiles', using template directory 'XXX', created in:
    D:\nirsoft\zipfiles

You can start your first spider with:

    cd zipfiles
    scrapy genspider example example.com

Spider Templates in Scrapy

There are four templates available in Scrapy. These can be used in different scenarios. You can use any of them to download files with scrapy. The final use depends on how you want to reach the pages with download links.

It's time to check the templates offered by Scrapy.

scrapy genspider -l
This outputs:
Available templates:
  basic
  crawl
  csvfeed
  xmlfeed
  • Basic – General purpose spider

  • Crawl – Spider for crawling, or following links

  • csvfeed – Suitable for parsing CSV files

  • xmlfeed – Suitable for parsing XML files

crawl : Most Suitable to Download All Files with Scrapy

For this scenario, the most suitable template is crawl. Let’s create the basic structure of the spider (aka scaffold) using this template.

cd zipfiles
scrapy genspider -t crawl nirsoft www.nirsoft.net

This creates an “empty” crawl spider:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class NirsoftSpider(CrawlSpider):
   name = 'nirsoft'
   allowed_domains = ['www.nirsoft.net']
   start_urls = ['http://www.nirsoft.net/']
   rules = (
       Rule(LinkExtractor(allow=r'Items/'), callback='parse_item', follow=True),
   )
   def parse_item(self, response):
       item = {}
       #item['domain_id'] = response.xpath('//input[@id="sid"]/@value').get()
       #item['name'] = response.xpath('//div[@id="name"]').get()
       #item['description'] = response.xpath('//div[@id="description"]').get()
       return item

Getting All the Links

Our crawler needs to know what links it should be following. This is where Rules and LinkExtactor come into the picture.

About Rules

The most important line of code here is the Rule. Rules are what define links that should be followed. These are what save us for loops. Rules can be created using many ways, including, but not limited to:

  • allow

  • deny

  • restrict_xpaths

  • restrict_css

The list is quite long. In this tutorial, we won’t need all those. If you want more information, take a look at the official documentation.

Analyzing the Links

One quick look at the nirsoft.net site will reveal that the details URL of utilities begin with the prefix utils. For example:http://www.nirsoft.net/utils/web_browser_password.htmlhttp://www.nirsoft.net/utils/outlook_nk2_edit.htmlhttp://www.nirsoft.net/utils/usb_devices_view.htmlIt makes things easier as we can narrow down pages to crawl.The update rule now looks like this:

rules = (
        Rule(LinkExtractor(allow=r'utils/'), callback='parse_item', follow=True),
    )

So now our crawler is going to all the pages. Now its time to get the link to the download file.

Parsing the Crawled Page

Let’s navigate to any of the details pages, and Inspect the download link (F12 - Developer Tools), Sweet! the download link has the class downloadline. This looks pretty easy. Let’s use the CSS selector and extract the href attribute text:

def parse_item(self, response):
   file_url = response.css('.downloadline::attr(href)').get()

Relative Links to Absolute LinksThe links are relative and we need absolute links. In the newer versions of Scrapy, its super easy. Just call response.urljoin() The code is now

def parse_item(self, response):
    file_url = response.css('.downloadline::attr(href)').get()
    file_url = response.urljoin(file_url)
    yield { 'file_url': file_url }

At this point, if we run the crawler, we will have a complete list of files.

scrapy crawl nirsoft

Next step, downloading the files.

Downloading Files

Let’s update the item class that was generated with the project and add two fields. NOTE: The field names have exactly the same for this to work. See Scrapy documentation.

class ZipfilesItem(scrapy.Item):
    file_urls = scrapy.Field()
    files = scrapy.Field

Now let’s create ZipfilesItem object in the spider. We would need to set the file_urls attribute value to the url of the files that we are downloading. Again note that it needs to be a list. Here is the updated code.

def parse_item(self, response):
    file_url = response.css('.downloadline::attr(href)').get()
    file_url = response.urljoin(file_url)
    item = ZipfilesItem()
    item['file_urls'] = [file_url]
    yield item

Now let’s enable the file download pipeline in the settings.py file:

ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1}

The last step is to specify the download location in settings. This should be a valid directory and the setting name is FILES_STORE. I am using raw strings to avoid escaping backslashes on windows:

FILES_STORE = r'D:\nirsoft\downloads'

That’s all we need. Now when we run the crawler, the files will start downloading.

scrapy crawl nirsoft

Limiting File Types Downloaded

We would see however that there are few files which we don’t need. Let’s modify the parse_item method so that only zip and exe files are downloaded.Further, let’s add one more field to ZipfilesItem class and set it before yielding. (See next section for why we are doing this)This is how the parse_item looks at this point

def parse_item(self, response):
    file_url = response.css('.downloadline::attr(href)').get()
    file_url = response.urljoin(file_url)
    file_extension = file_url.split('.')[-1]
    if file_extension not in ('zip', 'exe', 'msi'):
        return
    item = ZipfilesItem()
    item['file_urls'] = [file_url]
    item['original_file_name'] = file_url.split('/')[-1]
    yield item

Keeping Original File Names

The default implementation of the FilesPipeline does not keep the original file names. How the File Names are generated. For example, the URL

http://www.nirsoft.net/toolsdownload/webbrowserpassview.zip

will have its SHA1 Hash as

0f2ad6a88c00028c5029e9cec0fea3be5d7fe434

This file be saved as

0f2ad6a88c00028c5029e9cec0fea3be5d7fe434.zip

To resolve this, we need to write our custom Pipeline.

Custom Pipeline

We already have ZipfilesPipeline generated in our code, but we are not using it. We can either modify this or create a new Class.If you look at ZipfilesPipeline class, it inherits from Object. We need to change it so that it inherits FilesPipeline. This would also mean importing the FilesPipeline in the file.

from scrapy.pipelines.files import FilesPipeline
class ZipfilesPipeline(FilesPipeline):
    pass

Controlling File Names

The next step is to override file_path method, which generates the file names. The default implementation of file_path looks like this:

def file_path(self, request, response=None, info=None):
    # DEFAULT IMPLENTATION
    media_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()
    media_ext = os.path.splitext(request.url)[1]
    return 'full/%s%s' % (media_guid, media_ext)

As the last part of the request is the file name, we can update this function and remove all the hash generation part.

def file_path(self, request, response=None, info=None):
    file_name: str = request.url.split("/")[-1]
    return file_name

Update Settings

Now let’s update Settings file to use our custom Pipeline instead of FilesPipeLine

ITEM_PIPELINES = {'zipfiles.pipelines.ZipfilesPipeline': 1}

Sometimes, you may see some errors because of user-agent and referer missing in the request. Let’s add these to the settings file.

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8',
    'Accept-Language': 'en',
    'Referer': 'https://www.nirsoft.net',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36'
}

Everything is ready now. Let’s run the crawler and within seconds we will have 252 files downloaded.

scrapy crawl nirsoft

How did you find this? Did you face any problems? Let me know!


r/pythontips • • Nov 29 '19

TIL: PIP_REQUIRE_VIRTUALENV disables pip outside of a virtualenv

34 Upvotes

One can just set this environment variable (`export PIP_REQUIRE_VIRTUALENV=true` in bash) and pip will no longer install packages outside of virtual environments! I wish I knew about it before, I've messed up my dependencies way too many times ;)

https://switowski.com/blog/disable-pip-outside-of-virtual-environments


r/pythontips • • Mar 07 '19

Python Sets: What, Why and How

34 Upvotes

Explore the way that Python Sets can help you not just with readability, but also speeding up your programs execution time.

Python Sets: What, Why and How


r/pythontips • • Dec 23 '18

How and Why to use Lambda functions in Python[Tutorial]

34 Upvotes

Lambda functions intimidate people way more than they should. They are a great way to write short, one statement expression functions. Did you know they are called anonymous functions? Watch the video to learn why and everything else about lambda.


r/pythontips • • Jun 05 '18

Module Domino animation - only 10 lines of Python code

34 Upvotes

r/pythontips • • Aug 31 '23

Short_Video Pro Tip for Beginners: Enhance Code Readability with the Dictionary .get() Method

33 Upvotes

During coding interviews, we frequently encounter straightforward dictionaries with a structure as shown below:

my_dict = {"key1": 10, "key2": 20, "key3": 30}

In many scenarios, our aim is to determine the existence of a key within a dictionary. If the key exists, we intend to take action based on that key and then update its value. Here's an example:

key = 'something'

if key in my_dict: print('Already Exists') value = my_dict[key] else: print('Adding key') value = 0 my_dict[key] = value + 1

This is a routine process commonly encountered in various coding challenges and real-world programming tasks.
Nevertheless, this approach has some drawbacks and can become a bit cumbersome. Fortunately, the same task can be accomplished using the .get() method available for Python dictionaries:

value = my_dict.get(key, 0)

my_dict[key] = value + 1

This accomplishes the same objective as the previous code snippet but with fewer lines and reduced direct interactions with the dictionary itself.
For those who are new to Python, I recommend being aware of this alternative method. To delve deeper into this topic, I've created a YouTube video providing more comprehensive insights: https://www.youtube.com/watch?v=uNcvhS5OepM
If you're a Python novice seeking uncomplicated techniques to enhance your Python skills, I invite you to appreciate the video and consider subscribing to my channel. Your support would mean a lot, and I believe you'll acquire valuable skills along the way!


r/pythontips • • Mar 07 '23

Standard_Lib Newbie Tips you WISH you had when you started your Python Journey.

33 Upvotes

Hello everyone. First time posting on reddit. Apologies if this is not the way.

I am beginning my python journey as a means to get out of my dead end job & hopefully land a higher paying job. The only coding experience I have is from when I was in high school & built a website to get around our school's firewalls, mainly HTML/XHTML.

So. What are some tips you wish you had when you started your Python journey? As a reference, I am considering this 100 Days of Code - The Complete Python Guide to use as a good starting point. Any helpful resources or tips would be greatly appreciated!


r/pythontips • • Nov 09 '22

Short_Video Python Small Project Idea

33 Upvotes

Hello, Everyone. For the school project, I was assigned to take a 3-min video of teaching how to write a program with Python. But I have no idea which small project I should teach everyone in the video. Asking for ideas of small Python projects that can be shown everyone in 3min. thank you in advance


r/pythontips • • Dec 05 '21

Data_Science Finding useful python project to do

33 Upvotes

Hey everyone,

I am looking to work on a python project to improve my skills but I can't think of a unique project that is actually useful once it is completed. So I was wondering if you guys have any unique and useful project ideas.

Cheers


r/pythontips • • Aug 20 '21

Python3_Specific What if you forget to add Python to PATH?

33 Upvotes

If you forget to add Python to the PATH, you won't be able to run Python in your command line from any other directory.

You will get a message saying,

'python' is not recognized as an internal or external command.

This is because Windows is trying to search for Python in its PATH variable and is unable to find it.

Here is a step-by-step instruction to add Python to the PATH variable in Windows 10.

  1. Copy the directory to which Python is being installed.

  2. Search for System Properties in the Search Bar.

  3. Click on the "Environment Variables" button in System Properties.

  4. Click on the "Path" variable and then click on "Edit".

  5. Now here's the deal. Pay Attention! Paste the directory path from Step 1 at the top of the list. You can use the "Move up" button to get it to the top. Now click on "Ok".

Now you can open up your command line and run python. The Python interactive shell will open up.

*Here is a little extra tip! *

Add the Scripts folder to the PATH variable too. This is the folder where all your libraries will be installed.

If the instructions are not clear enough for you, you can also check out the visual guide containing images of every step, on my blog for achieving the above objective.

Good luck and let me know!


r/pythontips • • Jul 19 '21

Module 🐍 Serverless Django APIs with AWS Lambda and Zappa

33 Upvotes

Hey folks! How is it going! I’d like to share with you a very interesting article about Serverless Django APIs with AWS Lambda and Zappa. I’m super excited to know if you tried something like this 🤯. Read more about the Tutorial here.


r/pythontips • • Jul 17 '21

Module How to encrypt my python script?

33 Upvotes

I have 1 python script. I have to give this script to 3rd party client but don't want to expose code to them.

how can I do this? want something like .exe files only executables.


r/pythontips • • Jun 03 '21

Syntax Learn the short-circuiting functionality of the Boolean operators 'and'/'or'

33 Upvotes

"Short-circuiting" means that and and or only evaluate the right operand if the value of the left operand is not enough to determine the final result. This might seem odd at first, but learning this means you'll be able to use it to write idiomatic code, and

  • flatten and simplify some if structures;

  • write idiomatic code to assign default values to variables;

  • optimise some conditional statements;

among other things.

You can read about this in more detail in the docs or in this article I wrote with many examples from real code in the Python Std Library.


r/pythontips • • May 04 '21

Syntax 12 Function Related Exercise Every Python Developer Should Learn!

33 Upvotes

r/pythontips • • Jan 26 '21

Module Diving deeper into Openpyxl

33 Upvotes

This week, I decided to improve my file handling skills a little bit more and focus on handling Excel files with Openpyxl. Before this, I only used pandas methods to read Excel file. But Openpyxl seems to be a great module for performing various Excel operations and automating excel reports.

Check out my blog on using Openpyxl.