r/Python • • Nov 14 '17

Senior Python Programmers, what tricks do you want to impart to us young guns?

Like basic looping, performance improvement, etc.

1.3k Upvotes

629 comments sorted by

View all comments

945

u/vosper1 Nov 14 '17 edited Nov 14 '17

Random braindump

  • Use Python 3.6, there are some significant improvements over 2.7 - like enums and fstrings (I would switch to 3.6 just for fstrings, TBH)

  • .open() or .close() is often a code smell - you probably should be using a with block

  • Use virtualenv for every project - don't install python packages at the system level. This keeps your project environment isolated and reproducible

  • Use the csv module for CSVs (you'd be surprised...)

  • Don't nest comprehensions, it makes your code hard to read (this one from the Google style guide, IIRC)

  • If you need a counter along with the items from the thing you're looping over, use enumerate(items)

  • If you're using an IDE (as a Vim user I say you're crazy if you're not using Pycharm with Ideavim) take the time to learn it's features. Especially how to use the debugger, set breakpoints, and step through code

  • multiprocessing, not threading

  • Developing with a REPL like ipython or Jupyter alongside your IDE can be very productive. I am often jumping back and forth between them. Writing pure functions makes them easy to test / develop / use in the REPL. ipython and Jupyter have helpful magics like %time and %prun for profiling

  • Use destructuring assignment, not indices, for multiple assignment first, second, *_ = (1,2,3,4)

  • Avoid *args or **kwargs unless you know you need them - it makes your function signatures hard to read, and code-completion less helpful

155

u/hovissimo Nov 14 '17

Avoid *args or **kwargs unless you know you need them - it makes your function signatures hard to read, and code-completion less helpful

This is my evil. I really need to stop this.

29

u/W88D Nov 14 '17

I wrote a decorator that validates arguments but has no knowledge of the number of arguments. That seems to be a pretty valid use. Other than that I really hate when I shift+tab in jupyter and there's just a *args, **kwargs signature.

How do you use them?

20

u/tunisia3507 Nov 14 '17

If I understand you correctly, consider using functools.wraps: it copies the docstring and possibly the signature of the decorated function into the wrapping function.

→ More replies (6)
→ More replies (14)

59

u/[deleted] Nov 14 '17 edited Nov 14 '17

Use the csv module for CSVs (you'd be surprised...)

People loooooove parsing CSV by hand. "No it's fine I'll just split on commas" - famous last words

Developing with a REPL like ipython or Jupyter alongside your IDE can be very productive. I am often jumping back and forth between them.

If you're a Vim user, I found an awesome plugin recently that puts an inline REPL in your buffer. It would evaluate each line as you wrote it, putting the result on the right side of the screen. Makes for a great scratch pad. However, I couldn't get it to work, and now I can't remember the name. If anyone's interested I can dig for it. EDIT: Found it! https://github.com/metakirby5/codi.vim

21

u/claird Nov 14 '17

I, too, underline this. Yes, the best neophytes invariably say, "I'll just split this CSV on commas", and then muddle for months before acquisition of the humility to realize that the standard module is the right solution.

In the same category: parsing JSON, XML (HTML5, ...), or timestamps by hand. Take advantage of those who've traveled this path before, folks. Yes, your enthusiasm for what a couple of well-crafted regex-es can do is adorable, but believe us that it's misplaced for these domains.

I write this while in the middle of committing a two-line update that applies re to an XML fragment. I know what I'm doing, though; do NOT take this as an excuse to write your own parser for these standard languages.

E-mail addresses are another subject. I'll leave that aside for now.

On the subject of good advice: I'm surprised no one has yet brought up SQLite in this thread.

→ More replies (5)
→ More replies (9)

43

u/nickcash Nov 14 '17
  • multiprocessing, not threading

I assume your point here is about the GIL?

I'd say there are definitely uses for both multiprocessing and threading, and a good Python dev should know when to use which.

I'd also add async to this as well. Either with the Python 3 builtins, or in Python 2 with gevent/eventlet/greenlet.

13

u/emandero Nov 14 '17

This is what I wanted to write. GIL just prevents python code from running parallel. All other code is fine. Especially where you have a lot of i/o like files read, http request etc GIL is not making it work worse.

→ More replies (6)

6

u/Dreamercz QA engineer Nov 14 '17

So I have this simple selenium-based automation for a website that I'd like to speed up by running the task in more browser windows at the same time. Would this be a good case to use any form of the Python concurrency?

→ More replies (2)
→ More replies (1)

56

u/fermilevel Nov 14 '17

+1 for WITH block for opening file, it just makes the code more predictable

→ More replies (21)

18

u/manickam91 Nov 14 '17 edited Nov 14 '17

be conscious with print and oneliners if you are writing in 2.7 and will be porting to 3.x. map returns list in 2.x but itrerator in 3.x

11

u/kthepropogation Nov 14 '17

Since we're talking about 2.7/3.x wackiness:

// is the integer division operator in both 2.7 and 3.x.

/ has different functionality in each. In 3.x, it's the float division operator. In 2.x, it's just the "division" operator, and behaves as float division if one or both of its operands are floats, and int division otherwise.

8

u/masklinn Nov 14 '17 edited Nov 15 '17

from __future__ import division fixes this discrepancy (by backporting Python 3 behaviour to Python 2).

Also // is not the integer division but the floor division operator, this makes a difference when passing floats in

>>> 3.2 / 2.5
1.28
>>> 3.2 // 2.5
1.0

An other subtle difference between P2 and P3 is the behaviour of the round builtin:

  • In Python 3, it implements banker's rounding (round-to-even), Python 2 uses away-from-zero. So round(2.5) is 2 in Python 3 and 3 in Python 2.

  • In Python 3, round(n) returns an int but round(n, k) returns a float (unless k is None), in Python 2 round always returns a float.

Finally while most issues with mixing bytes and str are noisy in Python 3, byte and str objects are always different there. In Python 2, str and unicode could be equal if the implicit decoding (of the str to unicode) succeeded and the resulting unicode objects were equal, and you would get a warning if the decoding failed (the Python 3 behaviour is something of a downgrade there).

→ More replies (2)

17

u/Petrarch1603 Nov 14 '17

what do you mean by 'code smell'?

52

u/mrq02 Nov 14 '17

"Code smell" is when you look at a block of code, and while you're not sure what's wrong with it, something about it seems a bit fishy... Basically, it means "probably sub-optimal code".

→ More replies (2)

21

u/iceardor Nov 14 '17

Code that hints that something is wrong without having to read or understand anything else around it. It gets its name from anything in real that stinks, which you'll smell before you see.

open/close without a context managers will likely result in a resource leak if done outside a try/finally block. And if you're going to add a try/finally block, why not make it a with statement and save yourself 2 lines of code...

→ More replies (3)
→ More replies (6)

49

u/[deleted] Nov 14 '17

A few minor thoughts:

  • Try out pipenv to make virtualenvs simpler (among other things)

  • concurrent.futures vs raw multiprocessing/threading (which of the executors depends on your need, no need to be dogmatic which)

  • try ptpython or bpython if you’re in a repl a lot. Ipython is great, but the others are an even bigger improvement!

9

u/firefrommoonlight Nov 14 '17

Note that Pipenv is currently hard broken.

→ More replies (12)

22

u/TheArvinInUs Nov 14 '17

The only thing i disagree with is using the csv module for CSVs. I've found that using pandas and pandas.from_csv to be much more productive in almost every way.

11

u/patentmedicine Nov 14 '17

For a lot of things, pandas is overkill. If you’re parsing each line of a csv and don’t need a picture of the data as a whole, use the csv module.

10

u/geosoco Nov 14 '17

Agreed. It handles a lot of things a lot better than the csv module.

Plus if you're still stuck on python 2, and your CSV has non-ascii characters -- welcome to hell. Even with the recipe from the docs it turned into a nightmare.

→ More replies (6)
→ More replies (3)

9

u/[deleted] Nov 14 '17

I love fstrings, it's just so convenient

→ More replies (1)

24

u/chillysurfer Nov 14 '17

virtualenv or containers, I'd argue. Good list too, +1

→ More replies (19)

6

u/leom4862 Nov 14 '17 edited Nov 14 '17

Use virtualenv for every project - don't install python packages at the system level. This keeps your project environment isolated and reproducible

I recommend: https://github.com/kennethreitz/pipenv It makes your life a lot easier, when it comes to virtual environments and package dependencies.

Use Python 3.6, there are some significant improvements over 2.7 - like enums and fstrings (I would switch to 3.6 just for fstrings, TBH)

I'd like to add, that 3.5 introduced the typing module, which got lots of improvements in 3.6. The typing module allows you to add type annotations to your code. If you use it, a good IDE will catch type errors right away and your code becomes easier to reason about.

6

u/masklinn Nov 14 '17

.open() or .close() is often a code smell - you probably should be using a with block

Also always always always ALWAYS pass in an encoding to "text-mode" open (the default in Python 3), otherwise Python falls back not on UTF-8 but on locale.getpreferredencoding(False), which may be utf-8 but may also be ascii or mbcs or cp850 or any other sort of nonsense.

If you want the user-configured locale (because you're reading user-provided data) pass it explicitly, but you really do not want whatever garbage has been configured as the locale's encoding when reading or writing your configuration files.

Use the csv module for CSVs (you'd be surprised...)

Also don't use CSVs if you can avoid it, clients will open them in Excel and generate absolute garbage out. Produce and consume Excel or ODF files instead if you can, Python has fairly mature packages to handle Excel data (not sure about ODF). Or sqlite if that's an option (though it usually isn't).

Avoid *args or **kwargs unless you know you need them - it makes your function signatures hard to read, and code-completion less helpful

And if you can restrict your code to P3-only, leverage "keyword-only" parameters (both required and optional) e.g.

def compare(a, b, *, case_insensitive=False, locale_aware=False)

requires providing any parameter other than 1 and 2 by keywords (and works even if they don't have default values), no more e.g.:

compare(a, b, True, False) # dafuq?
→ More replies (2)
→ More replies (73)