r/Python • • Feb 14 '12

Python Performance Tips, Part 1

http://blog.monitis.com/index.php/2012/02/13/python-performance-tips-part-1/
20 Upvotes

53 comments sorted by

View all comments

12

u/pingveno pinch of this, pinch of that Feb 14 '12 edited Feb 14 '12

A few additions:

Section 6:

Don't go overboard with lazy importing inside of function calls. Importing many times can get expensive. Imports that aren't at the top also severely reduce readability. It can be tough to debug code where an import is tucked away somewhere in the body of the file.

Section 7:

In Python 3, True is a keyword, not a built-in. That means the byte code compiler can put in a cheap simple jump instead of an expensive global lookup. In Python 2, True could be changed, so a global lookup was required as a check. 1 is a constant, so the bytecode compiler can depend on it. Basically: while 1 in Python 2, while True in Python 3.

Section 13:

Internally, the Python deque implementation uses a doubly linked list of blocks. Each block has space for 62 objects, plus 1 left pointer and 1 right pointer (64 pointers). That gives the time efficiency of a linked list with the space efficiency of an array. For those learning C, I suggest looking at the Modules/collectionsmodule.c. For a list type with fast insertion, take a look at blist (search on PyPi).

Section 17:

Don't use threads to manage forked processes. They only provide additional overhead in memory and thinking ability. Use the multiprocessing module.

Section 18:

Absolutely absolutely absolutely. The Python source code is an excellent place to get examples. Part of my introduction to C was reading through bits and pieces of implementation code.

6

u/ascii Feb 14 '12

I just did a quick test under Python 2, and found no performance difference between while 1 and while True. I believe that one may exist, but it's so tiny to be lost in the noise of all but the most contrived cases.

1

u/haldean (lambda x: x.decode('base64'))('PDMgcHlweQ==\n') Feb 14 '12

1

u/pingveno pinch of this, pinch of that Feb 14 '12

There's a slight difference, but it is indeed dwarfed by anything else. I was able to measure some difference with this code:

def while1():
    i = 0
    a = 0
    while 1:
        a = 4
        i += 1
        if i > 100000:
            break

For while True, I simply replaced 1 with True. The results in IPython were

%timeit -r200 -n2 whileTrue()
2 loops, best of 200: 13.2 ms per loop

%timeit -r200 -n2 while1()
2 loops, best of 200: 9.33 ms per loop

Conclusion: The difference could only be vaguely significant in extreme conditions. If you run into one of those conditions, just rewrite in Cython.

4

u/[deleted] Feb 14 '12

Importing many times can get expensive.

Surely once a module is imported, a second import will simply be a cheap lookup of the sys.modules dict?

1

u/pingveno pinch of this, pinch of that Feb 14 '12

It's more complex than a simple lookup. I don't entirely understand the C code, but there's a lot more happening than sys.modules["module_name"]. The time penalty isn't significant if you just import a few times, but importing in a function that you're going to call a few thousand times is unwise.

0

u/Genmutant Feb 14 '12

Yes, but cheap is still more than nothing.

4

u/[deleted] Feb 14 '12

Why program in Python if you're worried about the cost of a dictionary lookup? Almost everything in Python needs a dictionary lookup!

1

u/Genmutant Feb 14 '12

Well the topic is Python performance tips, and this one can be a performance impact. Normally it isn't, but it can be and is easy to avoid.

0

u/fdemmer Feb 14 '12

imo, it mostly depends of the kind of application you write. eg. in a webapp, that starts once and runs forever, i'd import as much as possible on the initial load and not later when a function is called.

0

u/fdemmer Feb 14 '12

imo, it depends the most on the kind of application you write. eg. in a webapp, that starts once and runs forever, i'd import as much as possible on the initial load and not later when a function is called.

3

u/flying-sheep Feb 14 '12

my rule of thumb for 6: import as soon as you know you’ll need it.

2

u/Mattho Feb 14 '12

ad 7:

Also it's True (as you wrote it) and not true as is in article.

1

u/[deleted] Feb 14 '12

i've found it often much clearer to read throug hthe source for modules and libraries i use rather than fudge through the official documentation. 30 minutes spent reading code (usually) better written than mine saves 1h+ worth of googling.