r/Python • u/nofishme • Jul 31 '17
Why is Python 50% faster under Windows Subsystem for Linux?
> python.exe -c "import sys; print sys.version"
2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)]
> python.exe -m test.pystone 1000000
Pystone(1.1) time for 1000000 passes = 4.98978
This machine benchmarks at 200410 pystones/second
> bash.exe
$ python -c "import sys; print sys.version"
2.7.13 (default, Jul 31 2017, 11:24:24)
[GCC 5.4.0 20160609]
$ python -m test.pystone 1000000
Pystone(1.1) time for 1000000 passes = 3.34375
This machine benchmarks at 299065 pystones/second95
u/rpns Jul 31 '17
Try again with Python 3.6, that uses a much more recent version of MSVC. See https://wiki.python.org/moin/WindowsCompilers for a table matching Visual Studio versions to Python versions.
9
u/IronManMark20 Jul 31 '17
I tried with 3.5 (which uses the same MSVC version as 3.6) and it was still faster than on Windows. See my theory as to the difference https://www.reddit.com/r/Python/comments/6qndr6/why_is_python_50_faster_under_windows_subsystem/dkzgggb/
11
u/IronManMark20 Jul 31 '17
Part of the reason for this is that on the WSL, Python takes advantage of "Computed Gotos" to optimize the interpreter inner loop. Basically, the bytecode Python executes branches a lot. Computed gotos allow the processor to predict the opcode (like add) Python will execute.
You can run python -c "import sysconfig;print(sysconfig.get_config_var('USE_COMPUTED_GOTOS'))" to find out if your Python uses it (try it in bash, it should print 1).
This is a feature of GCC, the C compiler Python is compiled with on Linux. On Windows, Python cannot use this trick, thus the inner loop of the interpreter runs slower. You can read more about computed gotos in the main interpreter loop here: https://github.com/python/cpython/blob/master/Python/ceval.c#L714 and https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html
6
u/GitHubPermalinkBot Jul 31 '17
I tried to turn your GitHub links into permanent links (press "y" to do this yourself):
Shoot me a PM if you think I'm doing something wrong. To delete this, click ^here.
16
u/ldpreload Jul 31 '17
One thing that's worth noting is that WSL is a subsystem, just like Win32 is a subsystem: you're actually bypassing parts of normal Windows, and using just the underlying NT kernel + the Linux interface on top of it, instead of the usual Win32 interface. This includes what UNIX people would call the C library, which includes things like memory allocation. It's entirely possible that the glibc memory allocator is algorithmically faster or just more efficient (talks to the OS less often and maintains a larger pool of memory) than the Windows one.
I think you'd need to find some sort of profiling tool to get a real answer. Windows Performance Toolkit sounds like the right place to start, and probably it's capable of tracing WSL processes. See if you can generate flamegraphs of the two and see where things are slower in the Windows version.
2
u/Treyzania Jul 31 '17
It's entirely possible that the glibc memory allocator is algorithmically faster or just more efficient (talks to the OS less often and maintains a larger pool of memory) than the Windows one.
It's all probably faster.
18
u/gangtraet Jul 31 '17
Just a guess: it is compiled with gcc, which is very good at optimizing code. A bit surprising if a Microsoft compiler is that much slower, I would have expected only a marginal difference. But this is my best guess.
7
Jul 31 '17
It might be that Microsoft compiler don't support computed goto which can make a huge difference in an interpreter. I believe other specialties of gcc are also used which also could make difference. I believe the main reason is that the main part developers has primary target unixes and so target to gcc or clang. So they use gcc and clang idioms when possible. It could exists some idioms which makes Visual C code faster but there is not enough windows coder amoung core python developer to use it.
9
u/masklinn Jul 31 '17
It might be that Microsoft compiler don't support computed goto
To the best of my knowledge, it's supported by other major compilers such as ICC and Clang, but not by Visual C++.
and that's unlikely to have changed since it's a GCC extension of C.
(caveat for the quote's correctness, I believe ICC only supports computed gotos on Linux as it strives for compatibility with each platform's primary compiler).
9
u/lion_rouge Jul 31 '17
It has nothing to do with compilers. The author said he BUILT python for WSL from source. It's the difference between generic binary build and custom optimized build.
5
u/caffeinedrinker Jul 31 '17
Part of me was hoping this was a rhetorical question and some explaining would be presented when i clicked the link. I was wrong.
2
u/ionelmc .ro Jul 31 '17 edited Jul 31 '17
Ubuntu's python package use PGO (some overview: https://www.activestate.com/blog/2014/06/python-performance-boost-using-profile-guided-optimization) - I suspect Windows builds don't (2.7 is ancient).
Also, pystones don't necessarily correlate to real world performance.
1
u/IAmALinux Jul 31 '17
Out of curiosity, can you run the same test on your machine under native GNU/Linux?
1
1
0
u/brennanfee Aug 01 '17
<snide comment incoming>
Because everything works better under Linux. :-)
More seriously... I think it's because the Python on windows uses MinGW which before WSL was a popular way to get things to work on Windows. Given that it is external from the OS I would imagine it would be slower. WSL is designed and implemented by MS so I imagine they know better how to create a lightweight translation layer.
-5
u/the-kid89 Jul 31 '17
I'm just going to put this out there for you. Your statement is invalid. There are a number of issues with your test. The first of which is you have not run your tests for long enough. The second big one is that you are testing how fast test.pystone runs and sys.version can print the version number out. They both seem like things that don't matter in a production app. I can't think of a time I needed to run test.pystone or print the version in a production app.
38
u/[deleted] Jul 31 '17
I don't think psf builds from the website have all the optimisations on in case of edge case bugs.
I can't read your post due to formatting so I can't tell what your running, so probably an optimised build.
I noticed that Ubuntu's build by Ubuntu is faster in an Ubuntu VM on a Mac, then native Mac build from psf. So the compiler optimisation gain is more than the overhead for running a VM.