r/Python • u/sportifynews • May 14 '21
Discussion Python programming: We want to make the language twice as fast, says its creator
https://www.tectalk.co/python-programming-we-want-to-make-the-language-twice-as-fast-says-its-creator/306
u/Satoshiman256 May 14 '21
Just do less print statements/s
174
u/sizable_data May 14 '21
How else are you supposed to debug it?
74
u/Satoshiman256 May 14 '21
Haha ye. Someone posted a new logging module here a while back and I was like.... but print?
104
u/Napan0s May 14 '21
Logging? What about hundreds of 'I am here' prints
35
u/abrazilianinreddit May 14 '21
12
May 14 '21 edited May 14 '21
Lol mine is more like "we made it to <function>"
1
1
May 15 '21
Ah, Tarja, always been in love with her.
2
u/abrazilianinreddit May 15 '21
Bruh, that's Floor Jansen.
Tarja hasn't been with Nightwish since 2005, and the video is from the Endless Forms Most Beautiful World Tour (2015 and 2016).
2
0
17
u/bin-c May 14 '21
way too verbose imo
i go with:
here1
here2
here3
here4
12
u/Killed_Mufasa May 14 '21
ah yes and don't forget the "WTF HOW DID IT GOT HERE" one
7
7
3
8
3
2
1
May 15 '21
a lot of
if debug: print("function X called with args Y")then you set debug = False and you don't need to get rid of them
12
u/ThatPostingPoster May 14 '21
Loggers are dope. You have to remove print statements, you don't remove smart logging statements. When you remove the print, you end up reading it sometimes. It let's you set up, always log aka print, warning log, and error log. And depending on how you run the file the logger prints those ones.
5
u/Satoshiman256 May 14 '21
Ye I need to use them. Can you suggest one?
29
May 14 '21
Just use the built in logging module. It’s really the best there is. And you can pass the logging information to log files rather than to the console.
To give an example of how useful it is: a project I’m doing has me working with this horrible API. I pass the responses to a log file so I can see what I’m working with at any stage. I also have a logger/log file that checks the server connection and looks out for any exceptions triggered in the code. This is a big project, and I started with print statements but realised they slow me down significantly
2
u/Satoshiman256 May 14 '21
Ok cheers will check it 👍
2
1
0
u/PinBot1138 May 14 '21
The Logbook package makes it even easier.
3
May 14 '21
I disagree, the default logging module is already very simple, flexible and powerful. It should cover 99% of cases. There’s no need to use a wrapper around it
2
u/whateverathrowaway00 May 14 '21
I agree.
That said I work with people who suck and I’d rather they use a wrapper then calling basicConfig on the root logger or other such anti patterns.
If they don’t understand package encapsulation and how to make their logging play nice, the wrappers available are beautiful.
I recommend loguru to coworkers like this all the time and they love it.
→ More replies (1)1
u/PinBot1138 May 15 '21
You disagree with what, exactly? I said that it makes it even easier, and left it at that. Originally, I used to use the default Python logging module and agree with what you’re saying, but IMHO, the Logbook module simplifies the process, especially when collaborating with others. There’s even an advantages blurb that explains some of the enhancements.
0
May 14 '21
There are probably better ones for specific purposes but for the 90% problem it works quite well.
8
u/ThatPostingPoster May 14 '21 edited Nov 02 '21
gg ez deleted cause reasons lets go ok gg is this enough characters to not flag auto mod i hope so lmao
3
u/whateverathrowaway00 May 14 '21
Yup.
And the python built in logger is fantastic. It’s a straight port of log4j and is ridiculously easy to use well once you play with it once or twice.
Pretty much everything I make is filled with log statements and since I always use the module of packaged library with a CLI entry point that runs it, the CLI entry point just attaches the logger to the stdout, enables debug if it sees the flag and boom.
Everything always usable as module in silent mode or verbose with the right entry point.
1
u/m0Xd9LgnF3kKNrj May 14 '21
QueueHandler 4eva
1
u/ThatPostingPoster May 15 '21
QueueHandler
Why? A few seconds of googling doesnt explain why its needed. Never used this, just built in logger.log with threads works fine
0
u/m0Xd9LgnF3kKNrj May 15 '21
What do you mean built-in logger.log with threads?
QueueHandler logs to a queue, which makes all log statements nonblocking. A dedicated thread handles delivering the message to the final destination.
2
u/Timo6506 May 15 '21
Could you link me to that new logging module post?
1
u/Satoshiman256 May 15 '21
I'll try find it for you later when on pc. I think I saved it because I wanted to use it.
1
1
u/Timo6506 May 19 '21
What about now
1
1
3
5
u/Jmortswimmer6 May 14 '21
Vscode has a nice integrated debugger....honestly, you’re just shooting yourself in the foot if not using vscode/pylance/venv
13
u/sizable_data May 14 '21
My preference is pycharm, big debate on my team with VS code users though. A lot of my work is also in Jupyter lab which also supports debugging with the 3.0 release. You need a custom kernel though (xeus Python)
8
u/Jmortswimmer6 May 14 '21
A lot of people love pycharm. I tried it in college....It is just far too nice to have vscode’s C/C++/.sh/etc. support right next to my python, often i have these languages also incorporated into any project. The integrated terminal is a nice feature, and the editor has many other general editing features that make writing python much nicer.
1
u/sizable_data May 14 '21
I’ve stuck with pycharm mostly out of habit, trying vs code is definitely on my todo list
2
-1
u/ThatPostingPoster May 14 '21
If that's not a joke, you use the debugger. You should never be putting down print('here') statements lol. We all started there, but it's about time you finally graduate.
10
May 14 '21 edited May 14 '21
you should use a logging module also. you can't always break in with a debugger especially when things are asynchronous and depend on outside resources.
1
u/ThatPostingPoster May 15 '21
Once you are advanced enough to reach that point sure. If you havent learned it by the time you first use threads you'll realize print sucks haha.
1
May 15 '21
Yeah, I learned the hard way for sure, took me a day or so to figure out why outputs were so messed up.
5
u/whateverathrowaway00 May 14 '21
I mean, the debugger is valuable, but so is proper logging.
I’d argue both are crucial skills to call yourself advanced or professional.
If I had to pick one I’d pick logging every time. That said, I agree with you - the debugger capabilities in python are incredible and once you unwind something with it you never look back lol. Being able to set break points and just poke at the stack frame / variable values is incredible
3
u/ThatPostingPoster May 14 '21
For what possible purpose do you ever need print('here')? We all did it when we were new. But just breakpoint the line next to where you would put that. Ah ok, its there. Logging is a different story and using proper warn/err logs into logfiles is good.
1
→ More replies (6)6
May 14 '21 edited May 14 '21
You laugh but one of the first thing I do when I start a new project is write a dprint() so that I can customize it to my project. I prefer the logger module when things start getting serious but sometimes if I'm just playing around I don't need the added complexity until I really get going.
20
May 14 '21
Anyone know why they don't bring over PyPy stuff and use it? Is CPython architecture just too different?
18
u/Mehdi2277 May 15 '21
pypy does not support c extensions well which breaks a lot of the numerical python ecosystem. It has gotten better at numpy support but still does it in an indirect way. As a side effect while I work on performance sensitive code pypy is unusable as most of the main libraries I care about are heavy users of c extensions.
172
u/sizable_data May 14 '21
In my experience the speed of development/refactor in Python makes up for its execution speeds.
8
2
u/hsvd May 15 '21
Development speed also means you have more time to profile and optimize hot code. The python ecosystem makes this pretty easy (lineprofiler, cython, numba).
3
u/Chinpanze May 14 '21
I'm just a junior developer. But if I were to design a infrastructure from zero, I would go for microservices written in python with refactors in something like rust or go for tasks where it's needed.
90% of stuff can be made in python and run just fine.
1
u/double_en10dre May 15 '21
Yeah that’s a great way to go nowadays, especially if you have everything running in k8s
1
u/koffiezet May 15 '21
I really like Python as a language, but I find myself just picking other alternatives in the last few years, Go has become a pretty damn good-one. It's simple, has tons of libraries (although Python probably still wins out here?), tooling has become pretty competitive (although the delve debugger could use some work) and compiles to native code (which isn't as fast/optimized as C/C++ or Rust, but more than good enough). It wins big time on distribution, where you can build everything as a single zero-dependency file.
And then there's javascript/nodejs and typescript, which I don't like as much, but it's everywhere and don't always have a choice. But the speed of execution is lightyears ahead of Python thanks to v8.
The only times I still use Python these days is when I need to work with some freeform input stuff where strict typing makes things a lot more complex, especially dealing with unknown yaml or json formats. Also, for introducing other people to coding, it's excellent since there are very few lower-level language complications, can be used to teach both standard procedural stuff for basic things and OO, only ducktyping is a bit less practical (which is nicer in typescript & go)
1
May 14 '21
I write financial simulations that currently take about 45 minutes to run on my production machines. If I could cut that by even 20%, it would be a huge improvement.
1
u/Lobbel1992 May 15 '21
@kkirchhoff, can you maybe explain a bit more about your job ? I wanna switch jobs in the future but I don't know which job I want. I have a financial background + programming experience.
1
May 16 '21
I work as a quant analyst and developer. I implement models to value financial derivatives and manage everything but trading for clients’ risk management portfolios.
0
May 14 '21
[deleted]
21
u/LightShadow 3.13-dev in prod May 14 '21
+$500/mo on an AWS bill is an order of magnitude cheaper than a competent, experienced and specialized (C++/Rust), developer.
0
May 14 '21
[deleted]
4
u/nosmokingbandit May 14 '21
Have you considered C#? It's much faster than python and much easier to write and maintain than rust.
1
u/danuker May 15 '21
I have had a sluggish experience with developing C#. Non-MS IDEs are fast but completion/navigation is not as good, and VS itself just... reacts slowly.
1
u/TheTerrasque May 15 '21
VS Code is really gotten good at that lately for .NET Core code. It's still a bit glitchy on .net framework code
1
u/nosmokingbandit May 16 '21
VS is awful unless you need the profiling tools. VSCode and C# are a great pair tho.
0
u/n-of-one May 14 '21
lol rust is widely used in production and has been for years, it is ready foe the prime time. It slowed you down because you had to learn a whole new language and stack, not because of the language itself.
0
May 14 '21
[deleted]
2
u/n-of-one May 14 '21
Rust’s error messages are the best ones I’ve ever encountered and usually spell out what the issue is in plain detail. Sounds like you just didn’t know what you were doing and having to learn on the fly. There’s plenty of help with rust available out there if you just look. Sorry a language that’s been around for 5 years doesn’t have the same stackoverflow presence as languages that have been around for 30.
7
u/white_rob_ May 15 '21
Did you just disagree with them, insult them, then apologize while agreeing with them?
0
May 14 '21
[deleted]
2
u/danuker May 15 '21
Do you have unit tests? Are they testing the insides of the class instead of more stable interfaces?
0
May 15 '21
[deleted]
1
u/danuker May 15 '21
If you draw some lines and say "we'll try not to change these interfaces", and write your tests using those interfaces, it might not be so difficult.
0
May 14 '21
[deleted]
3
u/Ecstatic-Artist May 14 '21
The GIL is definitely an issue, but with async its doable
38
u/66bananasandagrape May 14 '21
Async doesn't really change anything about the GIL.
The two big uses of concurrency are CPU-bound tasks (doing many more computations) and IO-bound tasks (like handling many simultaneous open files or database connections or sockets or waiting on things).
In Python, async and threading are both useful to solve IO-bound workloads. Threading is sometimes easier to implement (just put multiple threads into your program), while async generally gives more structured and maintainable code on a larger scale. With threading, you surround the critical section of your program with locks to opt out of concurrency for that section, whereas with async you opt in to concurrency at each "await" statement. But in any case, these techniques just help programmers write programs where just one CPU can effectively juggle many tasks waiting on it.
On the other hand, Multiprocessing is a python library that will spin up multiple python interpreters that can communicate with one another through the operating system while running on completely different isolated CPUs. This helps many CPU-bound tasks.
All the GIL does is stop multiple processors from being used within the same Python interpreter OS process. In a hypothetical GIL-less world, perhaps threading or async would help CPU-bound tasks as well, but I'm not sure that's really possible with the existing ecosystem of C extensions that rely on the guarantees of the GIL. Right now, the GIL lets, e.g., numpy assume that "this object won't get deleted out from under me while I'm using it".
6
u/bearcatgary May 14 '21
This is about the best explanation I’ve seen on threads, processes, asynch and the GIL. And many people have tried explaining it. Thanks.
3
3
May 14 '21
If you want to get around the GIL you'll have to use Jython, or multiprocessing module or something similar
1
u/danuker May 15 '21
Beware of the cost of spawning a process: it takes about 200-300ms depending on your RAM speed.
1
May 15 '21
Not on linux, python uses fork rather than spawn. Also it depend on the sheer amount of data you are sharing between processes. Also I typically only need to do it with long running processing like splitting up UI from business logic from control logic (or i/o)
1
u/danuker May 15 '21
Long-running processes work great in your use case. Cool!
But what fork does is create (spawn) a new process and copy the memory of the parent (including the python interpreter) into it. For my use case it was not well-suited; I have to rethink the execution structure.
2
May 15 '21
Yeah, it is truly not a one size fits all. Fork is very light weight on linux. If you're doing heavy numerical processing that benefits from multiprocessing and gets past some of the process stuff is to use ray, which uses shared memory but requires quite a bit of setup and thinking about the issue at hand in a new way than either using the threading or multiprocessing libraries. Worth it though if your use case fits it's features.
0
May 14 '21
[deleted]
64
u/O_X_E_Y May 14 '21
If you can, then yeah. I'm fine with just using C++ for stuff that needs to be fast but you're right, if you can have python with speeds that are closer to C... Sign me up 👀
4
4
4
u/rstuart85 May 15 '21
Have you taken a look at Cython? The premise is write mostly python but, when you need speed, write something halfway between C and Python. It even let's you release the gil.
4
u/Thorbinator May 15 '21
I've done cython and found numba to be much easier to use with similar to better speeds. Then again I don't have a C background so your mileage may vary.
2
0
May 14 '21
[deleted]
13
u/whateverathrowaway00 May 14 '21
Sometimes.
Sometimes the communication between python and said compiled extensions aren’t worth it.
Not talking about numpy here numpy is amazing. But some tasks in the end are simply better suited for a compiled language and that’s fine. Pythons amazing it doesn’t have to be for literally everything
16
May 14 '21
Or use numba, which may be even faster, since it compiles specifically for your CPU using all the specific instructions.
1
u/FreeWildbahn May 15 '21
I don't get it. C/C++ can also be compiled for your architecture?
2
u/Thorbinator May 15 '21
Numba has the advantage of having all your glue code be in nice familiar python with whatever type-shifting shenanigans you want, then that one performance critical function can be numba-njit decorated and elevated to C/C++ -like speeds.
It's kind of having your cake and eating it too, for python devs.
1
8
u/xigoi May 14 '21
You mean Nim? It can even interoperate with C/C++ (because it compiles to one of them) and with Python.
1
May 15 '21
I think we’re seeing a lot of cool trends in this space. Kotlin comes to mind for example. It’s definitely more ruby inspired, and it isn’t C++ fast in a lot of cases, but the JVM is no joke.
1
May 14 '21
Try D. Well, D adoption is pitiful, but have you heard about our savior Rust?
3
u/tunisia3507 May 15 '21
Rust may be faster to (safely) develop in than C but development is still an order of magnitude slower than python.
2
May 15 '21
I guess that depends.
Complex projects with a ton of code where people have taken liberties with dynamic days structures become unmaintainable in Python. In Rust you enforce at least some rules and you always know what type you have somewhere, which makes code easier to understand.
I didn't see a huge difference in the 2 languages when trying to do similar things, like a micro service. Maybe I just didn't work so much in Rust.
I can see python being fast to develop in, when it comes to academic problems (leetcode stuff) or when you depend a lot on dynamic code (data analysis and data science).
47
u/JoeUgly May 14 '21
Why does this article sound like a shameless plug for Microsoft? As if people never heard of it.
"Don't forget, real champions eat at Microsoft!"
12
60
u/rothbart_brb May 14 '21
"This is Microsoft's way of giving back to Python"... What an ominous statement. I know the words are "giving back" but I expect them to manifest in a different way... the same way Microsoft embraces outside technology... by getting its tendrils in it and somehow steering it in a way that somehow benefits Microsoft.
47
u/O_X_E_Y May 14 '21
With those speed gains you must now import and use Cortana in every python project you make!
23
2
5
13
u/masteryod May 14 '21
You realize that the creator and BDFL (until 2018) of Python - Guido van Rossum works for Microsoft?
→ More replies (3)-12
u/rothbart_brb May 14 '21
yes, but also, there's the "works for Microsoft" part that I am aware of too. ;-)
2
May 14 '21
Guido doesn't have to work for Microsoft, I suspect if he even got a whiff of them trying to make it Microsoft only he'd bail immediately.
0
u/zeebrow May 15 '21
I started using VS Code recently, so today I was surprised by an update which installed a Python """language server""" extension. Didn't ask for it, no clue how tf it works, (are there open ports on my machine now?? -no, i think) so I dug into the docs a bit...
Seems harmless, dare I say helpful. One of the first things the docs mention is how to uninstall it. So that let my guard down a bit, we're already talking about marriage and kids.
0
u/koffiezet May 15 '21
Why would you not want a python specific language server if you use Python? It's not a 'server' in the sense of a webserver, it's a 'server' process your IDE (vscode in this case) talks to, to get more insight into the source-code. The IDE is language-agnostic, the server does stuff like parse the AST and offer an IDE with more insight into the code-structure, autocomplete, refactoring, ...
1
u/zeebrow May 15 '21
I probably should have said
I started using VS Code recentlyI started using IDEs recentlyWhere I come from, 'server' implies something accepting network connections. lol. I don't know enough windows to know how to monitor system sockets to understand off the cuff how it's being exposed, what's connecting to it, etc.
0
u/TroubledForearm May 16 '21
also Tcpview, netstat etc
1
u/zeebrow May 16 '21
not sure if this was said but I'm on a windows machine. so
netstatonly returns IP, IPv6, ICMP, ICMPv6, TCP, TCPv6, UDP, or UDPv6.....1
u/zeebrow May 16 '21
Also FWIW there is no reason that any language server can't be implemented in tcp.
0
May 14 '21
[deleted]
1
u/koffiezet May 15 '21
Microsoft knows it lost an entire generation of developers in the later years of Ballmer missing out on both mobile and the web, which picked Mac as the de-facto default developer platform. So now they're trying to get them back, and doing pretty well I must say. I'm starting to prefer WSL2+vscode over my mac.
-15
9
May 15 '21
GIL-less threading would be nice. Something like goroutines for Python. Async just doesn’t cut it for a lot of situations. I’m hoping they can do it with sub interpreters, but it’s certainly a complicated thing at this point. Python has been around so long it’s hard to do anything massive without breaking a lot of things.
I’m glad this is finally getting attention. It’s usually the first thing people gripe about.
3
2
u/MOVai May 15 '21
N00b here. Why isn't the multiprocessing module a solution? How do other languages do parallelism in a way that python can't?
4
May 15 '21 edited May 15 '21
I do multiprocessing a lot in Python since it’s really the only true option for running things in parallel. The problem is it’s heavy, and complicated.
Once you fork a new thread you have an entire copy of the parent. That’s inefficient. Then you have to worry about IPC if you need your processes to talk to one another. Queues pipes all that fun stuff. Then you have to make sure you clean up your processes nicely. Did any child processes spawn a new process? What happens if the parent dies before it can clean those up? Zombies! So there’s tons of considerations with multiprocessing.
I’m not super well versed in goroutines except I understand that they intelligently dispatch operations across OS level threads. When you create a thread in Python it’s not moving operations between threads to keep things running. Python doesn’t do that.
2
u/aden1ne May 15 '21
With multiprocessing, one spawns multiple processes, whose memory is completely independent of one another. Spawning processes is expensive, but this may not be such a bottleneck if your processes are long-lived. For me, the real problem with multiprocessing is that you can't _really_ share memory. Your processes can't easily communicate; they invariably do so with some form of pickle which a) is slow, and b) by far not everything can be pickled, and c) the communication either has to go over the network or via some file-based mechanism, but of which are horrendously slow. This means that with multiprocessing one tends to communicate rarely.
Other languages, specifically compiled ones, usually let you share memory. This means both threads can have access to the same objects in memory. This is orders of magnitude faster, but also opens up pandora's box full of memory bugs, concurrency issues and race conditions.
1
u/MOVai May 15 '21
Does SharedMemory help, or does it still leave some omissions?
My impression has been that the multiprocessing module tries to encourage you to use the inbuilt messaging system, and to minimize communication as much as possible. But I don't really have much experience about how practical this approach is for performance-critical applications.
2
u/aden1ne May 15 '21
The
shared_memorymodule solves some issues, but certainly doesn't solve all. It solves the serialization/deserialization problem, but it's still a rather slow IPC method. In fact, some people have found it's in fact slower slower than the naive approach in certain contexts. It's also pretty cumbersome to work with, with some very unpythonic constraints.As a comparison, I made a very simple program in both Rust and Python that spawns 10 threads or processes respectively, each of which send a single hello-world message back to the main thread. The Python example uses the
shared_memorymodule, whereas the Rust example uses channels.Rust example. Also see the Rust Playground snippet
use std::sync::mpsc; use std::thread; fn main() { let arr = [1u8,2,3,4,5,6,7,8,9,10]; let (transmitter, receiver) = mpsc::channel(); for element in arr.iter() { // We have to clone the transmitter and value // because element and transmitter don't live long // enough. let tx = transmitter.clone(); let ne = element.clone(); thread::spawn(move || { let message = format!("Hello from thread {}!", ne); tx.send(message).unwrap(); }); } // Print all messages as they come in. for received_message in receiver { println!("{}", received_message); } }Python example:
from multiprocessing import shared_memory from multiprocessing import Process from multiprocessing.managers import SharedMemoryManager from typing import List def send_message(shared_list: shared_memory.ShareableList, process_n: int) -> None: message = f"Hello from process {process_n}!" # We can't do 'append', we can only mutate an existing index, so you have to # know in advance how many messages you're going to send, or pre-allocate a much # larger block than necessary. shared_list[process_n-1] = message with SharedMemoryManager() as smm: # We must initialize the shared list, and each item in the shared list is of a # rather fixed size and cannot grow, thus initializing with empty string or similar # will raise an error when sending the actual message. Therefore we initialize with # a string that is known to be larger than each message. initial_input = "some_very_long_string_because_the_items_may_not_actually_grow" shared_list = smm.ShareableList([initial_input]*10) processes: List[Process] = [] for i in range(1, 11): process = Process(target=send_message, args=(shared_list, i)) processes.append(process) # Start all processes for p in processes: p.start() # Wait for all processes to complete for p in processes: p.join() for received_message in shared_list: print(received_message)
ShareableListhas some very unpythonic constraints. You need to initialize it up front, and each element has a fixed bytesize, so you can't shove in a larger element. Additionally, it's limited to 10 MB, and only the builtin primitives are allowed (str, int, float, bool, bytes and None). Feels like writing C rather than python.1
u/MOVai May 15 '21
The shared_memory module solves some issues, but certainly doesn't solve all. It solves the serialization/deserialization problem, but it's still a rather slow IPC method. In fact, some people have found it's in fact slower slower than the naive approach in certain contexts.
I think I see what's going on here: The Queue implementation is slicing the data before delivering it to the worker threads. There, it can optimize the hell out of it, which is why increasing the size from 99 to 99999 only increases the runtime by a factor of 2.9. That means it's 352 times more efficient. The implmenentation is sublinear.
The SharedMemory implementation, on the other hand, is preventing the optimizer from working properly. That's because the worker needs to read the memory every iteration, as it can never be sure that the data hasn't changed under its nose. This also has the side-effect of obliterating your cache hits. As a consequence, the process with 99999 ints is 10 times less efficient when the problem gets bigger, i.e. superlinear.
This isn't showing any problem with SharedMemory in Python. It's a nice demo of what can go wrong when peopl naively use paralellism without understanding the complexity. The exact same thing happens when you use pointers in C.
You could argue that the limitations improved performance, as they encouraged programmers to keep data sharing to an absolute minimum, and avoid premature optimization.
My (N00bish) take is that if Python's inter-process-communication is bottlenecking your performance, then chances are you're doing parallel computing wrong an should work on your algorithm.
But again, I'm just a N00b and would appreciate if someone with experience could explain what real-world algorithms actually have some intractable performance issues due to Pyhon's multiprocessing model.
2
u/Mehdi2277 May 16 '21
Data transfers is a pretty common bottleneck for parallel heavy code. GPUs are probably poster child here as many ML workloads get bottlenecked by cpu to gpu transfers leading to a lot of hardware work on increasing throughput of data transfers.
If you try applying similar algorithms on high core cpu you'd likely need to be careful of process communication. Memory/transfers are often the slowest parts of computations and part of why keeping things in L1-L3 caches and then ram is very important. Although personal experience is most of the time people care about this they write c++ and then use something like pybind to wrap it in python. Stuff like cython/numba help but having used them a good numba/cython implementation sped by code up heavily (factor of 10x+) but a simple c++ implementation still beat it by another several time speed up. For simple enough numpy code maybe numba will equal or come close to c++ but for longer chunks it'll likely just lose.
Even wrapping is sometimes not good enough if you care strongly on performance where an increase in time by say 30% is bad. In those cases you end up fully giving up python and just having c++. That is uncommon but I sometimes see it for large cpu heavy workloads that can cost millions of compute per year that saving 30% is worth it. It's why it's common to take an ml model trained in python and then export it to a model graph that you deploy in pure c++. For a small company/medium traffic this is an unnecessary optimization.
1
u/backtickbot May 15 '21
11
11
May 14 '21
<whispers> get rid of the GIL
9
u/AReluctantRedditor May 14 '21
God they’re trying with subinterpeters. Talk python to me had a short discussion about it a few weeks ago
3
4
u/ventuspilot May 15 '21
Strange times. From the article
open-source friendly Microsoft
and I'm like well currently that's not wrong...
3
u/awesomeprogramer May 15 '21
Microsoft is also focussing on securing the main package repository PyPI (Python Package Index).
What do they mean by this? How can they possibly do this? Last thing we'd want is for Microsoft to dictate what can and can't go on PyPI. We have the app store for that!
2
6
6
4
2
u/Deezl-Vegas May 15 '21
Just to clarify, L1-cache optimized code can be up to 200 times faster than Python. It's not clear to me that a 2x speedup will make a large difference in language acceptance or usability.
1
u/Fantastic-Orange-155 May 20 '21
Let:
xi,j ∈{0,1}, be a binary variable with xi,j = 1 indicating Arc (i,j)∈A is used in the solution
and xi,j = 0 otherwise;
p_Kbe the set of all simple paths with exactly K vertices; and
= (i1…..iK) ∈PK be an arbitrary simple path with exactly K vertices, i.e., the arcs in thepath are: (i1,i2); (i2,i3),…, (iK-1, iK)
xij ∈{0,1}, ∀(i,j)∈A
Constraint (1)
Can someone write a OPL CPLEX model for the above constraint
1
-2
-11
u/drbobb May 14 '21
Twice as fast is not enough — that would mean it's still far behind javascript (v8).
9
u/O_X_E_Y May 14 '21
No you think it should stay as is? This is a very confusing statement
-3
u/drbobb May 14 '21
What's confusing about saying that python would need to be much more than twice as fast to match JS? Of course the python runtime ought to be improved, it just has a long way to go.
3
u/Deemonfire May 14 '21
Their plans extend far beyond 2x speedup. 2x is what they expect to be able to acheive by python 3.11
1
u/drbobb May 14 '21
Good news.
Trying to speed up python code with numba is quite frustrating, because much of python isn't supported by numba. PyPy has its compatibility woes as well.
2
0
u/Awkward_Tour8180 May 14 '21
I had my own share of experience using print statements for debugging , unless you have a tea automatic tool in the PR to catch it , my debug print went to prod code - so unlucky . The good side is it’s the admin module ,
Print statements for debugging is always a bad approach , logging libraries will make you feel safe and even if your forget to remove it still be compartmentalizes to DEBUG, INFO, ERROR as a base to start with
-11
u/60secs May 14 '21 edited May 14 '21
Great! Python would only be 10x to 20x slower than a performant language.
-1
-2
244
u/Jugad Py3 ftw May 14 '21 edited May 19 '21
CPython stands for "the Python interpreter written in C language", not "Core Python". On similar naming lines, there is also Jython, written in Java, etc, and then there is Cython, which compiles certain annotated python files into C (for speedups).