r/Python • • Oct 24 '22

News Python 3.11 is out! Huzzah!

https://www.python.org/downloads/release/python-3110/

Some highlights from the release notes:

PERFORMANCE: 10-60% faster code, for free!

ERROR HANDLING: Exception groups and except* syntax. Also includes precise error locations in tracebacks.

ASYNCIO: Task groups

TOML: Ability to parse TOML is part of the standard library.

REGEX: Atomic grouping and possessive quantifiers are now supported

Plus changes to typing and a lot more. Congrats to everyone that worked hard to make this happen. Your work is helping millions of people to build awesome stuff. 🎉

1.3k Upvotes

233 comments sorted by

296

u/staticcast Oct 24 '22 edited Oct 25 '22

PERFORMANCE: 10-60% faster code, for free!

Wait what ? Seriously ?

4

u/OpeningJump Oct 25 '22

How is it achieved tho?

12

u/GettingBlockered Oct 25 '22

The CPython team has been doing significant optimization work to improve things, like reducing the overhead of function calls, zero cost exception handling and a lot more. This doesn’t impact libraries that are optimized with C (Numpy). But pure Python code should get some nice speed ups!

275

u/-LeopardShark- Oct 24 '22 edited Oct 25 '22

Yes. The only real caveat is that if your code already spends much of its time in C functions (e.g. NumPy) or doing IO, you won't gain a lot. But for interpreting Python itself, it's a pretty nice boost. There'll probably be more to come in 3.12 as well.

105

u/[deleted] Oct 25 '22

Their goal is ~5x of 3.9 levels in 4-5 years IIRC

-20

u/Voxandr Oct 25 '22

Their goal is already achieved by pypy team by the way.

27

u/[deleted] Oct 25 '22

While true, there are some compatibility issues with pypy :( No Black or Orjson, for example.

9

u/zurtex Oct 25 '22

Getting big performance improvements in Pypy is very situational.

I've definitely looked at it for large pure Python code bases before where squeezing a little extra performance out of it was helpful. When I ran it Pypy it was consistently just over 1% slower.

4

u/Voxandr Oct 25 '22

Might be long ago , it might be when PyPy python 3.x version was not optimized and no cpyext optimization is there. We have 100k users with 20-30k concurrent connection running on asynchronous network system .

  • 6x lesser memory used

  • 2-3x performance on average

  • cheap hardware

We will provide proper benchmark when free

2

u/zurtex Oct 25 '22

It was in 2019, I wasn't using any packages that were using the C API, and according to Pypys website at the time it should have been well optimized for Python 3.

I would spend more time looking at it but whenever I look at how to debug a Pypy issue and inspect the internals of the Pypy source code generation my head starts to hurt.

1

u/Voxandr Oct 26 '22

2019 was not optimized for cpyext and pypy3 support is still in early optimization, still slower than pypy2. Cypext support start to mature on late 2020-2021, and can run numpy at same speed around 2022.

What are you saying about PyPy source code generation? Are you mistaking PyPy with nutika or something? PyPy is JIT not source code generator.

→ More replies (1)

2

u/PaintItPurple Oct 25 '22 edited Oct 25 '22

Was this a long-running process or a frequently called script? In my experience, Pypy is everything people claim for the first case, but terrible for the latter case. In more concrete terms, Pypy's strength is that it's really good at optimizing things that are called in loops.

2

u/zurtex Oct 25 '22

I guess it would technically be a frequently called script.

But it ran for over 5 minutes and had some hot loops in it so I hoped for at least some minor performance improvement.

1

u/pepoluan Oct 31 '22

PyPy grinds to molasses when doing multiprocessing, sadly.

And it needs cffi to go to C, and that adds quite a bit of overhead.

13

u/FruitierGnome Oct 25 '22

So if having a long initial wait time loading a csv file into my program this would potentially be faster? Or am I misreading this? I'm pretty new to this.

32

u/-LeopardShark- Oct 25 '22

I don't think loading CSVs will gain much, sadly.

3

u/Wilfred-kun Oct 25 '22

Time to use TOML instead :P

35

u/yvrelna Oct 25 '22

Depends on what part of CSV loading.

If you're talking about the call to csv.reader() itself, then no, that's already calling into a C library so you won't likely get much performance improvements.

But if you're talking about the code that's processing the rows of data line by line, then yes, that is definitely going to benefit from the improvements.

11

u/graphicteadatasci Oct 25 '22

Use .parquet files when you can. Much faster loading, smaller storage, saves types instead having you cast or infer them when you load something.

7

u/BobHogan Oct 25 '22

Parquet is not the solution to everything. We use it at my work and its a fucking nightmare and I'd love to see it burned to the ground

3

u/madness_of_the_order Oct 25 '22

Can you elaborate?

5

u/gagarin_kid Oct 25 '22

For small files where humans want to inspect data, using parquet is pain in the ass because you cannot open it in a text editor - you have to load it in pandas, see which columns you have, navigate in code to a particular cell/row... etc.

Of course for big data I fully understand the motivation but not for each problem

2

u/madness_of_the_order Oct 26 '22

I’m not telling you should use parquet for everything, but you can try dtale for interactive exploration

3

u/cmcclu5 Oct 25 '22

Parquet is also a pain in the ass when you want to move between systems e.g., from a data feed into a relational database. Python typing does NOT play well with field types in relational databases when saving to parquet and then copying from said parquet into Redshift. Learned that the hard way in the past. It’s several times faster than CSV, though. I just compromised and used JSON formats. Decent size improvement with a similar speed to parquet when writing from Python or reading to a db.

1

u/madness_of_the_order Oct 26 '22

How untyped format helped you solve a typing problem?

→ More replies (5)

1

u/BobHogan Oct 26 '22

We run into constant issues with parquet in our product, to the point that we've completely stripped it out in newer versions in favor of other solutions which I am not allowed to discuss publicly :(

We see parquet metadata get corrupted fairly regularly, being able to inspect what data is actually in the parquet files to track down issues is significantly more annoying and involved than it should be. And we've also run into limitations in the format itself that cause it to just shit itself and fail, limitations that are both fairly arbitrary and should be easy for the format to work around if the people that wrote it cared at all, but they don't. Overall its been an incredibly fragile format that makes it harder than it needs to be to work with the actual data compared to other formats, doesn't provide any significant performance improvements we've been able to measure, and breaks randomly.

1

u/madness_of_the_order Oct 26 '22

This sounds like it could be a really interesting blog post with concrete examples

1

u/graphicteadatasci Oct 27 '22

fastparquet says you can append to a file but it is a terrible lie.

What else?

9

u/fukitol- Oct 25 '22

Loading of a csv into memory is going to depend far more on the size and speed of your memory and the speed of your disk. Negligible amounts of time will be spent in processing, which is where an application level performance boost would be had.

1

u/[deleted] Oct 25 '22

How big is the csv and how long does it take?

2

u/pepoluan Oct 31 '22

Depends on how the I/O happens.

If you do a lot of non-sequential I/O, thus leveraging async, you can get quite a speedup.

I have a script that pulls data from an API using coordinates. Then the data gets fed into a pool of multiprocessing workers. The workers use do some pillow processing.

I see a speedup between 10%-30% in total time.

For the async retrieval part, I see speedup between 25%-50%.

47

u/reivax Oct 24 '22

Faster frame allocations mean tightly recursive functions will be faster.

8

u/Pleasant-Cow-3898 Oct 25 '22

Nice excited to test this out!

-5

u/aitchnyu Oct 25 '22

Umm, what recursive functions do you use?

→ More replies (2)

12

u/wind_dude Oct 25 '22

damn! I have to benchmark this for my ETL pipelines ASAP! Could be a bigger boost than I've been hunting for from spacy and other libs I'm using.

2

u/myroommateisasian Oct 25 '22

There is no way...

2

u/DanganD Oct 25 '22

Yea, that's awesome.. Peoples biggest gripe is it's speed right?

1

u/o11c Oct 26 '22

They finally bothered to start implementing the usual stuff a real VM has.

For a long time, it was explicitly a goal of CPython to be "simple" even if that meant avoiding major optimizations.

1

u/pepoluan Oct 31 '22

Probably because before they have to maintain things in their spare time, so making things simple is of the utmost importance.

Now with lots of companies actually paying Python core devs some money to maintain Python, they can implement more complicated optimization methods. Because they're paid to do that now.

1

u/Daik_Reddit Nov 18 '22

Oh no! So if now i write time.sleep(6) It will lasts only 2,4 seconds? This will mess all my beautifull code. I didn't pay my license for this stupid features. Want my money back.

-24

u/Java-Zorbing Oct 25 '22

Don't worry, it's still up to 200x slower then most languages.

8

u/kalebludlow Oct 25 '22

Epic own bro

3

u/-LeopardShark- Oct 25 '22 edited Oct 25 '22

While technically true, ‘up to 200x slower’ is pretty meaningless. It’s closer to 10× on average.

3

u/viksi Oct 25 '22

Just get faster CPUs.

Hardware is a lot cheaper than developer time

→ More replies (4)

9

u/[deleted] Oct 25 '22

[removed] — view removed comment

15

u/Username_RANDINT Oct 25 '22

You'd have to profile to be absolutely sure. What's the slow part? The network connection? Rendering of the webpage? Database queries? There's so much going on in a webapp that these new speedups might be just a very little part of the work.

2

u/[deleted] Oct 25 '22

[removed] — view removed comment

9

u/GettingBlockered Oct 25 '22

“The slow part” is relative to your app. Profiling let’s you estimate what the actual impact will be for your users, and let’s you make better decisions about how you invest your time.

If it takes 1 second to load a page on your app, but the Python code is only responsible for 100ms of that, then you might see gains between 10-60ms (1-6% of total time). Which is still very nice, but quite different from the headline.

And every project is different. If it’s a 1 hour job to upgrade Python, then that’s awesome, seems like a no-brainer. But if it’s +20 hours due to some unknown complexities… dependency issues, build pipelines, tests, etc., then you might consider investing that time elsewhere, like optimizing JS payloads or image sizes, etc.

Anyway, i hope it’s an easy upgrade for you! I’m stoked the Python team is investing time on performance. It benefits literally everyone.

2

u/Conscious-Ball8373 Oct 25 '22

Not the GP commenter but I'm in a similar boat. I have a website. The front-end is nodejs, static files are served by nginx, the database is fast and local but the API the front-end uses is python+flask. The performance gain won't impact our initial page load time but I think it will be a significant boost to the responsiveness of the site as it's used.

4

u/[deleted] Oct 25 '22

[removed] — view removed comment

2

u/bfcdf3e Oct 25 '22

Dangerous assumption 🥲

2

u/tuckmuck203 Oct 26 '22

Don't do it yet if your site is more than a hobby project. You typically want to wait 3 to 4 months after a language release in order to let libraries patch and bugs to get sorted out. I've already seen at least one post about SQLalchemy having issues with 3.11 and postgresql drivers

21

u/_morgs_ Oct 25 '22

It should help, but there are many factors involved with website speed.

Caching, database (server) performance, database query complexity, front end size and performance, any APIs you might use...

4

u/yvrelna Oct 25 '22

If you've already optimised all the database and network queries, and adding appropriate caching whenever it makes sense, then at some point Python performance is the bottleneck, then yeah, you'll see a boost then.

In most cases though, most web applications aren't constrained by the speed of Python interpreter itself. It's usually the database and network performance that's the workhorse of most web application.

3

u/SilkTouchm Oct 25 '22

It won't. You're bottlenecked by i/o, not processing.

24

u/leppardfan Oct 24 '22

When will Ananconda ship 3.11? They just shipped 3.10 last week!

4

u/GettingBlockered Oct 25 '22

Live on Conda now! Huzzah!

https://anaconda.org/conda-forge/python

1

u/snake_case_captain Oct 25 '22

Pardon me for my blatant noobiness, do you have any idea why "conda install -c conda-forge python" won't gave me more than 3.9 ?

1

u/GettingBlockered Oct 25 '22

Not sure. Are you trying to install in an existing environment or a new environment?

I just made a new env with this command, worked perfectly: conda create --name py311 python=3.11

Maybe adding the =3.11 would do the trick for you.

1

u/snake_case_captain Oct 25 '22

It seems that the problem is within python=3.11 actually :

PackagesNotFoundError: The following packages are not available from current channels:  - python=3.11

21

u/BertShirt Oct 24 '22

3.10 has been available for a while on conda-forge. I suspect 3.11 will be available before the end of the week.

22

u/AKiss20 Oct 25 '22

Honestly I would just ditch anaconda. I used it for a while but the delay in getting updates and it’s overall clunkiness was too annoying.

9

u/BertShirt Oct 25 '22

I agree with you to a certain extent. Ever since Gohlke wheels stopped getting updates it has been hard to get numpy/scipy +MKL updates without conda.

If you're like me and you hate the bloat and extra bulk and overall sluggishness of conda you should look into mamba-forge/micromamba.

It doesn't solve all the conda problems, packages still aren't up to date as fast as pypi and still missing a lot but its a much better experience than Anaconda as a whole.

1

u/AKiss20 Oct 25 '22

I considered that for a bit but the multi-role of Vonda wasn’t necessary for me and I was already using pyenv for version management so I switched to venv (pyenv virtualenv plugin) and just used that instead. It’s worked for me thus far, granted my needs aren’t demanding or sophisticated.

4

u/zurtex Oct 25 '22

Conda has historically been great for bootstrapping a Python environment with any non Python packages you need.

Whereas Anaconda has historically been great for giving to a data scientist and showing them Jupyter and hoping they don't have any other requirements.

If you have any other use case it's probably time to learn to install Python and it's packages another way.

3

u/zurtex Oct 25 '22

Anaconda tends to wait until they are confident all the big data science packages are compatible, this takes about a year

1

u/BertShirt Oct 25 '22

3.11 is now available on conda forge https://anaconda.org/conda-forge/python

2

u/sapphire_striker Oct 25 '22

Can i update it with conda yet?

243

u/throwawaylurker012 Oct 24 '22

Wake up babe

New python just dropped

2

u/Achillesbellybutton Oct 25 '22

...and this one hits different.

125

u/M33PIT Oct 25 '22

Nice, maybe aws will support it by 2025

1

u/Texas_Technician Oct 25 '22

Is there a good tutorial on how to implement Azure scripts? Or a good YouTube who focuses on Azure development you can recommend?

1

u/[deleted] Oct 25 '22

Aws does 3.10

15

u/Gamecrazy721 Oct 25 '22

Lambda doesn't support 3.10, you have to install it on a container

→ More replies (6)

0

u/[deleted] Oct 25 '22

[deleted]

20

u/one-human-being Oct 25 '22

Azure App service just got 3.10 support

70

u/BeeApiary Oct 25 '22

Windows 3.11 was the first version of windows that was actually usable, so for numerologists this is a good portent.

60

u/mvdw73 Oct 25 '22

Ah yes, Microsoft Windows Version Numbering.

1990: "We use the Major-dot-minor version scheme - it's what all the cool kids use."

1995: "We should switch to a year scheme, so that people know when the software was released"

2000: "Oops, now we'll have to go to 4-digit years, otherwise we'll be releasing Windows 00 next"

2003: "I don't like that - let's use a cool set of letters and make the next version 'XP'"

2009: "Meh, whatever, let's just use numbers again. Start with 7! Because it might be version 7 we're up to, and Mac is up to 10 so we have to catch up!"

2012: "Version 8!! Consecutive consistent numbers for the first time since 1998!"

2015: "We can't have version 9. Have to go to 10. Mac is at 10!! We have to be at least at 10!!"

10

u/misaprop Oct 25 '22

why didn't they like version 9?

23

u/bakery2k Oct 25 '22

Because Windows 10 was supposed to be the “forever version” of Windows - so they couldn’t call it version 9 because that would be behind macOS, which had version 10 as its “forever version”.

Then, after Apple announced macOS 11, Microsoft announced Windows 11.

2

u/Texas_Technician Oct 25 '22

Ya, I remember that line too. Should started a betting pool.

16

u/reallyserious Oct 25 '22

They had already had Windows 95, so retards that checked for a 9 in the first position would have their code behave strange.

18

u/mgedmin Oct 25 '22

Windows 95 and Windows 98.

4

u/Texas_Technician Oct 25 '22

No way that's the real answer.

7

u/Brekkjern Oct 25 '22

I remember some Dev from MS talking about it back in the days, and then someone posting a GitHub search that did in fact show a lot of cases where people did that, so the story is at least plausible

1

u/troyunrau ... Oct 26 '22

Official reasons are quoted as marketing. The thing where they check if a version starts with a 9 is often suggested as the true reason, but no one can ever find a quote from anyone within MS to that effect.

1

u/ThroawayPartyer Nov 06 '22

iPhone also skipped 9, so maybe companies think that number is harder to market for some reason. 9 is personally my least favorite number, but I always thought it was just me...

1

u/spiker611 Oct 25 '22

One theory: because so much software was written to check for "9" in the first part of the version (and then either "5" or "8").

1

u/Java-Zorbing Oct 25 '22

What about windows11?

5

u/methnbeer Oct 25 '22

What about ME or Vista

3

u/Hiyaro Oct 25 '22

We don't talk about vista

1

u/chears2surfers Oct 25 '22

That's interesting! I didn't know that. Thanks! :)

0

u/[deleted] Oct 25 '22

[deleted]

13

u/alcalde Oct 25 '22

Windows For Workgroups!

6

u/spuds_in_town Oct 25 '22

Or as it was known in the trade, Windows for Warehouses

44

u/MrMxylptlyk Oct 24 '22

Woo excited about toml.

20

u/midnitte Oct 24 '22

Will be curious to see if any projects pick up toml over existing yaml support...

29

u/MrMxylptlyk Oct 24 '22

If you are in yaml already, probs not. I'm using ini files in my projects with config parser.. Maybe I can upgrade lol.

15

u/[deleted] Oct 25 '22

Is there any reason to switch to TOML if you're currently fine with YAML? The main difference I see is TOML is more explicit/structured. Are there any other advantages?

21

u/mgedmin Oct 25 '22

YAML is a mixed bag of niceties and traps. If you're a certain kind of person (a pedantic like me) who would never even think of omitting the space after a : or a -, then YAML will work for you great. If you're a regular person, you'll trip over indentation or missing spaces or the 2-letter country code for Norway being parsed as the boolean False (depending on the YAML standard version that your parser chose to support!), or hours:minutes getting automagically converted into minutes.

I haven't used TOML much, but from what I know it has fewer traps. It's stricter, though, more explicit syntax is required. I could probably get used to it, if I wanted to.

14

u/fatterSurfer Oct 25 '22

YAML has always struggled with security problems (insecure loading has been the default in python for ages). TOML is also likely faster, since it's a much simpler spec, though that's entirely speculation on my part and nothing I've seen benchmarked. But more and more parts of the python ecosystem have been moving to TOML (pyproject.toml being the most visible example), so that might be a reason unto itself

9

u/D-K-BO Oct 25 '22

YAML has some problems with ambiguous syntax. https://hitchdev.com/strictyaml/why/implicit-typing-removed/

4

u/angellus Oct 25 '22

Toml is far more readable then yaml.

2

u/Hatcherboy Oct 25 '22

Wearing my spacebar out!

2

u/zurtex Oct 25 '22

Or using JSON for config. Omg the horror of a large JSON config file, I'm going to be pushing hard to move to TOML.

3

u/jsalsman Oct 25 '22

I hope so. YAML is extremely difficult in more than a few common cases.

4

u/jantari Oct 25 '22

As someone who has literally never struggled with YAML, care to share a few examples?

I love the fact that it's simple, indented hierarchically, that I can use single-line lists for short lists and also multi-line lists where needed and the line-folding and literal-string operators >, >-, |, |- and |4 etc. are an absolute godsend compared to horriblly limited config languages such as HCL

5

u/-LeopardShark- Oct 25 '22

It has some features of negative value, such as those removed by StrictYAML.

2

u/jantari Oct 25 '22

Ok those are good points. I guess because I'm usually writing my own yaml I don't come across these problems but if I had to read other people's yaml and it made use of these I'd definitely complain too. Thanks

1

u/tunisia3507 Oct 26 '22

What I want is strictyaml plus TOML's type system.

2

u/jsalsman Oct 25 '22

The most recent example that comes to mind is https://github.com/GoogleCloudPlatform/batch-samples/blob/HEAD/primegen/workflow.yaml

that's about equivalent to around a dozen lines of shell.

2

u/o11c Oct 26 '22

Nobody ever said Yaml was a programming language. It's a markup language.

(ironically, I think XSLT is actually pretty neat)

1

u/tunisia3507 Oct 26 '22

I thought it Ain't a Markup Language?

1

u/o11c Oct 26 '22

Whenever an acronym claims it's not something, that means it really is (at least, for practical purposes).

  • GNU is, in fact, Unix (for practical purposes)

  • WINE is, in fact, an emulator (for practical purposes)

  • YAML is, in fact, a markup language (for practical purposes)

2

u/pepoluan Oct 31 '22

That's not YAML's fault.

Someone's trying to make a "declarative format" into a "procedural format", it will never end well.

3

u/trevg_123 Oct 25 '22

TOML is great for fairly simple things, like ini-style config files. However, anything with more than a few levels of nesting gets ugly and tough to read & write, so I suspect that anything using deeply-nested configuration will probably stick with yaml.

As somebody pointed out, yaml isn't in std either (and will likely remain that way - toml only got in because of pyproject.toml) and std toml can't serialize (will likely remain that way, since toml is largely config files) so it's not an apples to apples comparison anyway.

2

u/oramirite Oct 25 '22 edited Oct 25 '22

I prefer YAML honestly, wish there was a better builtin

EDIT: YANK, the lesser known subspec of YAML

11

u/darrenhoyland Oct 25 '22

Any chance you could provide a link so I don't have to Google "yank python"?

2

u/oramirite Oct 25 '22

Absolutely NOT

1

u/AlphaDolby Oct 26 '22

Don't pretend you are not Googling it all the time, anyway.

2

u/0tting Oct 25 '22

A mixed bag probably. No external dependency is great. But the new TOML function does not support flattening dicts back to TOML, so for example, you cannot write changes back to file.

1

u/mgedmin Oct 25 '22

There's no YAML parser in the Python standard library, last I checked?

2

u/midnitte Oct 25 '22

I mean projects that already use yaml since the standard library doesn't include a parser for it, and now includes one for toml*, apologies

1

u/pepoluan Oct 31 '22

Well, if they already require a YAML parser before, they probably will still require a YAML parser now.

Especially since converting YAML to TOML is not a trivial job.

14

u/tjt5754 Oct 25 '22

I'm personally interested in playing with Exception Notes from PEP-678

https://peps.python.org/pep-0678/

I saw it mentioned in ArjanCodes video about 3.11 but he glossed over it. I'm not SURE what specific need it fills, but I imagine there will be instances that this will be useful.

Similar to for/else... when I first saw it I wasn't sure I'd need it, but I find myself using it often.

4

u/mgedmin Oct 25 '22

I imagine it could be useful when you're writing interpreters, e.g. for a template language. When an exception happens it'd be nice to know that it happened while rendering templates/foo.j2 line 42, included from templates/bar.j2 line 55.

1

u/drenzorz Oct 25 '22

I would assume it's helpful when building tests/testing libraries. If you are running a bunch of automated tests adding some notes to help locate the problem could be really useful.

6

u/tjt5754 Oct 25 '22

Funny, I was thinking of deeply buried exceptions that you think won't ever happen, but you can add detail to so you know what weird scenario happened.

You know that code where you put in a comment that says "I don't think this is possible, but just in case..."

1

u/[deleted] Oct 25 '22

While this is definitely an improvement, I've used the structlog lib with structlog.processors.dict_tracebacks to get JSON styled stack traces. It's pretty nice when forwarded to something like Elastic, so you can query it via Kibana.

Centralized structured logging is nice.

44

u/trevg_123 Oct 25 '22 edited Oct 25 '22

Does this mean the maintainer of Flake8 will stop being a stubborn oaf and consider pyproject.toml support?

Seriously… the main objection was no standard support for toml (now solved), and complaints about build isolation being broken by pyproject.toml. Which, notably, doesn’t force anyone to use that config option. More importantly though, the behavior he describes wanting is literally incorrect usage of the pip build system

Mega annoying.

edit: yes, I know this is somewhat satirical. My goal was simply to share bitterness that maintenance of a core python project seems to be steered by the hubris of a maintainer, rather than community desires. The ruff alternative suggested by u/ballagarba looks like a great option that avoids the flake8 project.

20

u/stalkinplatypus Oct 25 '22

If you're sufficiently annoyed to accept an extra dependency over this (like I was), there's this: https://pypi.org/project/Flake8-pyproject/

16

u/ballagarba Oct 25 '22

I suspect Ruff[1] will replace Flake8 before that happens.

[1] https://github.com/charliermarsh/ruff

2

u/trevg_123 Oct 25 '22

I have to say, that project looks incredible, and the speed boost is probably great for huge projects. Plus the checks like docstrings builtins bugbear all in one is a lot nicer than just always remembering to add those packages.

Just as many stars on github... I really wouldn't mind if this becomes the de facto linter.

1

u/o11c Oct 26 '22

On the one hand, yay, flake8 sucks.

On the other hand, I really don't like when "python" packages turn out to install entire foreign runtimes.

On the third hand, it fails to solve 2 of the biggest concrete problems with flake: use of arcane error numbers instead of strings, and inability to filter diagnostics in a sane way.

(no, putting a comment at the end of every line is not sane, and in fact will often cause further linting errors! Do it like pylint, allowing per-file and per-block suppressions, please)

6

u/mgedmin Oct 25 '22

This doesn't look like a genuine question, but I'll bite. tomllib being in the standard library for 3.11 doesn't help projects that need to support older Python versions.

3

u/trevg_123 Oct 25 '22 edited Oct 25 '22

Of course it doesn't help older versions, but the suggestion was given to deprecate setup.cfg support and only support pyproject.toml in >=3.11 (not in the linked thread, but on some follow up). It was strongly shut down, for the other bad reason.

Anyway, you're correct that it's a loaded question. I just wished to share my irritation that the maintainer of a core python library seems to be making decisions rooted in hubris rather than community needs & desires.

2

u/mgedmin Oct 26 '22

As a member of the community in question I object to the statement that deprecating setup.cfg support in flake8 is something the community needs and desires.

Let people continue using setup.cfg if they want to.

2

u/Lindby Oct 25 '22

We simply stopped using flake8. There are other alternatives.

3

u/wdroz Oct 25 '22

I use only pyproject.toml for a while now. This work so perfectly, no more setup.py, setup.cfg, requirements.txt and requirements-dev.txt!

For the flake8 discussion, this seem to be from 2021. In 2021 pyproject.toml usage without third-party (like poetry) wasn't really "ready". It's only recently, with the latest versions of pip, that we can really use pyproject.toml without heavy coupled it with third-party.

3

u/trevg_123 Oct 25 '22

You're correct that the linked discussion is from before toml was announced as part of 3.11. But the discussion continued elsewhere once it was known, with similar results.

12

u/[deleted] Oct 25 '22

What’s with the funky black hole business?

1

u/[deleted] Oct 25 '22

Nerds being nerds

6

u/Tamagotono Oct 25 '22

Yay! Python for workgroups

11

u/shinitakunai Oct 25 '22

Cool but I depend on AWS and they don't even support 3.10 on lambdas. And on the default repo for amazon linux 2 they are on python 3.8 which is a BIG facepalm for lightsail users.

5

u/Suspicious_Compote56 Oct 25 '22

I would like to see built in exe packaging like golang. And auto conversion to cython

5

u/_limitless_ Oct 25 '22

return joy

4

u/aplund Oct 25 '22

The aside on the Kerr metric is all true.

4

u/dxa58 Oct 25 '22

Self type has been added! I'm so glad I can annotate it easily now

4

u/pepoluan Oct 31 '22

I am doing some data pulling and processing, so I use a lot of async & multiprocessing, and I can confirm 3.11 runs faster than 3.10, by approximately 10-30%.

Impressive!

2

u/[deleted] Oct 31 '22

And given how easier they made async programming in 3.11, it's really impressive. Especially since async/multiprocessing paradigms are rather fast already in 3.10

1

u/pepoluan Oct 31 '22

Indeed! I had the figurative jaw-dropped-to-the-floor moment the first time I use asyncio.

Then another jaw-dropped-to-the-floor moment seeing how 3.11 is able to speed async further.

asyncio + httpx + CPython 3.11 == Lots Of Wins!

This is on Windows, though. I wonder how it will compare if I use uvloop (which runs only on Linux sadly)...

1

u/[deleted] Oct 31 '22

I use Linux as a daily driver and I never needed uvloop to be honest. I'm sure some people need it but unless you want something ultra fast, you don't need it (in my opinion).

But hello httpx user. I thought most people jumped on aiohttp and forsook this beautiful library.

1

u/pepoluan Nov 07 '22

I use httpx because aiohttp doesn't seem to support HTTP/2 😉

1

u/Hyedwtditpm Oct 25 '22

Anaconda doesn't update to it.

Does anyone know when will it get the update?

1

u/GettingBlockered Oct 25 '22

It’s live on conda forge: https://anaconda.org/conda-forge/python

does this work? conda install -c conda-forge python

3

u/MeroLegend4 Oct 25 '22

Not only Python, but the whole ecosystem 👍👍👍

3

u/arkster Oct 25 '22

Too bad we are stuck with 3.9 on Aws lambda.

12

u/Wootai Oct 24 '22

Next thing you turn around and find the person is you
Thought a freak might be the thing
But you know this will pass, so just get off of your ass
You've got to trust your instinct. And let go of regret

5

u/wegoingfox Oct 25 '22

3.11 🤝

11

u/[deleted] Oct 25 '22

Finally we can stop getting punted by C++ Elitists for our speed.

45

u/[deleted] Oct 25 '22

Ummm I don't mean to discourage you, but Python is nearly indefinitely slower than C++ which is totally fine, it was never meant to be be an embedded system hyper fast language. Python becoming 2x faster (the aim of Faster CPython project) will still nearly indefinitely slower than C++

That being said, you nearly never need this speed if you can write decent code. You can bring moderate servers down with a decent async Python tool. Also Python makes it really easy to write efficient decent code, which is far from truth for C++

So yes C++ will always win in a field that's not always needed.

8

u/[deleted] Oct 25 '22

I guess in the end, speed isn't really something to worry about that much. (unless if you're doing massive algorithms.) besides, it's mostly just banter anyway, so ignore my earlier comment.

21

u/Solonotix Oct 25 '22

I mean, most of the time we say Python is ~200x slower than C++, but the claim is Python is now up to 60% faster. Let's put this in more concrete terms.

  • If C++ took 1 minute to do it, Python would take >3 hours

  • If Python used to take >3 hours, it now takes ~2 hours

Like the previous person said, it's still no competition as to what's superior, but that's a massive improvement for Python. Also, future improvements are aiming for a 5x improvement, which would mean Python could do that same arbitrary task in ~40 minutes.

7

u/[deleted] Oct 25 '22

Can't wait to see where this speed improvement goes nonetheless!

9

u/[deleted] Oct 25 '22

Another key area to remember is that for a lot of performance focus areas in python, you often would just be calling the underlying C functions or using various libraries that handle this to drastically speed up the code.

This is where he power of libraries like Numpy are very useful and even companies like NVIDIA offer various GPU acceleration libraries for python.

7

u/alcalde Oct 25 '22

So yes C++ will always win in a field that's not always needed.

Not with Cython and Numba and 700 JIT compiler projects in their infancy....

0

u/Ning1253 Oct 25 '22

See but here's the thing - even those end up being a few times slower than C (no idea about C++, I don't personally code in it). If I want to code in something high level with speed close to C, I'll just use Julia - why bother with a half-baked in development project when I can use a language built for the task?

4

u/yvrelna Oct 25 '22

According to recent Nvidia testing, their 1700 lines of hand-optimised CUDA by experienced CUDA engineer is about 5% or so faster than the equivalent 10 lines of Python code using what's basically a drop in replacement of numpy/pandas that's been optimised for GPU.

So, yeah, that really proves that Python is slow, so let's blow our development budget, spend the next three weeks to bloat up all our programs 170x times to squeeze that 5% improvement.

2

u/Ning1253 Oct 25 '22

I said nothing about GPU or NVidia here, and this does nothing to diminish point that Julia tends to be way faster than Python with barely any extra work in writing the code. It serves my use cases, so I use it - but I then use Python in other cases instead when those suit me better.

I would also a) be really interested in reading a write-up/article about the NVidia thing you mentioned, which sounds awesome, and b) without having yet read it am doubtful to what extent the reason why NVidia were barely able to improve on the code wasn't just that the Python code was just making calls to the graphics card and running stuff there, which isn't a solution which easily applies to massive programs... In which case NVidia's 5% lead would drastically increase.

Then again I haven't read the article, so I guess I'll wait until I've read it to give a concrete opinion on it! It does sound pretty damn cool either way

1

u/MardiFoufs Oct 25 '22

Is it native python ? If the 10 lines of code are numpy/pandas, those are already written using c++/c. Using python as a shim to run c++ code does not make python itsef any faster (but that's not to say that being slower is an issue in most cases)

2

u/bryancole Oct 25 '22

An alternative viewpoint is that any C++ program which increases in complexity ultimately slows down to the speed of python.

With high program complexity, the micro-optimisations that the C++ compiler can perform at the low level become outweighed by the over-arching limitations of the hardware in terms of memory and disk access. Ultimately, you gain the most by choosing an optimal program architecture and python's speed of development makes finding/choosing the best architecture faster than C++ (although not guaranteed, of course. You can shoot yourself in the foot in either language but your foot-target-visibility is improved with python).

When you have to page lots of data in and out of disk and/or wait for IO, the overhead of the python VM and garbage-collector become insignificant.

2

u/Voxandr Oct 25 '22

You guys are forgetting PyPy project, it is close to C speed. PyPy.org

2

u/pufdo Oct 25 '22

*Types sudo pacman -Syu every 10 seconds

No python 3.11 yet? Unacceptable...

2

u/[deleted] Oct 25 '22

🚀🌕

2

u/TheRealCorwii Oct 27 '22

Been trying all day to get pygame to install, it just refuses

2

u/Proper-Turnover-5537 Nov 08 '22

I can't believe I missed this update, hitting it now. Thanks for the heads up Python-Pals

4

u/JPHorn94 Oct 25 '22

If I have 3.10 installed, what happens if I download this versiĂłn in my computer? ( it just getting started and I downloaded 3.10 last week)

19

u/lowerdev00 Oct 25 '22

Take a look at managing different Python versions with pyenv

2

u/JPHorn94 Oct 25 '22

Ok, I will! Thank you!

4

u/Log2 Oct 25 '22

If you're just starting you can stick with 3.10 without any issues.

1

u/JPHorn94 Oct 25 '22

Ok! Thanks for the heads up!

1

u/3Domse3 Oct 25 '22

PERFORMANCE: 10-60% faster code, for free!

Hooow? :O

1

u/IlliterateJedi Oct 25 '22

Reading the release notes made me realize that TypedDict was a thing, and I wish I'd known that six months ago.

1

u/Outrageous_Suit_135 Oct 25 '22

I/O bound apps won’t gain a lot. The performance enhancements should target loops with calculations that require a lot of memory read/writes.

2

u/pepoluan Oct 31 '22

But if you use a lot of asyncio, the context switching between the coroutines seems to be quite faster.

1

u/babuloseo Oct 25 '22

That toml inclusion ;)

1

u/Southern-Geek Oct 25 '22

Nice performance boost

1

u/ArtOfWarfare Oct 25 '22

Performance is somewhere between unchanged and worse for me.

I have a project with a test suite. Times in seconds from 3 runs under each:

3.9.6: 12.858, 13.210, 13.874

3.11.0: 13.284, 13.617, 13.768

Also, I’m forgiving/ignoring that the real first run in 3.11.0 took 15.168 seconds.

Seems to be 5% slower on average for me.

1

u/Groomsi Oct 26 '22

How is the speed compared to C++?

1

u/zurtex Oct 28 '22

I benchmarked it and found Python at least 10000x faster, my Python code finished running I'm still reading learncpp.com for my C++ code.

1

u/Nelson_Chow Jan 17 '23

not working with Tensor Flow,

not working with PyTorch,

not working with PyArrow

Yeah, very fast...

1

u/mid_night12 Oct 25 '22

I love python. Every day I try to learn something new with python.

1

u/Tree_Mage Oct 25 '22

Did AIFF end up getting removed as was previously threatened?

0

u/PfuiDeibel Oct 25 '22

Did nobody read the last paragraph of the change log? Crazy