r/Python • • Apr 30 '18

xkcd: Python Environment

Post image
2.4k Upvotes

388 comments sorted by

View all comments

117

u/Tweak_Imp Apr 30 '18

I really dont understand why python and its dependencies can be such a big mess. Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages. Just search and load from the manager and if you dont want a package any more, delete it and remove all dependencies that are not needed by others. Is that really so hard to do?

51

u/ursvp Apr 30 '18 edited Apr 30 '18
  • egg vs wheel

  • universal vs pure wheel

  • setuptools vs distutils

  • install_requires vs requirements.txt

  • data_files vs MANIFEST.in

  • package vs distribution vs binaries

  • pip vs conda

  • install -e vs install

  • install from PyPI vs .git vs .tar.gz

  • build vs environment

  • virtualenv vs venv vs pipenv

  • for each above: python2 vs python3

  • 2to3 vs six

  • absolute vs relative vs circular import

  • oh, pip is incapable of true dependency resolution

  • complexity behind __version__

... in contradiction of Pythonic principle of clarity and simplicity.

17

u/mfitzp mfitzp.com Apr 30 '18

It's because Python developers are forced to write clean code all the time. The filth has to come out somewhere

4

u/Log2 Apr 30 '18

The one thing that really pisses me off is that it's apparently impossible to package a project with all it's dependencies in Python. I'd love to use setup.py to create RPMs (it can do that out of the box), but I just can't figure out how to include the dependencies.

14

u/TheNamelessKing Apr 30 '18

I got so bored of dealing with that issue that at work I’ve started writing things in Rust or Haskell so I only have to distribute a single binary.

Life has gotten better. Sorry /r/Python.

3

u/adambrenecki May 01 '18

It's not quite a RPM, but there's a few projects that can produce Zip archives that Python can open, including pyzzer and pex.

The problem is, all your dependencies need to be zipsafe :(

→ More replies (3)

2

u/Zxian May 01 '18

I haven't used it for this purpose (python -> rpm), but have a look at fpm. It helps with package creation/conversion.

→ More replies (1)

2

u/ivosaurus pip'ing it up May 01 '18

Because then the python developers would have to both figure out, and write code, to interface with RHEL linux and Fedora. Make sure man files get put in just the right place. Package data is correct. Desktop files, syslog, is [some system-level daemon] now systemd based and new, or an older one? What audio service are we using? What's changed between RHEL 6 and 7?

And of course, to be fair, make sure they have everything configured for Debian's liking. But also account for the small changes and version updates that Ubuntu does.

And then also ensuring we haven't forgotten about portage and pacman. Oh, and YaST.

Oh, and then also homebrew.


And quickly, for the mostly volunteer workforce that does python packaging, the task of correctly interfacing, special-casing, path-configuring, etc becomes a matrix explosion of work from hell.

We'd love to be able to do that... but it's simply not a feasible thing to achieve.

You'll notice that practically all other scripting languages have the same position.

→ More replies (5)
→ More replies (2)

63

u/origin415 Apr 30 '18

Conda does this but doesn't have all PyPI packages. Also, occasionally you have things that assume that python references the system installed Python 2 rather than your default conda env. Way better than anything else I've seen though.

27

u/Tweak_Imp Apr 30 '18

Why are so many people still on older versions of python? I can see why it doesnt just update itself (for commercial python use for example), but Python 2.7.0 was released on July 3rd, 2010... 8 years ago. Isnt an update to a higher version with the update of the code not worth it?

16

u/origin415 Apr 30 '18

It's a lot of code to update. Most open source libraries are compatible with python 3 but a lot of companies aren't willing to migrate entire codebases internally. Also, as far as programs assuming you have Python 2 in your path, that's because OSX and most Linux distributions have it that way and very few have python 3.

→ More replies (11)

55

u/[deleted] Apr 30 '18

If you're really asking this question, you haven't been developing with python long enough.

The real problem with Python 3 is that the core development team set out to do a lot of good things, but broke compatibility in the process. And with that breakage meant that everyone using python had to step back and question whether it was worth it to scrap or upgrade years of legacy code. So when comparing the two together, there was almost no advantage in moving from python 2 to python 3. Today, the movement still has low value for these large legacy systems. There's no real performance gain, there's no must-have features, etc. The only gain here is that most of the systems are finally moving to python 3 as their stable python. And if a company did not decide to move to python 3, they will find support non existent starting in 2020.

To maybe also set your expectation, the core development team does not recommend any python version prior to 3.4, which was actually released in 2014, a full 6 years after the alpha and beta 3.0 - 3.3 pythons were released. And to add to that, I would argue that 3.6 (released in Dec 2016) was the first release where people should have started migrating. Last year is probably the first year where there was enough migration momentum that we're really starting to see strong 3.x traction. But note that it's truly only been the past 2 years.

27

u/buttery_shame_cave Apr 30 '18

there's no must-have features, etc.

well yeah because the dev team kept back-porting all the new stuff they were coming up with to 2.7.x - if they'd cut off the flow the pressure to migrate would have come on a lot sooner.

18

u/[deleted] Apr 30 '18

Basically the difference between the two versions now is "the one that breaks old code" vs "the one that breaks new code"

→ More replies (3)
→ More replies (1)

2

u/magnetic-nebula May 01 '18

I work on a scientific experiment where the development of the code began in the 90s. We need our code to work across a large variety of systems (including old computers that aren't reliably connected to the Internet).

Plus we're scientists, not programmers. NSF isn't giving us money to update code, so we don't have the $$$ to pay people to do it.

There's literally no reason for us to spend the time upgrading.

→ More replies (2)
→ More replies (1)

45

u/28f272fe556a1363cc31 Apr 30 '18

Why isnt there just one python installer

Oh, you mean like this: https://m.xkcd.com/927/

4

u/PeridexisErrant May 01 '18

See: pipenv and poetryfor recent examples...

→ More replies (2)

41

u/Yoghurt42 Apr 30 '18

Why isnt there just one python installer that installs the current version of python, sets every setting you need by himself like the PATH and then has a manager for all packages.

relevant xkcd

3

u/1024KiB Apr 30 '18

Honestly I also use perl and ruby and it's roughly the same kind of mess.

→ More replies (1)

7

u/takluyver IPython, Py3, etc Apr 30 '18

Is that really so hard to do?

Yes! Producing one installer and package manager that suits everyone's needs is almost impossible, and persuading everyone that it's better than the stuff they already know is even harder.

There are plenty of tools that work well in particular situations - e.g. for scientific programming, lots of people happily use conda. But you can also install packages through apt, or through homebrew, or manage Python versions with pyenv, and different people like all of those options.

→ More replies (3)

4

u/not_perfect_yet Apr 30 '18

Dude. Yes. Yes it's hard.

That's why there are so many different package management systems. Some for super pure open source, some that accept proprietary binaries, some for python, some for javascript...

Also, because screw shared libraries, snaps exists and venvs and pipvenv.

2

u/[deleted] Apr 30 '18

They don't. OSX in particular has a really crappy story around it.

2

u/BitLooter Apr 30 '18

I use pyenv with the virtualenv plugin. It will automatically download and install any Python version you want. Create a virtualenv and use pip to install whatever packages you want. It does all of this in ~/.pyenv, so none of it touches the system Python, and it's all isolated so you can delete a python version or virtualenv without affecting anything else.

It's built on shell scripts though, so you need a *nix shell for it to work. Obviously Linux and OSX work fine, but if you want to use it on Windows you'll need to find a Bash shell first - WLS is probably the easiest way, though it will probably work on Cygwin/MSYS as well.

→ More replies (4)