r/programminghorror • • Jan 12 '22

Python environment

https://imgs.xkcd.com/comics/python_environment.png

[removed] — view removed post

1.4k Upvotes

101 comments sorted by

View all comments

124

u/Instatetragrammaton Jan 12 '22 edited Jan 12 '22

How did it get to this? Furthermore - is just throwing each of these in a Docker container a viable solution?

Kind of strange for a language that has "there should be one— and preferably only one —obvious way to do it" as a design philosophy.

60

u/PostFunktionalist Jan 12 '22

The best practice is to make a new Python environment for every project using virtualenv or some convenience wrapper around it.

There are at least 4 different wrappers.

Good shit.

6

u/Yelov Jan 12 '22

I'm using conda for having multiple Python environments. How does using virtualenv for every project work? Eg if you have multiple projects with very similar packages, do you have to have them installed X times for every project? Or is there like a central package location that gets shared?

10

u/brews Jan 12 '22

Have one virtual environment for each project. Doesn't matter how close or far apart each projects dependencies are. One for each project. The whole point is to keep them separate.

4

u/DrMaxwellEdison Jan 12 '22

Each environment is completely separate from each other, with no shared packages. That may seem like overkill, but supposing two projects rely on the same package and you start upgrading one of them, you don't inadvertently update and potentially break the other.

2

u/Ollymid2 Jan 12 '22

Recommend pyenv - use it for work and so far I’ve managed to keep things organised

79

u/mortecouille Jan 12 '22

It got to this mostly because they made the terrible, terrible decision to make python 3 not backwards-compatible. Python 3 came out in 2008 and in 2022, we are now barely seeing the end of python 2, which was officially EOL in 2020.

In comparison, that is only 2 years after java 6, which has now been basically defunct for years already. Not that it's a problem, because if you do have old java 6 code, you can still compile and run it with any subsequent java version. This means that people can painlessly upgrade: if you depend on a library that is not using java (say) 8, your java 8 program can depend on it without issues. This was not true for python. As a result, if you wanted to write something using an existing library, you were bound by the willingness of that library to upgrade to python 3. To this day, there are still old libraries out there that have not been ported to python 3.

The maintainers made the mistake of believing every python developer was equally invested in the language as they are, and assumed people would jump to upgrade their projects to python3. Unsurprisingly, this did not happen.

78

u/JackMacWindowsLinux Jan 12 '22

In particular, Python on macOS is just very messy overall. The default python binary points to 2.7, which is shipped with macOS. You can install newer versions of 2 and 3 from Python.org, but to avoid conflicts with the system, these are installed in /Library/Frameworks/Python.framework. Apparently both 2 and 3 are installed here, but the binaries in 3 are suffixed with 3 to avoid conflicts with the system version. Furthermore, you can install Python 2 (deprecated) or 3 through Homebrew, which sticks it in /usr/local/Cellar/python@x/x.y.z (referenced in the comic), and the files are symlinked to /usr/local/opt/python@x as well as the appropriate places in /usr/local. So if you installed all of these at some point, you'll have multiple ways to run Pythons of various versions:

  • /usr/bin/python => system 2.7

  • /Library/Frameworks/Python.framework/bin/python[3] => Python.org 2.7/3.x

  • /usr/local/bin/python = /usr/local/opt/python@2/bin/python = /usr/local/Cellar/python@2/2.7.18/bin/python => Homebrew 2.7

  • /usr/local/bin/python3 = /usr/local/opt/python@3/bin/python3 = /usr/local/Cellar/python@3/3.y.z/bin/python3 => Homebrew 3.x

I don't have experience with Anaconda, but if it ships its own Python too then that's another way to run it.

In the end, Python on macOS really is a mess, and it would be nice if Apple could ship Python 3 as default, but that would break things that rely on the system version being 2.7, and, well, Apple.

30

u/R3D3-1 Jan 12 '22

It would also be nice if the Python 3 executable, being not backwards-compatible with any Python 2 executable possibly already existing, would always be called python3 and not python. Regardless of how and on what OS it is installed.

12

u/abcteryx Jan 12 '22

Never mind the fact that installing Python from python.org on Windows doesn't let you just run python or python3, but rather you type py to run the "default" (defaults to latest version installed) or e.g. "py -3.7" to run an old version. And e.g. "py -m venv .venv" to source a virtual environment from your default Python.

It's actually quite useful and is a first-party supported solution bundled with Python. It's very similar to pyenv and the like.

The downside is when trying to help newbies, you have to know all the intricacies of every platform, and know exactly how they installed their Python.

For example, the Windows Store now has multiple Python versions available for install. I haven't checked out this approach yet, so I don't know where they install to on the computer. I know other such "apps" install into gibberish folder names in a protected apps folder, perhaps the Python apps go to that senseless place as well.

But if you install Python via Windows Store, you invoke it by typing python or python3. But what if you have multiple 3.X versions installed that way? I don't know, as I haven't tried.

And at least as of a year or so ago, the Windows Store would even intercept typing python in an activated virtual environment (where such a command is a valid way to launch the venv Python). It would take you to the Windows Store to install Python the "new way".

Python version management is a moving target on every platform, and if you hope to help friends and colleagues get into it, you need to know the intricacies of every one in order to just get them started.

→ More replies (1)

14

u/AlternativeAardvark6 Jan 12 '22

Full time Java (mostly) Dev here: Java 6 code might not run as well in more recent JVMs as they make you believe. I have clients that are still on 6 because their massive monoliths don't work on Java 8 or newer.

11

u/mortecouille Jan 12 '22

I'm also a full time java developer, you must admit that this is the exception rather than the rule. There are incompatibilities between versions, however very rarely anything as breaking as python2 vs 3. One example that comes to mind is usage of com.sun classes, which should not have been done in the first place.

Like probably everywhere else, we maintain ancient codebases and I can count the times I had to find an old version of the JDK on the fingers of one hand. And usually for troubleshooting a JDK-specific bug, not even because the code wouldn't run.

4

u/AlternativeAardvark6 Jan 12 '22

To be fair, we are a company with many clients that have decade old systems, last year I fixed a bug in a class that hasn't been touched since 2003. Millions of lines of code in systems that are used daily and where all new code depends on, so if there is a chance for something to go wrong by using JDK 8 we have a big chance of it happening sooner or later.

11

u/indign Jan 12 '22

Python 3 was always going to be incompatible with Python 2 because that's how version numbers work. Major version increment => breaking changes. What you're saying is that you wish Python 3 was never released at all. Well, the Python 2 interpreter is still available if you want to deal with its nonsense.

There were a lot of things about Python 2 that royally sucked (such as scoping rules, text encoding, and the ABI for language extension) that would have been impossible to fix in a backwards-compatible way. Python 3 is better, full stop.

0

u/[deleted] Jan 12 '22

[deleted]

0

u/[deleted] Jan 12 '22

[deleted]

→ More replies (4)

6

u/mortecouille Jan 12 '22

The point is precisely that people haven't ported their stuff. As a user (not even a developer) the fact remains that you have to install both versions and fuck around.

Certainly some blame lies on developers: probably some were stubborn, some depended on abandoned projects which had no maintainer to port them, some had massive projects were the effort to port was too large.

But to me, the fact that it took a decade points to the language not providing an adequate update path, sufficient deprecation period and didn't sufficiently mitigate the incompatibilities. Python just went scorched earth and expected people to adapt just like that.

4

u/muikrad Jan 12 '22

Hmm, what you say is true but I'm not sure why it's related to the graphic.

8

u/mortecouille Jan 12 '22

A lot (granted, not all) of the clusterfuck comes from usually needing python2 and 3 to coexist. It's true that there's other things involved, like virtualenvs, conda, etc.

2

u/muikrad Jan 12 '22

But I also ran into these things when I started out and py3 was not even a thing. Installing things in my OS python for development and noobing around like that.

These days I need to have 2.7 and several 3.x at once, I work in dozens of different repos at the same time and they all need their own interpreter and dependencies (i.e. virtualenvs). All the problems I had were due to me not understanding how it works and my OS sure ended up looking like the graphic! 😭

I mean, python is clearly to blame for not streamlining these things, and py2 vs py3 was a mess, but this graphic is about user errors.

→ More replies (1)

2

u/KagakuNinja Jan 12 '22

The real problem is how Python tries to install in a standard location such as /usr/bin/python, and how they manage dependencies. The Java world has never had this problem, as you can just change JAVA_HOME, or the class path.

1

u/aaronfranke Jan 12 '22

Breaking compatibility is not a terrible decision. The problem is that devs refuse to stop using Python 2.

Furthermore, it is totally possible to write software that is compatible with both versions.

→ More replies (2)
→ More replies (5)

2

u/inputwtf Jan 12 '22

At least with Ansible, you'd have to deal with mounting file paths in your docker container correctly, and having everyone create a filesystem layout that is exactly the same so they can use the same file mounting path args in their docker run command.

It's just trading one shitshow for another.

2

u/shinjuku1730 Jan 12 '22

Yeah I do that. I use Docker for each Python project, Makefile for quick commands. Works like a charm and it's super easy to collaborate.