r/programminghorror • • Jan 12 '22

Python environment

https://imgs.xkcd.com/comics/python_environment.png

[removed] — view removed post

1.4k Upvotes

101 comments sorted by

View all comments

Show parent comments

79

u/mortecouille Jan 12 '22

It got to this mostly because they made the terrible, terrible decision to make python 3 not backwards-compatible. Python 3 came out in 2008 and in 2022, we are now barely seeing the end of python 2, which was officially EOL in 2020.

In comparison, that is only 2 years after java 6, which has now been basically defunct for years already. Not that it's a problem, because if you do have old java 6 code, you can still compile and run it with any subsequent java version. This means that people can painlessly upgrade: if you depend on a library that is not using java (say) 8, your java 8 program can depend on it without issues. This was not true for python. As a result, if you wanted to write something using an existing library, you were bound by the willingness of that library to upgrade to python 3. To this day, there are still old libraries out there that have not been ported to python 3.

The maintainers made the mistake of believing every python developer was equally invested in the language as they are, and assumed people would jump to upgrade their projects to python3. Unsurprisingly, this did not happen.

78

u/JackMacWindowsLinux Jan 12 '22

In particular, Python on macOS is just very messy overall. The default python binary points to 2.7, which is shipped with macOS. You can install newer versions of 2 and 3 from Python.org, but to avoid conflicts with the system, these are installed in /Library/Frameworks/Python.framework. Apparently both 2 and 3 are installed here, but the binaries in 3 are suffixed with 3 to avoid conflicts with the system version. Furthermore, you can install Python 2 (deprecated) or 3 through Homebrew, which sticks it in /usr/local/Cellar/python@x/x.y.z (referenced in the comic), and the files are symlinked to /usr/local/opt/python@x as well as the appropriate places in /usr/local. So if you installed all of these at some point, you'll have multiple ways to run Pythons of various versions:

  • /usr/bin/python => system 2.7

  • /Library/Frameworks/Python.framework/bin/python[3] => Python.org 2.7/3.x

  • /usr/local/bin/python = /usr/local/opt/python@2/bin/python = /usr/local/Cellar/python@2/2.7.18/bin/python => Homebrew 2.7

  • /usr/local/bin/python3 = /usr/local/opt/python@3/bin/python3 = /usr/local/Cellar/python@3/3.y.z/bin/python3 => Homebrew 3.x

I don't have experience with Anaconda, but if it ships its own Python too then that's another way to run it.

In the end, Python on macOS really is a mess, and it would be nice if Apple could ship Python 3 as default, but that would break things that rely on the system version being 2.7, and, well, Apple.

30

u/R3D3-1 Jan 12 '22

It would also be nice if the Python 3 executable, being not backwards-compatible with any Python 2 executable possibly already existing, would always be called python3 and not python. Regardless of how and on what OS it is installed.

12

u/abcteryx Jan 12 '22

Never mind the fact that installing Python from python.org on Windows doesn't let you just run python or python3, but rather you type py to run the "default" (defaults to latest version installed) or e.g. "py -3.7" to run an old version. And e.g. "py -m venv .venv" to source a virtual environment from your default Python.

It's actually quite useful and is a first-party supported solution bundled with Python. It's very similar to pyenv and the like.

The downside is when trying to help newbies, you have to know all the intricacies of every platform, and know exactly how they installed their Python.

For example, the Windows Store now has multiple Python versions available for install. I haven't checked out this approach yet, so I don't know where they install to on the computer. I know other such "apps" install into gibberish folder names in a protected apps folder, perhaps the Python apps go to that senseless place as well.

But if you install Python via Windows Store, you invoke it by typing python or python3. But what if you have multiple 3.X versions installed that way? I don't know, as I haven't tried.

And at least as of a year or so ago, the Windows Store would even intercept typing python in an activated virtual environment (where such a command is a valid way to launch the venv Python). It would take you to the Windows Store to install Python the "new way".

Python version management is a moving target on every platform, and if you hope to help friends and colleagues get into it, you need to know the intricacies of every one in order to just get them started.

→ More replies (1)

14

u/AlternativeAardvark6 Jan 12 '22

Full time Java (mostly) Dev here: Java 6 code might not run as well in more recent JVMs as they make you believe. I have clients that are still on 6 because their massive monoliths don't work on Java 8 or newer.

11

u/mortecouille Jan 12 '22

I'm also a full time java developer, you must admit that this is the exception rather than the rule. There are incompatibilities between versions, however very rarely anything as breaking as python2 vs 3. One example that comes to mind is usage of com.sun classes, which should not have been done in the first place.

Like probably everywhere else, we maintain ancient codebases and I can count the times I had to find an old version of the JDK on the fingers of one hand. And usually for troubleshooting a JDK-specific bug, not even because the code wouldn't run.

4

u/AlternativeAardvark6 Jan 12 '22

To be fair, we are a company with many clients that have decade old systems, last year I fixed a bug in a class that hasn't been touched since 2003. Millions of lines of code in systems that are used daily and where all new code depends on, so if there is a chance for something to go wrong by using JDK 8 we have a big chance of it happening sooner or later.

11

u/indign Jan 12 '22

Python 3 was always going to be incompatible with Python 2 because that's how version numbers work. Major version increment => breaking changes. What you're saying is that you wish Python 3 was never released at all. Well, the Python 2 interpreter is still available if you want to deal with its nonsense.

There were a lot of things about Python 2 that royally sucked (such as scoping rules, text encoding, and the ABI for language extension) that would have been impossible to fix in a backwards-compatible way. Python 3 is better, full stop.

0

u/[deleted] Jan 12 '22

[deleted]

0

u/[deleted] Jan 12 '22

[deleted]

0

u/[deleted] Jan 12 '22

[deleted]

0

u/[deleted] Jan 12 '22

[deleted]

2

u/konaya Jan 12 '22

Nobody's going to spend years rewriting their software every time there's a version bump, that's insane

Which isn't at all what's on the table here. There were serious flaws in Python 2. They got corrected in Python 3. The people who were too busy making up excused not to look over their code once in a while got left in the dust where they belong. That's all there is to it.

2

u/JellyTwank Jan 12 '22

It is not necessarily even that. Keep in mind that development costs money. If the company you are working for has spent millions in developing their product or support suite of code, then it can be prohibitively expensive to rewrite/refactor that code. The cost of maintenance on that now soon-to-be or already-deprecated code can be the cheaper, although painful, route. Consider for example all of the still running COBOL stuff out there. (I worked on Y2K mitigation in '95, '96 and '97). Were there "better" programming languages and environments? For sure, for years. But it is not laziness that keeps that COBOL stuff running and maintained to this day.

Decisions by language developers to abandon older frameworks or models are not lightly undertaken, for sure. But there obviously can be fallout from those decisions. The mess with all the Python environment fun outlined by XKCD is precisely because of the sunk costs involved by the end users of Python and not lazy coders that will not change to newer libraries.

6

u/mortecouille Jan 12 '22

The point is precisely that people haven't ported their stuff. As a user (not even a developer) the fact remains that you have to install both versions and fuck around.

Certainly some blame lies on developers: probably some were stubborn, some depended on abandoned projects which had no maintainer to port them, some had massive projects were the effort to port was too large.

But to me, the fact that it took a decade points to the language not providing an adequate update path, sufficient deprecation period and didn't sufficiently mitigate the incompatibilities. Python just went scorched earth and expected people to adapt just like that.

4

u/muikrad Jan 12 '22

Hmm, what you say is true but I'm not sure why it's related to the graphic.

8

u/mortecouille Jan 12 '22

A lot (granted, not all) of the clusterfuck comes from usually needing python2 and 3 to coexist. It's true that there's other things involved, like virtualenvs, conda, etc.

2

u/muikrad Jan 12 '22

But I also ran into these things when I started out and py3 was not even a thing. Installing things in my OS python for development and noobing around like that.

These days I need to have 2.7 and several 3.x at once, I work in dozens of different repos at the same time and they all need their own interpreter and dependencies (i.e. virtualenvs). All the problems I had were due to me not understanding how it works and my OS sure ended up looking like the graphic! 😭

I mean, python is clearly to blame for not streamlining these things, and py2 vs py3 was a mess, but this graphic is about user errors.

→ More replies (1)

2

u/KagakuNinja Jan 12 '22

The real problem is how Python tries to install in a standard location such as /usr/bin/python, and how they manage dependencies. The Java world has never had this problem, as you can just change JAVA_HOME, or the class path.

1

u/aaronfranke Jan 12 '22

Breaking compatibility is not a terrible decision. The problem is that devs refuse to stop using Python 2.

Furthermore, it is totally possible to write software that is compatible with both versions.

7

u/mortecouille Jan 12 '22

You can't just introduce drastically breaking changes and complain that people are reluctant to upgrade. It can be a massive amount of work.

I write APIs and SDKs for a living. Incompatible changes aren't a thing you take lightly: we absolutely avoid them as much as possible, and for every breaking change, we provide a mitigation plan. Yes, that means that sometimes, if for instance there is a spelling mistake in an API method, it'll stay there. We might add another method with the right name, but the other one will just be deprecated, and redirect to the other call. We have people upgrading legacy systems from 5, 6 major versions ago, completely transparently.

3

u/Illiux Jan 12 '22

Not everything is maintained. There are plenty of abandoned projects on python 2 that someone might need to run for one reason or another.

Breaking compatibility is almost always a terrible decision. There's a reason why the Linux kernel uses a guiding principle of "don't break userspace", which is equivalent to infinite backwards compatibility. There's also a reason why Microsoft has invested incredible amounts of engineering effort to get old binaries running - there is a real need for it.

When you do break backwards compatibility, you must live with the fact that the old version will still be needed until the end of time. You can't blame stubborn devs for the very real problem of needing to run abandonware.

0

u/[deleted] Jan 12 '22

[deleted]

2

u/KagakuNinja Jan 12 '22

Java has had very few breaking changes, if you stick to the standard API. One source of problems were devs relying on internal sun packages. Java 9 also broke things because some fancy libraries were relying on the package structure of the standard library in the JRE.

I've been using Java for 20 years, and Java 9 is the only issue I've had.

→ More replies (3)