r/Python • • Jan 05 '18

Really??

Post image
2.0k Upvotes

317 comments sorted by

View all comments

Show parent comments

16

u/danielbibit Jan 05 '18

How about a conditional import ?

0

u/[deleted] Jan 05 '18

Would have been an option in the beginning of the project, maybe.

In the current state of the project though it would require some major refactoring for each individual optional dependency. Overriding the import has turned out to be the better solution.

30

u/rhytnen Jan 05 '18

I have a really hard time believing this was ever a good idea. Absolutely insane solution if you aren't leaving out important details.

5

u/[deleted] Jan 05 '18

Well, maybe one missing detail is that it doesn't really override the original import but just wraps it. The original import instruction is used internally to import all other non-optional modules and optional modules that are installed.

2

u/rhytnen Jan 05 '18

help me understand .... here are some scenarios. what is happening your infrastructure that requires something else?

from your_lib import required_libs
try:
    from your_lib import optional_lib
except ImportError:
    logging.warning('optional_lib not found, xyz not supported')

try:
    import some_lib
except ImportError:
    import some_other_lib as some_lib

1

u/[deleted] Jan 06 '18

Okay...

Yes, that's one way to do it, but...

What about the places where the modules are actually used? We would have had to catch those calls as well in the case that a module is not available. That were a huge amount of additional try-except blocks and another huge amount of actual error handling (avoiding all calls to that module). We tried that. It's ugly.

Specifically for matplotlib that hasn't made much sense to us. It's totally okay that the program runs with or without matplotlib since matplotlib does not alter anything. The MagicMock objects returned by the wrapped import do no harm. They just pretend to do what matplotlib was supposed to do - without actually doing it. A dry run, if you will. The user will be informed that matplotlib is optional and not installed so that they don't wonder why visualization does not work.

To help you understand: we were after a solution where most of the already existing code could remain untouched. Only specific modules are marked optional. That is, if a module is missing that is not explicitly marked optional, Python will still raise an ImportError. Otherwise it will only show a warning, but the program will continue to run. The thing is, the existing code base is quite large and quite old. We still have instructions/syntax in there that date back to Python 2.5. Messing with that code aside from upgrading it to at least Python 2.7 is a risky challenge.

Furthermore, it's political. There's currently no official project that involves working on the code. So everything that has to be done, must take as little time as possible.

Wrapping import was the quickest and cleanest solution we could come up with within the available time. It may not be beautiful but it works and we're okay with it. It's also documented so that future developers will understand what's happening.

2

u/rhytnen Jan 06 '18

I mean I do exactly that. Here's an excerpt from some code of mine.

try:
    import matplotlib.pyplot as plt
except ImportError:
    plt = None
    print( "Warning: could not import from matplotlib. Echogram.show() will not work.")

Now later on in the class that would call imshow ...

    def show(self):
    '''
    Plot an image of the echogram.
    '''
    if plt is None:
        logging.warning("matplotlib was not imported, cannot plot")
        return

    plt.imshow(self.data, aspect='auto')

So your code (apparently) ...

  • literally changes the semantic meaning of a python builtin

  • alters behavior the user may not be expecting

  • probably makes the code seem odd to a new comer

I would prefer my solution - which isn't full of thousands of try / excepts - any day.

1

u/[deleted] Jan 06 '18

That's pretty much what we had before. But it turned out really hard to maintain because other developers of certain modules (plugins) did not take the fact of optionality of some modules into account. They just imported without checking and since the plugin system that we have just bulk loads everything at once, a broken plugin could bring the whole program to its knees. It's not my job to fix plugin after plugin like this. So instead of trying to teach every developer to obey our optionality rules, this method gives them the freedom to code like all the modules were installed, and the checking takes place somewhere else.

To be honest, the whole project has become awfully hard to maintain because so many developers with different kinds of experience had their fingers in it in the past. Mathematicians with some basic programming skills as well as computer scientists with all the theory but no practice. I'm the first one who throws in a good amount of theoretical and practical experience and I think I know quite well what I'm doing.

Yes, it tinkers with Python builtins and it may confuse future developers at first. But I also think I've documented/commented it sufficiently so that any future developer should be able to quickly grasp what's going on.

It may be removed again at some point in the future when the overall project has a better structure. One of the things I intend to do is to build a better plugin system which is capable of dealing with plugin meta data and load plugins on demand instead of just bulk importing everything at once. It's a lot of work and this import wrapping is just some dirty little helper for the time being.

1

u/rhytnen Jan 06 '18

I think the normal this use the technique i showed but insde of init the all and to get the behavior you want.

You have to admit changing import is a sign something went wrong.

1

u/[deleted] Jan 06 '18

Um, could you fix your first sentence? I'm not sure if I understand what you're trying to say. No offense.

Yes, I admit that. But there's not enough (paid) time to fix everything and there are more important things to fix/change/refactor. It may be a temporary solution or, like a usual makeshift, will last in there forever. I can't tell. It works and that's all that counts for the moment. Maybe in the future in the course of refactoring the project it will be fixed as well. Let me quote one of the greates musicians of our time, Mike Oldfield: only time will tell.

→ More replies (0)