r/Python • u/[deleted] • Jul 22 '21
Resource Do not use mutable objects as default arguments in Python
https://medium.com/geekculture/do-not-use-objects-as-default-arguments-in-python-1c940212db2e69
u/JohnDoe2991 Jul 22 '21
I learned this the hard way...
8
u/tuckmuck203 Jul 22 '21
Yeah, I discovered this while writing a recursive function to get the changed attributes between two instances of an object. Unfortunately for me, some of the attributes were objects with attributes that have the same names. It was hell trying to figure out why only some attributes seemed to be appearing and why some were incorrect values.
13
u/chalky331 Jul 22 '21
Care to elaborate?
39
u/JohnDoe2991 Jul 22 '21
Well I wrote pretty much the function OP describes in his article within a telematics application.
The function itself was very simple, so I didn't expect any mistake in there.
My lists simply kept growing and I didn't understand why. I searched all my code where the old list elements would be re-added again.
I tried debugging the code and couldn't find the issue.
I've rewritten several chuncks of code but the issue was still there.
Only after several hours and writing a minimal example I realized that these mutable objects as default arguments are persistent between calls.
I facepalmed myself to the moon, made a little fix in the function and called it a day.
42
u/thisismyfavoritename Jul 22 '21
The mutable object in the function argument gets initialized only once when the function is defined. So that variable always contains a reference to the same list for every call you make and not a new list like the signature might suggest.
OP probably encountered issues because of this surprising behavior
6
u/stackered Jul 22 '21
its basically like creating a global variable. if you have experience with Python you know to create function level variables every time
4
u/pudds Jul 22 '21 edited Jul 22 '21
Here's an easy code snippet (untested, but based on my experience) that should repro the problem:
class Foo: def __init__(self, items = []): self.items = items foo = Foo() foo.items.append("bar") print(foo.items) # should see a list containing bar foo2 = Foo() print(foo.items) # you'd expect an empty list, but you'll see bar againTo avoid the problem, you can do something like this instead:
class Foo: def __init__(self, items = None): self.items = items or [] foo = Foo() foo.items.append("bar") print(foo.items) # should see a list containing bar foo2 = Foo() print(foo2.items) # you should see an empty list this time3
u/backtickbot Jul 22 '21
1
u/redderper Jul 22 '21
print(foo.items) # you'd expect an empty list, but you'll see bar again
Shouldn't this be print(foo2.items)?
1
3
u/soulstuff_ Jul 22 '21
I'm just assuming here. But I think its because maybe the interpreter handles default variables as part of the function definition and therefor saves specific memory and creates the variable on definition instead of creating it when the function its called. So, each time you called it without parameters you use the same list over and over (like in the example it keeps adding data) because the variable it's outside the fucntion's scope as it's part of the definition.
Then again this is just eyeballing I don't know anything about the inner workings of the python interpreter but to be safe I just do arg= None and an if inside the function to create the list/mutable.
edit: punctuation
2
u/bacondev Py3k Jul 23 '21
Yep, pretty much. This was a design decision for the sake of optimization. Creating an immutable object potentially each time the function is called is inefficient. I guess that they wanted the behavior for mutable objects to be consistent with that.
1
18
14
u/cheese_is_available Jul 22 '21
Same for default date 'datetime.today()' on servers that stay online for years.
2
11
u/Efficient-Chair6250 Jul 22 '21
That tip doesn't only apply to functions, but classes/dataclasses as well. There is a reason dataclasses 'field' method has a 'default_factory' parameter
4
4
u/stevenjd Jul 23 '21
Do not use mutable objects as default arguments in Python
Unless you want to. Using mutable defaults is fine if you know what you're doing.
1
3
2
u/-UltraAverageJoe- Jul 23 '21
The real lesson should be don’t use default arguments when your function requires a list to append some data to. This case should never happen so the example is moot.
6
u/thedominux Jul 22 '21 edited Jul 23 '21
Btw using mutable objects as class attributes leaves its mark - every instance of the class will have the same object as its attribute
class Lol:
kek = []
Lol().kek.append(None)
print(Lol().kek)
# Will be: [None]I'm not against it, it can also be an option to use, for example, in cases when you need joint context
21
u/koogas Jul 22 '21
Well, yeah, but this is obvious no?
It's literally the purpose of class attributes
→ More replies (10)6
u/Efficient-Chair6250 Jul 22 '21
Type hints blurred the lines. It's a trap especially for dataclasses
9
u/backtickbot Jul 22 '21
2
2
1
u/BlobbyMcBlobber Jul 23 '21
Huge pitfall. Took me way too long to figure it out the first time I tried it.
1
1
u/ggchappell Jul 23 '21
My solution to this is to make my default arguments None, always. Yes, numbers don't have the problem, so I could make a default argument a number -- or a tuple, or a frozenset, or .... But a policy of always using None means one less thing to think about and possibly get wrong.
2
u/asday_ Jul 23 '21
I really want to disagree with you but I can't find it in myself to do so. Consistency is key, even if it's a bit verbose.
1
u/VictosVertex Jul 23 '21
I know the implications of this but so far I think this shouldn't be a problem if you follow basic best practices anyways.
But I may be wrong here so I would like to ask:
Can someone provide a function that is perfectly valid in regards to side effects, single responsibility and not reassigning arguments that would still have the same problem?
Like literally every single example shown here breaks either or all of these guidelines and I would consider them bad no matter what.
For example:
def append_10(arr = []):
arr.append(10)
return arrThis function is obviously used to indicate how bad mutable default arguments are, but I think this function is bad even if you fix the default parameter.
Why? because the solution to the above problem is this:
def append_10(arr = None):
if arr is None:
arr = []
arr.append(10)
return arrBut this function:
reassigns an argument
does two things in the case no list is given (create as well as append)
I think, isn't logically consistent
To these 3 points:
reassigning arguments is bad in my eyes, they are arguments, not variables
doesn't follow single-responsibility, if its responsibility is to append an element to a list, then it should take a list as a parameter and reject everything else. If someone wants to append 10 to an empty list, they should provide said empty list to the function.
If I append 10, as the function name implies, to "nothing" then to me a list isn't what logically follows. If I have an empty list and append 10 then sure, a list with the element 10 follows. But if I don't even have a list then I simply can not append an item to it. Now this may be personal preference but I think it goes hand in hand with 2. and I think it is bad practice as it is ambiguous and doesn't follow single-responsibility.
The following version suggested by another person is even worse:
def append_10(arr = None):
arr = arr or []
arr.append(10)
return arrThis function returns a list, with the element 10 in it, for every single falsy value. It also obviously crashes for every truthy value except for those that have the method append which takes a single int as an argument. This again is logically inconsistent and behaviour, I think, one shouldn't expect from a function.
append_10(False) -> [10]
append_10(0) -> [10]
append_10(6<2) -> [10]
append_10("") -> [10]
append_10({}) -> [10]
append_10(()) -> [10]
append_10(set()) -> [10]
(I especially like the 3rd case, makes zero sense if spoken out loud, but works)
On top of this I also think the statement "Do not use mutable objects as default arguments in Python" itself is too general. It should be "Do not use mutable objects as default arguments in functions which mutate said object". Because I don't see how the problem would arise if you don't alter the argument in the first place.
For example if you had a list of terms you wanted to exclude and in case no terms are provided you use an empty list. This, I think, is valid but the general statement would tell you not to do it.
def add_10_but_exclude(arr, exclude = []):
for i,value in enumerate(arr):
if i not in exclude:
arr[i] = value + 10
return arr(yes I know ´the function is bad, inefficient and doesn't follow any best practices either, but it's simple and gets the point across)
This function also uses a default parameter for exclude which is mutable. But as the state is never mutated to begin with, the above statement to "not use it" doesn't apply.
So, can anyone please provide a function that is "logically consistent", follows basic best practices and still requires a mutable default (that then has to be changed as mentioned in the article)?
1
u/asday_ Jul 23 '21
reassigns an argument
Who cares? It's just a precondition. The argument needs a default and Python is being a douche about how it has to be specified.
def f(x=0): return x + 1You don't have a problem with that because the language is doing something sensible.
Imagine a Python where default args could only be
Noneand had to be specified in the body. That would then become:def f(x=None): if x is None: x = 0 return x + 1Note how the language being a douche caused the code to have to do something weird.
does two things in the case no list is given (create as well as append)
Doesn't matter. The entire point of functions is to group one OR MORE operations into a reusable package. Default arguments also create.
>>> def f(x=0): ... pass ... >>> f.__defaults__ (0,) >>>I think, isn't logically consistent
Subjective.
2
u/VictosVertex Jul 23 '21
Why so aggressive? I literally opened by stating I may be wrong here and that I would like to see better examples.
You don't have a problem with that because the language is doing something sensible.
Well if I wanted to be pedantic then yes I would still have a problem with that. Because you've literally just provided the successor function with the unnecessary step of providing a 0 element in case no element is inserted. This also means a call to f() returns 1, which I again would regard as false. 1 is simply not the successor of None, it is the successor of 0 and those are not the same.
But that was my entire point: The examples used are pointless because they are flawed even without what this discussion is about.
Which is exactly why I - asked - for someone to provide a better example.
Doesn't matter. The entire point of functions is to group one OR MORE operations into a reusable package. Default arguments also create.
Cool, and the entire point of a program is to solve a problem. Yet we still try to solve them efficiently and follow best practices for various reasons.
A function having multiple operations doesn't necessarily equal a function having multiple responsibilities.
Subjective.
Care to elaborate when it is logical for a function called append_10 to return [10] when called with False?
At the very least this doesn't follow the principle of least astonishment.
Also what is the reason to not enforce append_10 to be called with a list in the first place? A user surely would expect a function requiring some argument to not work if said argument is not provided.
To me the so far presented examples just use hacky solutions in order to prevent false user input.
Again, I know these are just examples that try to get the point across. But I think these are bad examples and I would like to see a better one, that's all I was asking for.
1
u/asday_ Jul 23 '21
Why so aggressive?
That's your reading of my message. I'm not being aggressive, I'm engaging in discussion.
1 is simply not the successor of None, it is the successor of 0 and those are not the same.
You're getting hung up on the details here. The important thing is
Nonedenoting "use default value".Noneis not being treated as a value in and of itself. Because the language is not stupid when it comes to ints as they're immutable, I used those as an example. I could quite easily have used a string.Similarly I could quite easily have done something like this:
use_default = object() def f(x=use_default): if x is use_default: x = some_default ...Because Python is stupid with mutable values in
__defaults__(and__kwdefaults__), you're forced to do the above. In the case of immutable values, Python is doing that for you. There's no world where that's not happening, it's just a case of whose responsibility it is to write it, the interpreter or the programmer.Now it just so happens that we already have an object in Python that's easily comparable and predefined, and has generally been used semantically as "use the default argument if this is something that's meant to have a default argument but can't because mutable arguments are handled terribly", and that's
None, so the above becomes:def f(x=None): if x is None: x = some_default ...Which is exactly why I - asked - for someone to provide a better example.
My contention isn't the quality of the examples in the article, it's with the definition of quality of examples in your post. None of the three points you declared are definitive of quality. Quality occasionally exhibits your points, but it's not the other way around. Reiterating that you want better examples in reply to my comment is ignoring the points I'm making and is best served as a reply for someone who is arguing that the existing examples fit your definition of quality, rather than someone who is arguing that your definition of quality is suspicious.
A function having multiple operations doesn't necessarily equal a function having multiple responsibilities.
Then where do you draw the line? We use programming languages to solve real problems in the real world and NOBODY wants to pay you to maintain this trash:
def default(value, default): return default if value is None else value def f(x=None): x_with_a_different_name = default(x, some_default) ...Care to elaborate when it is logical for a function called append_10 to return [10] when called with False?
Your position has changed. The appropriate naming of functions was not one of your three measures of quality. I do not disagree that functions should be named appropriately. Nobody disagrees with that.
2
u/VictosVertex Jul 23 '21
You're getting hung up on the details here. The important thing is None denoting "use default value". None is not being treated as a value in and of itself. Because the language is not stupid when it comes to ints as they're immutable, I used those as an example. I could quite easily have used a string.
I'm not talking about the "None" in the function though, maybe I should've phrased it differently and named it "nothing" instead of "None". I talk about the parameters the function is called with.
If you call f() you call a function without a parameter, you call it with "nothing".
But your "successor function with default at 0" puts out f()->1 which maps "nothing" to 1. f(3)->4 makes sense, f(0)->makes sense, f()->1 doesn't.
There simply is no need for any of the so far provided examples to use a default value in the first place, other than wanting to write f() instead of f(default).
All of the so far provided functions are of the same nature:
def func(some_mutable_argument=None): if some_mutable_argument is None: some_mutable_argument = a_new_instance_of_the_argument() # Do some shit with some_mutable_argument ...But why? Why not just use...:
def func(some_mutable_argument): # Do some shit with some_mutable_argument ......and let the user of the function provide the object the function should mutate?
Like isn't that literally the purpose of the function, to mutate the state of a provided object? From my point of view the (single) responsibility of the function is to do stuff with a provided object. The user is responsible to provide said object.
That is why I asked for a function that actually - requires - a mutable default argument, so I can get a better grasp of the use-case. Basically a real world example.
Your position has changed. The appropriate naming of functions was not one of your three measures of quality. I do not disagree that functions should be named appropriately. Nobody disagrees with that.
Ok so how would you name such a function "appropriately"?
1
u/catorchid Jul 24 '21
I will never understand this sub. Sure, it's a rather common anti-pattern, but more than a thousand upvotes for this post? That links to Medium?
There are people out there that do amazing stuff from scratch, approach entirely new domains of applications of Python, come up with clever ways to represent or interpret data...
And here we are, burying this under an avalanche of upvotes.
Seek originality, innovation, and creativity, or this place will become a boring graveyard
0
Jul 29 '21
[deleted]
1
u/catorchid Jul 29 '21
Sure, it's an important concept, but so is the for-loop. One thing is to build skyscrapers, another is to milk upvotes.
Of course, we're arguing about our respective opinions, and there are 1200+ upvotes saying that I'm clearly losing here
1
-2
u/jet_heller Jul 22 '21
I really hate when people make blanket "don't" statements. Almost everything has some reason to be done, you just have to know what and why.
52
u/Ph0X Jul 22 '21
In this case it very much is a blanket statement. Unless you want unreadable hacky code. If you truly want a global object, actually define one at the global scope and access it inside the function instead of relying on the kwarg default value being the same. Achieves the same and is orders of magnitude more readable. Remember explicit is better than implicit.
-3
u/jet_heller Jul 22 '21
It can be used to keep information between different calls of the function without needing to write a bunch of infrastructure around it. If that's what you want, this is acceptable.
For just about everything there is a valid use for it and you just need to know about the possible catches.
6
u/cheese_is_available Jul 22 '21
I use that for caching. But this is a hack. If there is a breaking change in python on the subject I deserved to be punished.
15
u/eveninghighlight Jul 22 '21
Yeah that's a global object
-5
u/jet_heller Jul 22 '21
That doesn't interfere with any other scope. It could be very useful in very specific scenarios.
3
u/eveninghighlight Jul 22 '21
Maybe - but I feel like it's more likely to just confuse things. There are juniors on my team and i know this kind of thing can just ruin their week
You're right that the blanket "don't" statement isn't technically true, but if you know enough to know when to use this you probably know enough to ignore this kind of medium article
→ More replies (5)1
u/diamondketo Jul 22 '21
Then use a different namespace. Don't hack it in the function signature.
A cool way to do is to set the namespace on the function attr.
def f(): ... f.cache = []IMO, I rather use a class but I get it, sometimes you don't want to rewrite everything as a class.
1
u/jet_heller Jul 22 '21
I'm sorry you think something specifically left in there by the language designers is a hack. It's not.
→ More replies (2)-1
u/FranticToaster Jul 22 '21
That's what I was thinking. Using this nuance of Python could make code easier to manage.
Rather than defining a global object and then operating on it with a function, the object could be the default arg of the function. That way, it's clear that the object belongs to the function. That its purpose is explicitly to serve as a complement to the function.
3
u/menge101 Jul 22 '21
If you were to do this, you would unambiguously be commenitng it with this intention clearly stated. On top of this, I think most linters would flag this, so you'd have a
# noqacomment on there as well.For me, as a matter of code review, I don't allow linter checks to be disabled without a comment explaining why.
2
u/swierdo Jul 22 '21
Don't allow general noqa. It should always except a specific error. Also,
# noqa: E501(line length) doesn't really need a comment.2
4
u/the_pw_is_in_this_ID Jul 22 '21
Thankfully, until you're senior enough to understand when a static mutable object is useful, you don't want to touch them yet. There's a lot of room for unexpected behavior in that combo.
1
Jul 22 '21
The fact is, that even if you actually wanted a persistent object, that would still be a whacky way to do it...
1
u/asday_ Jul 23 '21
def f(x=[]): passFunction's state changes over time, but isn't a generator. Surprising behaviour, probably a bug.
x = [] def f(): global x passVery obviously intentional, but uses a global. Kinda icky.
class Fer: def __init__(self): self.x = [] def f(self, x=None): if x is not None: self.x = x ... f = Fer().fUglier, but reuseable, and no global modification.
1
u/NegativeK Jul 27 '21
Blanket do/don't statements are a good idea when communicating to people who either aren't ready for or don't have the time for more nuanced points.
Don't ban don't statements.
1
u/jet_heller Jul 27 '21
All this does is teach people that nuance is not relevant. It doesn't help them learn a damn thing.
1
u/NegativeK Jul 27 '21
That's... Not true?
A beginner who is just learning about default arguments already has a lot that they're trying to remember. They'll pick up nuance later -- just like everyone does as they grow.
1
0
Jul 22 '21
A good practice would be to use typing+mypy which would tell that you are doing the wrong thing
11
u/TomBombadildozer Jul 22 '21
mypy won't warn you about mutable defaults. If the types are compatible, that's all it cares about.
Example
bad.py:from typing import List def f(a: List[str] = []) -> None: passRun mypy:
$ mypy bad.py Success: no issues found in 1 source fileTo do this correctly, use
Optional:from typing import List, Optional def f(a: Optional[List[str]] = None) -> None: a = a or []2
u/Ffsauta Jul 22 '21
Your ‚correct‘ solution will not work in cases where ‚a‘ is an output argument. If you want to optionally accumulate some intermediate results and pass an empty accumulator list it will be thrown away and nothing will be accumulated in the empty list you provided.
The function body should be ‚a = [] if a is None else a‘
1
u/TomBombadildozer Jul 22 '21 edited Jul 22 '21
The function body should be ‚a = [] if a is None else a‘
Your code and my code do exactly the same thing.
Python's ternary operator is hideous, and in this case, is unnecessarily verbose. We've constrained acceptable values to
Noneand lists of strings. Any value passed in that isn'tNoneor a list of string will be rejected by mypy, andNoneor empty list will be reassigned to[], which is exactly what we want.Edit: now realizing what you meant by this:
Your ‚correct‘ solution will not work in cases where ‚a‘ is an output argument.
Don't do that. This isn't C, where it's difficult to accumulate values without writing into a buffer passed into the function. Do your accumulation in the function and return a new list.
5
u/stratoscope Jul 22 '21
Your code and my code do exactly the same thing.
No, they don't do the same thing.
An empty list is "falsy". So in the case where
ais already an empty list,a = a or []replacesawith a different empty list.This can be demonstrated like so:
>>> a = [] >>> b = a >>> a is b True >>> a = a or [] >>> a is b False >>>In many cases this will not matter. But it could, and at the very least the use of
a = a or []may leave others wondering if there is a subtle bug lurking here.I agree with you that
a = [] if a is None else ais not a good way to do this, but it does have the correct behavior (it setsato an empty list only if it wasNone). It's much better to use:if a is None: a = []Or if you like a one-liner:
if a is None: a = []1
u/asday_ Jul 23 '21
Your code and my code do exactly the same thing.
>>> def f(a=None): ... a = a or [] ... print(a) ... >>> def g(a=None): ... a = [] if a is None else a ... print(a) ... >>> f(False) [] >>> g(False) False >>>
-3
u/Nidsan Jul 22 '21
Unless, this is the effect you desired. A stateful function.
3
u/mxzf Jul 22 '21
In the rare instance that it is what you wanted, you can always ignore the rule. If you understand why it's happening, you can make that decision yourself (and if you don't understand why it's happening, you should follow the article's advice).
1
u/Nidsan Jul 22 '21
No I don’t think there will ever be a case why you need this. I’ve been wrapping my head around why anyone would possibly need this and I can’t see any. You can have global or local variables that are more explicit and readable. You can also use functools cache and lru_cache if caching is the effect you desired. Or you can also just use a oop approach. Doing this goes against the zen of python in many ways.
1
u/mxzf Jul 23 '21
I agree that it's not something anyone should be doing. My point was that the advice is sound, and if anyone really needs to break the rule, they should only be doing so because they know exactly what and why they're doing what they're doing.
1
u/paul_miner Jul 23 '21
Even if you wanted this behavior, relying on a counterintuitive quirk would be a bad way to implement it.
1
u/asday_ Jul 23 '21
If only there was some way to associate a function with some kind of state.
1
u/Nidsan Jul 23 '21
If only we could put the function into a class, or use functools. Or just put the function inside another function and use nonlocal
1
-2
Jul 22 '21
Is this not considered a bug? Why don’t they change this in new versions?
7
u/luther9 Jul 22 '21
The main reason I'm aware of is performance. It's faster to store a default argument in the function object than to re-evaluate that argument every time the function runs. If the function runs many times, this speeds things up a lot.
1
u/asday_ Jul 23 '21
How is that an argument when you can use a global?
1
u/luther9 Jul 23 '21
What do you mean? A global name still has to be evaluated. Were you trying to reply to someone else?
1
u/asday_ Jul 23 '21
You have some immense misconceptions about how the Python interpreter works I think. ANY name has to be "evaluated", whether it's a local, arg, global, whatever. As a matter of fact, function arguments become locals within that function scope, so they have the exact same performance impact to use as something in globals.
Let's start again. Benjamin posits that the mutable default argument behaviour is bug-like. You posit that there's a reason to keep it, and that that reason is for performance. I point out that you can use a global instead of the function argument for things you'd like to mutate, the implication being "to not have to use mutable function default arguments" or "in a world where there aren't mutable function default arguments". Do you disagree with that?
1
u/luther9 Jul 23 '21
You have some immense misconceptions about how the Python interpreter works I think. ANY name has to be "evaluated", whether it's a local, arg, global, whatever. As a matter of fact, function arguments become locals within that function scope, so they have the exact same performance impact to use as something in globals.
Yes, of course I know that. The performance impact I'm talking about is not in the body of the function, but in establishing the default arguments themselves. Python stores the default arguments in the function's
__defaults__attribute so they don't have to be evaluated again. [0] The only alternative to this that I can think of would be for Python to add code to the function that can evaluate the default argument expressions when the function gets called. Do you agree that evaluating an expression once is faster than evaluating it many times? Sure, you'd have to factor in the attribute and tuple lookups, but that's surely faster than allocating memory for a new object, or whatever else an expression might do, which could be anything.Your second paragraph appears to come from misunderstanding the point I was making. Hopefully, I've cleared that up.
[0] https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy
7
u/KingHavana Jul 22 '21
This is intentional choice and a well known behavior. Like /u/jet_heller mentioned, you can use this technique to keep track of information between different calls of the function if you wanted. I understand it's confusing at first, but there are easy ways around it.
-17
u/istira_balegina Jul 22 '21
This is the second time this article has been posted and the headline is garbage advice.
Mutable objects are very useful as default arguments for caching/memoization.
Please delete.
21
u/TomBombadildozer Jul 22 '21
Mutable objects are very useful as default arguments for caching/memoization.
This is a terrible suggestion. Please don't do this. If someone reading the code didn't know about this implementation detail in Python, the behavior would be entirely counterintuitive.
>>> def f(x, lst=[]): ... lst.append(x) ... print(lst) ... >>> f('a') ['a'] >>> f('b') ['a', 'b'] >>> f('d', lst=['c']) ['c', 'd'] >>> f('e') ['a', 'b', 'e']You might expect
f('e')to produce['c', 'd', 'e']since we "replaced"lstin the previous call. That's not the case becauselstin the function definition is allocated in heap memory, whereaslstinf('c', lst=['c'])is allocated on the stack.If you need to do memoization, use the features provided in
functools.5
u/licht1nstein Jul 22 '21
Can you provide an example?
→ More replies (4)1
u/TSM- 🐱💻📚 Jul 22 '21
def get_from_cache(name, cache={}): if name in cache: return cache[name] cache[name] = expensive_calculation() return cache[name]Though usually memoization is done with decorators because linters and IDEs will warn you about mutable defaults as they trip people up and are easy to miss at a glance.
You'd instead do something like
from functools import cache @cache def factorial(n): return n * factorial(n-1) if n else 1 >>> factorial(10) # no previously cached result, makes 11 recursive calls 3628800 >>> factorial(5) # just looks up cached value result 120 >>> factorial(12) # makes two new recursive calls, the other 10 are cached 4790016002
u/earthboundkid Jul 22 '21
This also works and doesn't use a mutable arg:
def get_from_cache(name): if name in get_from_cache.cache: return get_from_cache.cache[name] get_from_cache.cache[name] = expensive_calculation() return get_from_cache.cache[name] get_from_cache.cache = {}7
u/Ph0X Jul 22 '21
It's always better to just define a global object and use that for caching than imply on the implicit fact that kwarg default values are mutable.
Explicit is better than implicit.
Or better, use functools.cache
2
u/Nidsan Jul 22 '21
What if I have functions that produce a function? Then I can’t have global variables because I want these functions created dynamically.
1
1
u/Ph0X Jul 22 '21
I'd need to see a specific, but I believe this is what nonlocal does?
def outer_func(): cache = {} def inner_func(foo): nonlocal cache if foo not in cache: cache[foo] = heavy_computation(foo) return cache[foo] return inner_func-1
u/istira_balegina Jul 22 '21
Not according to the docs, which says it is up to the programmer's discretion.
Also on enterprise programs default arguments can be considered cleaner than global objects.
3
u/Ph0X Jul 22 '21
A lot of things are up to the programmers discretion. Unlike some other languages, Python doesn't really babysit the programmer, it let's you do a lot. Hell you can completely override the import mechanism or replace all the modules in sys.modules if you want. You can set
True = Falseor replaceint.__add__. Doesn't mean any of these are a good idea.To be clear, I'm talking about code that will be seen by other people (though, yourself in 6 month also counts as someone else in most cases). You're free to do whatever you want in your own personal code, but if I ever saw this in a code review, I would shut it down real quick.
1
u/istira_balegina Jul 22 '21
That is true but python does have a style guide and in this instance the docs themselves state their recommendation.
2
u/Ph0X Jul 22 '21 edited Jul 22 '21
Link?
EDIT: The only code style I found is Google's, and they explicitly state to never do this: https://google.github.io/styleguide/pyguide.html#2124-decision
Most companies I know of have similar internal style gudes.
EDIT2: python-guide.org vaguely implies that it's fine to use, but that's not official Python either, it's some opinionated book.
1
u/istira_balegina Jul 22 '21
Here:
https://docs.python.org/3/faq/programming.html#id13
"Because of this feature, it is good programming practice to not use mutable objects as default values. Instead, use None as the default value and inside the function, check if the parameter is None and create a new list/dictionary/whatever if it is. For example, don’t write:
def foo(mydict={}): ...
but:
def foo(mydict=None): if mydict is None: mydict = {} # create a new dict for local namespace
This feature can be useful. When you have a function that’s time-consuming to compute, a common technique is to cache the parameters and the resulting value of each call to the function, and return the cached value if the same value is requested again. This is called “memoizing”, and can be implemented like this:
Callers can only provide two parameters and optionally pass _cache by keyword def expensive(arg1, arg2, *, _cache={}): if (arg1, arg2) in _cache: return _cache[(arg1, arg2)] # Calculate the value result = ... expensive computation ... _cache[(arg1, arg2)] = result # Store result in the cache return result
You could use a global variable containing a dictionary instead of the default value; it’s a matter of taste."
→ More replies (1)1
u/TSM- 🐱💻📚 Jul 22 '21
You can set
True = FalseSlight nitpick, but you cannot actually do that.
>>> True = False File "<stdin>", line 1 SyntaxError: cannot assign to True # (same goes for literals)Though I totally agree with everything you said. Replacing
__add__for fun instead of having an.add_tofunction is potentially confusing to others.I think you can make exceptions for mini-languages, like how pathlib overrides the division operator for path concatenation. And of course dataclasses. But generally speaking it is a bad idea.
Also, the Google style guide is a great reference for beginners. I recommend it to people all the time, it has good examples of the "do's and don'ts".
2
u/Ph0X Jul 22 '21
Slight nitpick, but you cannot actually do that.
Hah, they indeed fixed that in Py3, definitely could in Py2.
pathlib overrides the division operator for path
To be clear, overriding operators isn't the issue, doing it on a base class is the problem. pathlib does it only on pathlib.Path objects, which is well within it's reign.
2
3
u/mxzf Jul 22 '21
It remains good advice. There are exceptions to every rule, including this one, but the advice remains solid. If you need to break this "rule", then you know you need to and you know why and how. For the vast majority of people, however, following this advice is the correct course of action.
-9
u/stackered Jul 22 '21
I don't see why anyone would do anything this way regardless. Its basically a non issue you should be aware of... I think people, especially Python developers, care way too much about cutting a line or two out of their code and being "Pythonic" rather than making things more readable and adding a line or two... this problem is essentially avoided by just using new variables. Default arguments = noob IMO
1
u/BeKenny Jul 22 '21
How is something built into the language a non issue? The reason people would do it this way is because they can and they didn't learn otherwise. You can make an argument for never using default arguments ever if you'd like, but no need to be condescending.
→ More replies (1)
0
Jul 22 '21
[deleted]
92
u/stratoscope Jul 22 '21
It's customary to use
is Noneinstead of== None, and some linters will flag the latter.So I would suggest this:
if arr is None: arr = []25
u/danuker Jul 22 '21
Here's a brain twister:
>>> False is not None True >>> False is (not None) False16
u/Legionof1 Jul 22 '21
>>> (False is not None) == (False != None) True >>> not_none = not None; (False is (not None)) == (False == not_none) True27
1
u/Swinghodler Jul 23 '21
Is there anything that is (not None) ?
4
Jul 23 '21
The truthiness of None is always
False, sonot NoneisTrue.Since true and false are both singletons, the only thing that IS (not None) would be True.
The important part is the (), because this makes it a separate expression.
False is not Nonemeans, is the object False not the object None?
False is (not None)means, is the object False the object returned from the expression (not None)?3
4
u/fomorian Jul 22 '21
Should you always replace == with is, or only the this specific case where you are equating it with None?
9
Jul 22 '21 edited Jul 22 '21
There is only one None, so
is Noneworks; but unless you're dealing with these special cases (None,True,False) or unless you really want to check an object by id, you should use==. You can have two identical objects which do not share the same object ID and usingiswill evaluate to False with them. (sorry for the bad English).Ex,
a = {1: 2}a is {1: 2}will evaluate to False, because
ais not the dictionary object{1: 2}which was just created in the linea is {1: 2}.5
u/TSM- 🐱💻📚 Jul 22 '21
Literals should use
==. Althoughis Noneis common and doesn't give you a warning, literals compared withisgives you a syntax warning.>>> a=""; a is 5 <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="? FalseThis is because literals may not always be the same object. Two "asdf" strings may or may not be different objects depending on how they are formed (like at compile time and a dynamically constructed string with the same characters, the
iswill be false).Simple example with numbers:
>>> a = 1234 >>> b = 1234 >>> a is b Falseyet if defined simultaneously:
>>> a = b = 1234 >>> a is b Truebut:
>>> a = 123 >>> b = 123 >>> a is b TrueThat is why there's the syntax warning. Literals should be compared by value and not object identity, since the same literal value can be different objects in memory.
None is a special object like
FalseorTrue. It's a 'singleton' at runtime and always the same object.It is also maybe worth mentioning that
==can theoretically be overridden by an object's__eq__method (not that anyone does that).Usually of course
Noneused as a default argument/parameter. The expressionif foo is None:is sometimes preferred overif not foo:because the second one is is satisfied if it's an empty list, False, "", 0, or other "Falsy" values, when you are actually wanting to see if the optional argument was specified.3
u/TankorSmash Jul 22 '21
I think two short strings will always pass
is, right? Something to do with internal optimization I thought.2
u/TSM- 🐱💻📚 Jul 22 '21 edited Jul 22 '21
That's sort of right, but consider with respect to your example:
>>> foo = "asdf" >>> bar = "asd" + "f" >>> foo is bar True >>> bar = "asd" >>> bar = bar + "f" >>> foo is bar FalseThat is because it only uses the string cache when it can be directly deduced from the line and other conditions. But then again, only sometimes (like
b = ''.join('asdf')will beFalse). It's a fragile optimization. String interningIt also can easily differ between implementations and versions, like CPython vs PyPy or MicroPython. So your micropython code will behave differently from when you did it on desktop. I'm glad they added that syntax warning (and the new detailed Rust-like error messages).
1
u/algag Jul 22 '21 edited Apr 25 '23
......
1
u/Rabid_Gopher Jul 22 '21
I just threw "500 is 500" in a python interpreter and got True, can you clarify the difference and maybe why I should have gotten False?
7
u/algag Jul 22 '21 edited Apr 25 '23
.....
2
Jul 22 '21
where do large integers start and small integers stop?
3
u/qwerty1793 Jul 22 '21
In CPython the "small integers" are -5, -4, ..., 255, 256 and any other integer is a "large integer". However this is an implementation detail and is not part of the Python language specification. So if you switch to a different implementation such as PyPy or JPython you can see different behaviour.
2
u/stevenjd Jul 23 '21
The Python interpreter is free to cache or not cache immutable objects as it sees fit. This can and will change from version to version, and interpreter to interpreter (e.g. CPython, PyPy, IronPython, Jython, Stackless, Nuitka, ...). It can even change depending on whether you are running in the interactive interpreter or not.
In current versions of CPython, small integers in the range -5 through to 256 are permanently cached, and will always re-use the same object, but in the interactive interpreter it might temporarily reuse other values as well. So for example:
# Python3.9 >>> a = 600 >>> b = 600 >>> a is b False >>> a = 700; b = 700 >>> a is b TrueThe difference is that in the second example, both assignments are part of the same input line (separated by semicolons) so the interactive interpreter knows that there are two values 700 and can use the same object for both. Whereas in the first example the interpreter doesn't see the second value 600 until it has already finished with the first value 600, so it uses a different instance.
None of this is guaranteed by the language and it can all change without notice or warning, including in bug releases. You should not rely on it.
CPython has no cache for floats (even 0.0 is not cached) but the interactive interpreter is smart enough to temporarily reuse them:
>>> a = 17.34109; b = 17.34109 >>> a is b TrueDid I mention that you should not rely on any of this behaviour?
1
u/axonxorz pip'ing aint easy, especially on windows Jul 22 '21 edited Jul 22 '21
There must be some magic in the interpreter specifically for this case as I have the same behaviour on 3.9.6 and 2.7.18.
When you're using
a is b, you're essentially just doing shorthand ofid(a) == id(b)Python pre-caches small integers in the range [-5, 256] at startup (can be seen by doing
id(<small_int>), and getting a different value for each int). If you doid(<big_int>)outside the pre-cached range, you get a static number that doesn't change, no matter which int is requested.This is where it gets weird as
5000 is 5001is the same asid(5000) is id(5001), which returns False, even though (on my particular invocation)>>> id(5000) 139743277952304 >>> id(5001) 139743277952304And yet...
id(5000) is id(5001) # returns False # should be the same as 139743277952304 is 139743277952304 # returns Trueedit: as /u/qwerty1793 mentions below, it's only [-5, 256], not [-256, 256], not sure how I ended up with that range
1
u/qwerty1793 Jul 22 '21
CPython only pre-caches integers in the range [-5, 256]. This is also an implementation detail which is not in the Python language specification. So if you switch to a different implementation (for example, PyPy or JPython) you can see different behaviour.
1
u/irondust Jul 22 '21
That's just because in
id(5000)It creates the object "5000" on the fly, but then because you don't assign it to anything it can be immediately garbage collected, and the same id number can be assigned to the next object that gets created. Try evaluating
id(5000)a few times2
u/axonxorz pip'ing aint easy, especially on windows Jul 22 '21
Huh, didn't know the GC would kick in that fast, you seem to be correct as if I assign 5000 to something, that id is persistent, and is not shared across instances of 5000
1
u/qwerty1793 Aug 19 '21
So the statement `500 is 500` does not create two object. Since this is entirely withing a single statement, it can be optimised to create two references to a single `500` object. You can see this by disassembling the code:
>>> dis.dis(compile('500 is 500', '', 'exec'))
1 0 LOAD_CONST 0 (500)
2 LOAD_CONST 0 (500)
4 COMPARE_OP 8 (is)
6 POP_TOP
8 LOAD_CONST 1 (None)
10 RETURN_VALUEYou need multiple statements in order to force python to create multiple objects:
>>> x = 500
>>> y = 500
>>> x is y
False0
Jul 22 '21
[deleted]
19
u/avamk Jul 22 '21
Your comment is literally the first time the distinction between
==andisclicked with me. Thank you! :D-1
u/GraphicH Jul 22 '21
I prefer:
arr = arr or []When defaulting arguments with mutable objects.
Also, someone needed an article about this?
0
Jul 22 '21
[deleted]
4
Jul 23 '21 edited Jul 23 '21
Be careful though. It’s fine in most cases but an array is truths, so an empty array will be false. So if you pass an empty array to the function, it may provide different results than you expect.
def append10(arr=None): arr = arr or [] arr.append(10) return arr a = [] append10(arr=a) print(a) a = [1] append10(arr=a) print(a)Would output
[]
[1, 10]
0
Jul 23 '21
[deleted]
1
Jul 23 '21
My point is that there is a difference between checking the truthiness vs checking if an object is the same. In simple examples, like this one, it's fine, but people should be aware of the difference because it can and likely will cause bugs if you're developing larger applications or libraries to be used by the public.
That's why the zen of python says "explicit is better than implicit".
By using
arr = arr or []if I already had an empty list and pass that in, my list isn't being modified, but a new list is created.Similarly, what if I pass in 0, or ""? It would work, but I would be using the function wrong and have no idea, because it would just create a new list.
Here is a more comprehensive example:
>>> def append10(arr=None): ... arr = arr or [] ... arr.append(10) ... return arr ... >>> def append5(arr=None): ... if arr is None: ... arr = [] ... arr.append(5) ... return arr ... >>> a = [] >>> append10(arr=a) >>> a [] >>> append5(arr=a) >>> a [5] >>> append10(arr=a) >>> a [5, 10] >>> b = "" >>> append10(arr=b) >>> b '' >>> append5(arr=b) Traceback (most recent call last): File "<input>", line 1, in <module> File "<input>", line 4, in append5 AttributeError: 'str' object has no attribute 'append'Note that if I pass in anything other than a list, an object that implements an append method, or None, the append5 function will fail. Whereas passing any value that evaluates to false will work in append10, but won't modify my original object.
0
3
u/stevenjd Jul 23 '21
arr = arr or []Oh wow, I'm having a flashback to Python 2.4 and older! We've had ternary expressions for fifteen years now.
The above is fine until
arrmight be a falsey value that you want to use. Then you have a hard to diagnose bug in your code.5
u/diamondketo Jul 22 '21 edited Jul 23 '21
There is one flaw with this and that this destroys polymorphism.def f(arr=None): arr = arr or [] # do something with arr return arr
If the user inputs an array of type tuple, they may expect the output to be a tuple, but here you get a list back. Depending on the context, it might be very important to output the same type as the input.EDIT: This is not an issue actually. If the user doesn't input anything, they should expect the default behavior.
3
u/oathbreakerkeeper Jul 23 '21
Can you elaborate? I'm not sure what you're saying.
4
1
u/NerdEnPose Jul 23 '21
An empty tuple evaluates as False so this would replace a Tuple with a list. The other issue is that an empty list would be replaced as a different empty list in memory and might break something that is relying on this mutation. Both of these are cases are a bit of a stretch, IMO, given my idea of good practices but still. The main takeaway, again IMO, is that relying on the truthfulness of objects is never clear and an "is" conditional is still always preferred because it has less side effects. This seems cute to me.
0
Jul 22 '21
[deleted]
1
u/backtickbot Jul 22 '21
2
u/Own_Quality_5321 Jul 22 '21
Hello? Can everyone listen to this guy? Thanks for that beauty! I thought that the expression would just return True. Is fun to keep learning about these "hidden features" after years using Python!
In case anyone wants to know why this works: https://realpython.com/python-or-operator/
7
Jul 22 '21
without type enforcement there are a lot of things that would be evaluated as False. It's safer to have it explicit, even though slightly longer. Code shouldn't look hacky, it should be simple, easy to read, and safe.
0
Jul 22 '21
[deleted]
3
Jul 23 '21
The problem is the tendency to lay responsibility on other people.
So if there's a bug, someone else is responsible for calling this function wrong.
However, the bug could've easily been avoided by just not being subject to them.
If you're a developer that can make the code safer, then do it. Don't rely on other people to be perfect, because they're not, and there are also junior developers. Don't expect the next person to touch the code to be super smart.
Write the code in a way that allows even an idiot to avoid bugs.
1
u/Own_Quality_5321 Jul 22 '21
I take the point made two comments above, but I still agree here. It might not be a great idea for functions directly exposed to users, but it still seems better for internal code. Concise and easy to read if you know how it works (not a lot of people do, maybe, but that's probably because not a lot of people use it -vicious cycle).
This reminds me of the "do not use gotos and breaks" in C code, that are used even in the Linux kernel (https://www.kernel.org/doc/html/v4.17/process/coding-style.html).
0
Jul 22 '21
[deleted]
3
u/diamondketo Jul 22 '21 edited Jul 22 '21
Why would someone in their right mind even expect a collection to be evaluable as a boolean in the first place?
Okay, I assume you've never seen this famous behavior in Python.
if arr: ... else: print ("Array is empty")Evaluating truthy and falsey from various types (not just static types) is a Python perk and should've been taught in most courses.
2
u/RojerGS Author of “Pydon'ts” Jul 22 '21
It's a bit more general than just
or, it's called Boolean short-circuiting and it only works because the empty list has a Falsy value1
u/jackerhack from __future__ import 4.0 Jul 23 '21
If you use mypy for typechecking, it won’t like this. It’ll read the type of
arrasOptional[List]. However, if you useif arr is None: arr = [], it will recognise the type change to justList.0
0
Jul 22 '21
[deleted]
13
u/sumduud14 Jul 22 '21 edited Jul 23 '21
You aren't missing anything, it's just pointless obfuscation. It's "elegant" in the same way code golf is elegant.
If you want a one-liner, you can just use
if arr is None: arr = []Nothing wrong with that. Keep it simple.
In fact, it is better than the above code because I don't think it's obvious that the
ortrick will transform an empty set or a literal False into an array, but that's what it does.5
u/Pikalima Jul 23 '21 edited Jul 23 '21
I agree with this. As a general rule of thumb, it’s best not to rely on implicit boolean casting if you are specifically testing
None. Too many nights spent in the Python debugger caused by one of the many other false-y values in Python sneaking in unexpectedly. Also, another valid way to do this isarr = [] if arr is None else arr.→ More replies (1)4
u/RojerGS Author of “Pydon'ts” Jul 22 '21
I got you, this is called Boolean short-circuiting and you can read all about it here.
4
u/Smallpaul Jul 23 '21
Once you see it roughly 100 times in your team’s code base it is clearer than a conditional, because it is an expression of a very common pattern.
3
u/stevenjd Jul 23 '21
It's great to see Redditors and their teams still using bug magnets from the dark days of Python 1.5 in their code in 2021, fifteen years after conditional expressions were added to the language.
(Presumably just to save a few characters.)
Never change Reddit!
→ More replies (1)1
u/sumduud14 Jul 23 '21
Once you see it roughly 100 times in your team’s code base it is clearer than a conditional
Implicitly interpreting data structures as boolean in order to check for None is clearer than a conditional?
You seriously believe:
arr = arr or []Is more clear than:
if arr is None: arr = []I am astounded that you would claim something so obviously false.
The behaviour of the "or" method also switches around types in a non obvious way if you pass in a falsey value like False or an empty set.
→ More replies (3)2
Jul 23 '21
well .. I feel like nearly every programming language I've used behaves like this, feels intuitive after a bit? a lot of programming things aren't immediately intuitive but "feel great" once you have mastered them.
0
u/Caboose522 Jul 22 '21
It doesnt read well I agree, and in this case it probably wouldnt give any performance advantage. I like the concept though and I could think of other situations where this could make sense. The cool thing is if you grasp it you can use it to write more succinctly, and less code is usually a good thing.
→ More replies (4)0
16
u/anthony81212 Jul 22 '21 edited Jul 22 '21
Tip: use uBlock and block cookies from all medium-related domains 😉
Or it might work if you block cookies with your browser
Edit: oops, I might have meant uMatrix. Sorry, I will check tmrw and update my post here!
5
u/BridgeBum Jul 22 '21
Curious; trying to find this option in uBlock origin, not seeing it. Do you have to craft a custom rule?
2
u/anthony81212 Jul 22 '21
oops, I might have meant uMatrix. Sorry, I will check tmrw and update you here!
9
u/NaN_Loss Jul 22 '21
Just open the link in a private window
7
u/Grintor Jul 22 '21
That doesn't work, at least with Chrome. What does work though, is if you DM the message to yourself on Twitter and then click the link in the message. They never put Twitter referers behind the paywall.
3
u/axe319 Jul 22 '21
I know this is just an example but the method for adding to a
listisappend(), notpush().5
u/burlyginger Jul 22 '21
I prefer arr = arr or []
I've had this same issue, was annoying as hell to figure out why my 'empty' list had values in it :D
Thanks to the python discord on that one.
1
u/cbunn81 Jul 22 '21
I've seen this before in relation to initializing a mutable object. But what if the default is a list with some elements defined, and the list won't be altered by the function?
1
u/hehehahahohohuhuhu Jul 23 '21
Kinda off topic but to bypass the medium paywall, open up the devtools, select the applications tab and clear the local storage and cookies. Lastly refresh the site and it's good to go.
Not sure if there's an easier workaround available
1
0
Jul 22 '21
[deleted]
6
Jul 23 '21
I like that second way, verbose enough to be able to come back to it months later and understand what's going on
0
Jul 22 '21
[deleted]
1
u/got_outta_bed_4_this Jul 23 '21 edited Jul 23 '21
You can give a plain function a static variable by exploiting the fact that even a function itself is an object.
def keep_a_total(n: int) -> int: keep_a_total.total += n return keep_a_total.total keep_a_total.total = 0Essentially, initialize the "static" as a member of the function object right after defining the function object.
2
u/backtickbot Jul 23 '21
1
0
Jul 23 '21
[deleted]
1
u/asday_ Jul 23 '21
If Python didn't have this huge footgun we also wouldn't.
0
Jul 24 '21
[deleted]
1
u/asday_ Jul 24 '21
I don't think Python's ever been huge on doing something in a certain way just because other languages do. Iterating over dictionaries being keywise rather than valuewise is an old example,
matchis a new one.
0
Jul 23 '21
[deleted]
2
u/asday_ Jul 23 '21
>>> a = [10,20,30] >>> b = a >>> b[1] = 25 >>> print(a) [10, 25, 30] >>> print(b) [10, 25, 30] >>>Could be a layer 8 issue, or PEBCAK.
0
0
Jul 23 '21
[deleted]
1
u/asday_ Jul 23 '21
You wouldn't be wrong either.
>>> def f(x=0): ... print(x) ... >>> f() 0 >>> f.__defaults__ = (1, ) >>> f() 1 >>>
72
u/jjanx Jul 22 '21
This is my least favorite python footgun.