This blog was written as an IPython Notebook. Some of the implementation details it discusses are specific to CPython, the default implementation, and may work differently in other implementations. To keep the code clean, all necessary imports are made immediately below. I habitually import
print_function because I have become used to the Python 3 way of doing things, but this particular post was made using CPython 3.3.2.
In [1]:
from __future__ import print_function
from dis import dis
import sys
print(sys.version)
The Problem
In my Intermediate Python video series I discuss decorators, and point out that classes as well as functions can be decorated. As an example of this possibility I used a class decorator that wraps each of the class's methods in a function that prints out a message before calling the original method (or so I thought). Here is the relevant code.
In [2]:
def barking(cls):
for name in cls.__dict__:
if name.startswith("__"):
continue
func = getattr(cls, name)
def woofer(*args, **kw):
print("Woof")
return func(*args, **kw)
setattr(cls, name, woofer)
return cls
The essence of the decorator is to iterate over the class's
This seemed to work fine on a class with a single method.
__dict__, ignoring the so-called "dunder" names. Strictly we should perhaps also check that the name ends as well as begins with a double underscore, but it does not affect the example. Each method is then replaced with a wrapped version of itself that prints "Woof" before calling the original method.This seemed to work fine on a class with a single method.
In [3]:
@barking
class dog_1:
def shout(self):
print("hello from dog_1")
d1 = dog_1()
d1.shout()
I then defined a further class that inherited from
dog_1 with two additional methods, and decorated that. The inheritance is irrelevant: the significant fact is that two methods will be processed in the decorator loop, but it does demonstrate that superclass attributes do not appear in the subclass's __dict__.
In [4]:
@barking
class dog_3(dog_1):
def wag(self):
print("a dog_3 is happy")
def sniff(self):
print("a dog_3 is curious")
d3 = dog_3()
d3.wag(); d3.sniff(); d3.shout()
You have probably spotted the error already: calling both the
wag() and the sniff() methods gives the same result, which is to say that both wrapped methods in fact call the same original method. Unfortunately I missed this during development and production of the video code, but it was soon picked up by an eagle-eyed viewer who reported it via O'Reilly's web site.The Explanation
When a nested function makes references to variables in the containing function (non-local references) it is known as a closure. The interpreter has to ensure that the values referred to are still available after the call to the outer function terminates and the function call namespace is destroyed - otherwise, referenced objects could become garbage if the terminating function contained the only reference to the value. It does so by generating a cell object and storing the value in an element of the function's
__closure__ attribute, which is either None if the function does not refer to non-local cells or a tuple of non-local references.
The next two code blocks each define a function. The first is a regular function without non-local references, the second is a closure with a reference to the name
x from the enclosing function's namespace. You can see the difference in the bytecode reported by the dis module, and also that they return different results when created with and called with the same arguments.
In [5]:
def f_non_closure(x):
def inner(y):
return y
return inner
non_closure = f_non_closure(240)
dis(non_closure)
print("Value =", non_closure(128))
In [6]:
def f_closure(x):
def inner(y):
return x
return inner
closure = f_closure(240)
dis(closure)
print("Value =", closure(128))
You can clearly see the difference in the code. The non-closure uses a
LOAD_FAST operation that loads a local value from element 0 of the local namespace. The closure uses LOAD_DEREF, which loads a value from the function's __closure__ attribute. So let's take a look at the __closure__ of both functions.
In [7]:
print(non_closure.__closure__)
print(closure.__closure__)
Unsurprisingly the regular function does not have a value for its
_closure__ attribute. The closure, however, does - it is a tuple of cell objects, created by the interpreter. And if we want, we can obtain the values associated with the cells.
In [8]:
print(closure.__closure__[0].cell_contents)
This should make it obvious that the cell objects in the
__closure__ tuple are there so that values from the enclosing namespace remain available after that namespace has been destroyed. Those values herefore do not become garbage when the enclosing function terminates, returning the inner function (which is a closure). And the LOAD_DEREF 0 opcode simply loads the contents of the cell. It puts the function's __closure__[0].cell_contents onto the stack to be used (in this case) as a return value (and because it's written in C, it's much faster than the Python).
Let's complicate things a little by writing a function that returns a tuple of functions. The intention is that each function returned in the tuple should add successively larger numbers to its argument
x: the first one should return x+0, the second x+1, the third x+2 and so on. You find, however, that this does not happen.
In [9]:
def multiple_closures(n):
functions = []
for i in range(n):
def inner(x):
return x+i
functions.append(inner)
print(inner, "Returned", inner(10), inner.__closure__[0])
return functions
functions = multiple_closures(3)
print([f(10) for f in functions])
The output above might at first glance seem confusing. During creation of the functions inside
multiple_closures() they appear to work perfectly well. After returning from that function, however, they all return the same result when called with the same argument. We can find out why by examining the __closure__ of each function.
In [10]:
for f in functions:
print(f.__closure__[0])
The reason that the functions all give the same result is that they all use the same cell in their
__closure__. The interpreter assumes that since all functions refer to the same local variable they can all be represented by the same cell (despite the fact that the variable had different values at the different times it was used in different functions). Precisely the same thing is happening with the decorators in Intermediate Python.
If you read the code of the
barking() decorator carefully you can see that name, func and woofer are all names local to the barking() decorator function, and that func is used inside the inner function, making it a closure. Which means that all the methods end up referring to the last method processed, apparently in this case sniff().
In [11]:
print(d3.wag.__closure__[0].cell_contents.__name__) # In Python 2, use __func__
print(d3.sniff.__closure__[0].cell_contents.__name__) # In Python 2, use __func__
It now becomes obvious that the
func references in the inner woofer() function are all using the same local variable, which is represented by the same cell each time the loop body is executed. Hence, since a cell can only have a single value, they all refer to the same method.
In [12]:
print(d3.sniff.__closure__)
print(d3.wag.__closure__)
Is This a Bug?
I suspect this question is above my pay grade. It would certainly be nice if I could get this code to work as-is, but the simple fact is that at present it won't. Whether this is a bug I am happy to leave to the developers, so I will post an issue onbugs.python.org and see what they have to say. There's also the question of whether any current code is likely to be relying on this behavior (though I rather suspect not, given its unhelpful nature) - backwards compatibility should ideally not be broken.Workaround
The issue here is that different uses of the same non-local variable from a function will always reference the same cell, and no matter what the value was at the time it was referenced the cell always contains the final value of that variable.So a fairly simple, though somewhat contorted, workaround is to avoid multiple uses of the same non-local variable in different closures.
In [13]:
def isolated_closures(n):
functions = []
for i in range(n):
def wrapper(i=n):
def inner(x):
return x+i
return inner
f = wrapper(i)
functions.append(f)
print(f, "Returned", f(10), f.__closure__[0])
return functions
functions = isolated_closures(3)
print([f(10) for f in functions])
In the code above,
inner() is still a closure, but each time it is defined the definition takes plce in a different local namespace associated with a new call to wrapper(), and so each cell is a reference to a different local (to wrapper() - nonlocal to inner()) variable, and they do not collide with each other. Redefining the barking() decorator as follows works the same trick for that.
In [14]:
def barking(cls):
for name in cls.__dict__:
if name.startswith("__"):
continue
func = getattr(cls, name)
def wrapper(func=func):
def woofer(*args, **kw):
print("Woof")
return func(*args, **kw)
return woofer
setattr(cls, name, wrapper(func))
return cls
In [15]:
@barking
class dog_3(dog_1):
def wag(self):
print("a dog_3 is happy")
def sniff(self):
print("a dog_3 is curious")
d3 = dog_3()
d3.wag(); d3.sniff(); d3.shout()
This issue was quite fascinating to debug, and has led me to a much greater understanding of exactly how closures work in Python (though I still have to investigate some details). If I get time I may take a look at the CPython source to determine whether a fix can easily be made, but I wouldn't hold your breath. At this stage I have found the workaround I needed, and so will be able to respond to the viewer who raised it in the first place.
8 comments:
Some comments coming from the .NET world:
Microsoft actually made a breaking change to their C# compiler to avoid this issue altogether. I think they concluded it was too easy a mistake to make when using lambda expressions within a loop.
See http://blogs.msdn.com/b/ericlippert/archive/2009/11/12/closing-over-the-loop-variable-considered-harmful.aspx
Resharper used to add a warning to your C# code if you did this in visual studio. I've just tried the following in PyCharm (also from JetBrains, the same guys that make resharper):
x = []
for i in range(100):
x.append(lambda: i*2)
and too bad, the same warning is not issued by PyCharm
The traditional remedy to this situation is to take advantage of the fact that keyword argument defaults are evaluated at definition-time rather than at call-time.
In fact, providing a remedy for situations like this might be the whole reason for that convention in the first place, since it otherwise seems to always be a surprise to all programmers everywhere that keyword argument expressions are evaluated during the definition.
In any case, try this in your first example:
def woofer(*args, func=func, **kw):
print("Woof")
return func(*args, **kw)
You will find that during each definition, Python grabs the particular object that "func" references and makes it the default for the keyword "func" of that particular version of the function.
Thanks, @Brandon. In point of fact I believe this used to be the recommended way to achieve "closure-like" behavior before the language formalized the relationship between lexically-nested scopes.
It's certainly a much more elegant way to transfer a value through to the inner namespace, but there is the possible danger that the keyword argument name will conflict with a legitimate keyword argument of the wrapped function.
I think this one is related: http://bugs.python.org/issue7423
I stumbled upon the fact that this works as expected:
zip(*([(i, j) for j in 'ab'] for i in range(2)))
while this doesn't:
zip(*(((i, j) for j in 'ab') for i in range(2)))
@anonymous yes, the two do seem related. Thanks for your comment.
This is how it works in javascript closures as well. When using in closures, the value is tied to the variable and not the copy of the variable (Javascript good parts - Douglas Crockford)
Basically it needs a new namespace and the hack you showed is what it does ("wrapper" which is invoked with the value to create a new namespace for that function).
Great article! didn't realize it stores it in __closure__ (though should have guessed by now. For all sorts of magic, check dunder methods first)
Sai
@agileseeker
The problem here is the semantics of the "for" loop, rather than specifically closures. You can get obscure bugs from for loops anywhere that you don't duplicate the value of the internal reference. I imagine that'd be hard to clean up without seriously affecting the normal function of the for-loop, though, as it relates to the ability of for-loops to efficiently enclose scope and avoid leaking internal scoped variables.
Anyways, you can avoid this by never defining something in a for-loop scope that'll be strongly referential and re-used. I re-wrote the "barking" decorator as follows, and it works as intended:
def barking(cls):
def woofit(func):
def woofer(*a, **kw):
print("Woof")
return func(*a, **kw)
return woofer
for name in cls.__dict__:
if name.startswith("__"):
continue
func = getattr(cls, name)
setattr(cls, name, woofit(func))
return cls
(define (multiple-closures n) ; Scheme system : Racket v6.x
(define functions '())
(for ([i (in-range n)])
(define (inner x)
(+ x i))
(set! functions (cons inner functions)) ; reverse order, don't bother
(printf "~a returned ~a\n" inner (inner 10)))
functions)
(define functions (multiple-closures 3))
(for [(f functions)]
(printf "~a " (f 10))) ; 12 11 10 NOT THE SAME AS PYTHON.
Let me say I prefer Scheme behavior... Maybe due to the semantics of the for loop, in Scheme fresh locations are chosen at each iteration :
http://docs.racket-lang.org/reference/for.html?q=for#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29
Post a Comment