July 8, 2014

Is Python Under Threat from the IRS?

Readers should remember that I no longer have any official standing in the Foundation other than my long-standing membership, but this post is informed by eight years on the board, over three of them as its chair.

Nicholas Tollervey (author and performer of A Programmer Pythonical) asked an interesting question on the Python Software Foundation's membership list today. Despite his misgivings it is one that will be of interest to many Python users, so I am grateful for his permission to quote the essence of it here.

I've noticed rumblings on the web about the IRS denying nonprofit/charitable status to various free software based foundations/organisations/legal entities similar to the PSF. In case you've missed it, here is a high level summary.
My questions are:
1) Is the PSF likely to be affected by this change of view from the IRS?
2) If so, do we have contingency for dealing with this (basically,
what are our options)?

The answer to the first question is absolutely not. The PSF is a long-standing 501(c)(3) non-profit in good standing. The recently-reported issues are all to do with current applications for non-profit status. It's fair to say that the IRS is applying scrutiny to such applications, but they are, after all, responsible for making sure that applications are genuine. As long as an existing non-profit makes the necessary returns and complies with all applicable laws, and as long as it continues to honor its requirement to maintain broad public support, there is no reason why the IRS would represent any kind of threat.

The IRS does not exist to help open source
devotees to build a bright new world

Some of those rejected have been advised to apply for a different form of non-profit status, others are reconsidering their options. Good legal advice is imperative when starting any such enterprise, and that requires a specialist. There is a feeling that the IRS might make better decisions if it could be better-informed about open source. From my limited knowledge of the recent cases I'd say that it's essential to ensure that your application has a sound basis in non-profit law. The IRS does not exist to help open source devotees to build a bright new world.

The answer to the second question is that loss of non-profit status would be a blow to the Foundation, but there's no reason why even that should be fatal to Python, though I very much doubt there is serious planning for such an eventuality. Ultimately the Foundation's the bylaws include a standard winding-up clause that requires asset transfer to a similar/suitable non-profit, so assets cannot be stripped even under those circumstances, including the ability to license Python distributions.

The developers could simply migrate to a different development base. They aren't directed or controlled in any way by the Foundation, which in the light of recent decisions it turns out is probably a good model. If the PSF were directing the development of the language there might have been a real risk of it being seen as no different from a software house or vendor, and it is at that point that doubts about non-profit status can be raised.

The Foundation's mission is ... therefore
genuinely educational and charitable

The Foundation's mission is to promote the growth of the language (check) and the development of the international Python community (check) and its mission is therefore genuinely educational and charitable. So even were it to apply again today I suspect the application might succeed. Fortunately, that doesn't have to be put to the test.

July 2, 2014

Closures Aren't Easy

Edit: This issue was reported (http://bugs.python.org/issue21904) and closed as "not a bug"

This blog was written as an IPython Notebook. Some of the implementation details it discusses are specific to CPython, the default implementation, and may work differently in other implementations. To keep the code clean, all necessary imports are made immediately below. I habitually import print_function because I have become used to the Python 3 way of doing things, but this particular post was made using CPython 3.3.2.
In [1]:
from __future__ import print_function
from dis import dis
import sys
3.3.2 (default, Nov 19 2013, 03:15:33) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]

The Problem

In my Intermediate Python video series I discuss decorators, and point out that classes as well as functions can be decorated. As an example of this possibility I used a class decorator that wraps each of the class's methods in a function that prints out a message before calling the original method (or so I thought). Here is the relevant code.
In [2]:
def barking(cls):
    for name in cls.__dict__:
        if name.startswith("__"):
        func = getattr(cls, name)
        def woofer(*args, **kw):
            return func(*args, **kw)
        setattr(cls, name, woofer)
    return cls
The essence of the decorator is to iterate over the class's __dict__, ignoring the so-called "dunder" names. Strictly we should perhaps also check that the name ends as well as begins with a double underscore, but it does not affect the example. Each method is then replaced with a wrapped version of itself that prints "Woof" before calling the original method.
This seemed to work fine on a class with a single method.
In [3]:
class dog_1:
    def shout(self):
        print("hello from dog_1")

d1 = dog_1()
hello from dog_1

I then defined a further class that inherited from dog_1 with two additional methods, and decorated that. The inheritance is irrelevant: the significant fact is that two methods will be processed in the decorator loop, but it does demonstrate that superclass attributes do not appear in the subclass's __dict__.
In [4]:
class dog_3(dog_1):
    def wag(self):
        print("a dog_3 is happy")
    def sniff(self):
        print("a dog_3 is curious")

d3 = dog_3()
d3.wag(); d3.sniff(); d3.shout()
a dog_3 is curious
a dog_3 is curious
hello from dog_1

You have probably spotted the error already: calling both the wag() and the sniff() methods gives the same result, which is to say that both wrapped methods in fact call the same original method. Unfortunately I missed this during development and production of the video code, but it was soon picked up by an eagle-eyed viewer who reported it via O'Reilly's web site.

The Explanation

When a nested function makes references to variables in the containing function (non-local references) it is known as a closure. The interpreter has to ensure that the values referred to are still available after the call to the outer function terminates and the function call namespace is destroyed - otherwise, referenced objects could become garbage if the terminating function contained the only reference to the value. It does so by generating a cell object and storing the value in an element of the function's __closure__ attribute, which is either None if the function does not refer to non-local cells or a tuple of non-local references.
The next two code blocks each define a function. The first is a regular function without non-local references, the second is a closure with a reference to the name x from the enclosing function's namespace. You can see the difference in the bytecode reported by the dis module, and also that they return different results when created with and called with the same arguments.
In [5]:
def f_non_closure(x):
    def inner(y):
        return y
    return inner

non_closure = f_non_closure(240)
print("Value =", non_closure(128))
  3           0 LOAD_FAST                0 (y) 
              3 RETURN_VALUE         
Value = 128

In [6]:
def f_closure(x):
    def inner(y):
        return x
    return inner

closure = f_closure(240)
print("Value =", closure(128))
  3           0 LOAD_DEREF               0 (x) 
              3 RETURN_VALUE         
Value = 240

You can clearly see the difference in the code. The non-closure uses a LOAD_FAST operation that loads a local value from element 0 of the local namespace. The closure uses LOAD_DEREF, which loads a value from the function's __closure__ attribute. So let's take a look at the __closure__ of both functions.
In [7]:
(<cell at 0x10bbcb9f0: int object at 0x10a953ba0>,)

Unsurprisingly the regular function does not have a value for its _closure__ attribute. The closure, however, does - it is a tuple of cell objects, created by the interpreter. And if we want, we can obtain the values associated with the cells.
In [8]:

This should make it obvious that the cell objects in the __closure__ tuple are there so that values from the enclosing namespace remain available after that namespace has been destroyed. Those values herefore do not become garbage when the enclosing function terminates, returning the inner function (which is a closure). And the LOAD_DEREF 0 opcode simply loads the contents of the cell. It puts the function's __closure__[0].cell_contents onto the stack to be used (in this case) as a return value (and because it's written in C, it's much faster than the Python).
Let's complicate things a little by writing a function that returns a tuple of functions. The intention is that each function returned in the tuple should add successively larger numbers to its argument x: the first one should return x+0, the second x+1, the third x+2 and so on. You find, however, that this does not happen.
In [9]:
def multiple_closures(n):
    functions = []
    for i in range(n):
        def inner(x):
            return x+i
        print(inner, "Returned", inner(10), inner.__closure__[0])
    return functions

functions = multiple_closures(3)
print([f(10) for f in functions])
<function multiple_closures.<locals>.inner at 0x10c808440> Returned 10 <cell at 0x10c817948: int object at 0x10a951da0>
<function multiple_closures.<locals>.inner at 0x10c808560> Returned 11 <cell at 0x10c817948: int object at 0x10a951dc0>
<function multiple_closures.<locals>.inner at 0x10c808050> Returned 12 <cell at 0x10c817948: int object at 0x10a951de0>
[12, 12, 12]

The output above might at first glance seem confusing. During creation of the functions inside multiple_closures() they appear to work perfectly well. After returning from that function, however, they all return the same result when called with the same argument. We can find out why by examining the __closure__ of each function.
In [10]:
for f in functions:
<cell at 0x10c817948: int object at 0x10a951de0>
<cell at 0x10c817948: int object at 0x10a951de0>
<cell at 0x10c817948: int object at 0x10a951de0>

The reason that the functions all give the same result is that they all use the same cell in their __closure__. The interpreter assumes that since all functions refer to the same local variable they can all be represented by the same cell (despite the fact that the variable had different values at the different times it was used in different functions). Precisely the same thing is happening with the decorators in Intermediate Python.
If you read the code of the barking() decorator carefully you can see that name, func and woofer are all names local to the barking() decorator function, and that func is used inside the inner function, making it a closure. Which means that all the methods end up referring to the last method processed, apparently in this case sniff().
In [11]:
print(d3.wag.__closure__[0].cell_contents.__name__) # In Python 2, use __func__
print(d3.sniff.__closure__[0].cell_contents.__name__) # In Python 2, use __func__

It now becomes obvious that the func references in the inner woofer() function are all using the same local variable, which is represented by the same cell each time the loop body is executed. Hence, since a cell can only have a single value, they all refer to the same method.
In [12]:
(<cell at 0x10c817788: function object at 0x10c81f4d0>,)
(<cell at 0x10c817788: function object at 0x10c81f4d0>,)

Is This a Bug?

I suspect this question is above my pay grade. It would certainly be nice if I could get this code to work as-is, but the simple fact is that at present it won't. Whether this is a bug I am happy to leave to the developers, so I will post an issue on bugs.python.org and see what they have to say. There's also the question of whether any current code is likely to be relying on this behavior (though I rather suspect not, given its unhelpful nature) - backwards compatibility should ideally not be broken.


The issue here is that different uses of the same non-local variable from a function will always reference the same cell, and no matter what the value was at the time it was referenced the cell always contains the final value of that variable.
So a fairly simple, though somewhat contorted, workaround is to avoid multiple uses of the same non-local variable in different closures.
In [13]:
def isolated_closures(n):
    functions = []
    for i in range(n):
        def wrapper(i=n):
            def inner(x):
                return x+i
            return inner
        f = wrapper(i)
        print(f, "Returned", f(10), f.__closure__[0])
    return functions

functions = isolated_closures(3)
print([f(10) for f in functions])
<function isolated_closures.<locals>.wrapper.<locals>.inner at 0x10c826c20> Returned 10 <cell at 0x10c817e88: int object at 0x10a951da0>
<function isolated_closures.<locals>.wrapper.<locals>.inner at 0x10c8264d0> Returned 11 <cell at 0x10c817ec0: int object at 0x10a951dc0>
<function isolated_closures.<locals>.wrapper.<locals>.inner at 0x10c826b00> Returned 12 <cell at 0x10c817ef8: int object at 0x10a951de0>
[10, 11, 12]

In the code above, inner() is still a closure, but each time it is defined the definition takes plce in a different local namespace associated with a new call to wrapper(), and so each cell is a reference to a different local (to wrapper() - nonlocal to inner()) variable, and they do not collide with each other. Redefining the barking() decorator as follows works the same trick for that.
In [14]:
def barking(cls):
    for name in cls.__dict__:
        if name.startswith("__"):
        func = getattr(cls, name)
        def wrapper(func=func):
            def woofer(*args, **kw):
                return func(*args, **kw)
            return woofer
        setattr(cls, name, wrapper(func))
    return cls
In [15]:
class dog_3(dog_1):
    def wag(self):
        print("a dog_3 is happy")
    def sniff(self):
        print("a dog_3 is curious")

d3 = dog_3()
d3.wag(); d3.sniff(); d3.shout()
a dog_3 is happy
a dog_3 is curious
hello from dog_1

This issue was quite fascinating to debug, and has led me to a much greater understanding of exactly how closures work in Python (though I still have to investigate some details). If I get time I may take a look at the CPython source to determine whether a fix can easily be made, but I wouldn't hold your breath. At this stage I have found the workaround I needed, and so will be able to respond to the viewer who raised it in the first place.

April 21, 2014

Neat Notebook Trick

I'm still trying to digest the little I saw of PyCon. Sadly I was pretty wiped out by the three-day recording session for Intermediate Python in California and then the intensive editing work that followed to help the amazing O'Reilly team get the whole thing ready a day before PyCon officially opened.

I also made the tactical (and, as it turned out, strategic) mistake of choosing to stay at the Hyatt in Montreal. This meant a considerable walk (for a gimpy old geezer such as myself) to the conference site, when the Palais des Congres is already intimidatingly large.

So the combination of exhaustion and knee pain meant I hardly got to see any talks (not totally unheard of) but that I also got very little time in the hallway track either. Probably the most upsetting absence was missing the presentation of Raymond Hettinger's Lifetime Achievement Award. As a PSF director I instituted the Community Service Awards, but these have never really been entirely appropriate for developers. This award makes it much clearer just how significant Raymond's contributions have been.

Because of the video releases I did spend some time of the O'Reilly stand, and signed away 25 free copies of the videos. I was also collecting names and addresses to distribute free copied of the Python Pocket Reference. If you filled out a form, you should receive your book within the next three weeks. We'll mail you with a more exact delivery date shortly.

But the real reason for this post is that I had the pleasure of meeting Fernando Perez, one of the leaders of the IPython project. He was excited to hear that the Intermediate Python notebooks are already available on Github, and when he realized the notebooks were all held in the same directory he showed me that if I dropped that URL into the Notebook Viewer site I would get a web page with links to viewable versions of the notebook. [Please note: they aren't currently optimally configured for reading, so it's still best to run the notebooks interactively, but in the absence of a local notebook server this will be a lot better than nothing. It will get better over time].

He also mentioned a couple of other wrinkles I hadn't picked up on, and we briefly discussed some of the interesting aspects of Notebooks being data structures.

The conversation was interesting enough that I plan to visit Berkeley soon to try and infiltrate my way into the documentation team and see if we can't make the whole system even easier to use and understand. One way or another, open source seems to be in my bloodstream.

April 12, 2014

Intermediate Python: An Open Source Documentation Project

There is a huge demand for Python training materials, and there are many people who just don't have the spare cash to buy books or videos. That's one reason why, in conjunction with a new Intermediate Python video series I have just recorded for O'Reilly Media I am launching a new, open source, documentation project.

My intention in recording the videos was to produce a broad set of materials (the linked O'Reilly page contains a good summary of the contents). I figure that most Python programmers will receive excellent value for money even if they already know 75% of the content. Those to whom more is new will see a correspondingly greater benefit. But I was also keenly aware that many Python learners, particularly those in less-developed economies, would find the price of the videos prohibitive.

With O'Reilly's contractual approval the code that I used in the video modules, in IPython Notebooks, is going up on Github under a Creative Commons license [EDIT: The initial repository is now available and I very much look forward to hearing from readers and potential contributors - it's perfectly OK if you just want to read the notebooks, but any comments yuu have about your experiences will be read and responded to as time is available]. Some of it already contains markdown annotations among the code, other notebooks have little or no commentary. My intention is that ultimately the content will become more comprehensive than the videos, since I am using the video scripts as a starting point.

I hope that both learner programmers and experienced hands will help me turn it into a resource that groups and individuals all over the world can use to learn more about Python with no fees required. The current repository has to be brought up to date after a rapid spate of editing during the three-day recording session. It should go without saying that viewer input will be very welcome, since the most valuable opinions and information comes from those who have actually tried to use the videos to help them learn.

I hope this will also be a project that sees contributions from documentation professionals (and beginners they can help train), so I will be asking the WriteTheDocs NA team how we can lure some of those bright minds in.

Sadly it's unlikely I will be able to see their talented array of speakers as I will still be recovering from surgery. But a small party one evening or a brunch at the office might be possible. Knowing them it will likely involve sponsorship or beer. Or both. We shall see.

I think it's a worthwhile goal to have free intermediate-level Python sample code available, and I can't think of a better way for a relative beginner to get into an open source project. I also like the idea that two communities can come together over it and learn from each other. Suffice it to say, if there are enough people with a hundred bucks* in their pocket for a six-hour video training I am happy to use part of my share in the profits to support this project to some degree.

[DISCLOSURE: The author will receive a proportion of any profit from the O'Reilly Intermediate Python video series]

* This figure was plucked from the air before publication, and is still a good guideline, though as PyCon opened (Apr 11) a special deal was available on a package of both Jessica McKellar's Introduction to Python and my Intermediate Python.

A Rap @hyatt Customer Service Request

It's 2am and the wireless is down
@Hyatt ... #pycon
That's why my face is wearing a frown
Even though I'm at ... #pycon

I love all these Canadians
And Montreal is cool
But don't you know how not to run a network

If I were a rapper
Then you'd have to call me Milton
Because frankly I get much better service

I'm a businessman myself
And I know we're hard to please
So kindly please allow me
To put you at your ease

Your people are delightful
And as helpful as the best
I want to help, not diss you
I'm not angry like the rest

The food is amazing
And the bar could be geek heaven
If only you weren't calling
For last orders at eleven

We're virtual and sleepless
So we need your help to live
And most of us are more than glad
To pay for what you give

But imagine you're away from home
And want to call your Mom
The Internet's our family
So you've just dropped a bomb

I've had my ups and downs with Hyatt
Over many years
But never felt before
That it should fall on other's ears

I run conferences, for Pete's sake
And I want to spend my money
If only I could reach someone
And I'm NOT being funny

PyCon is my baby
So I cherish it somewhat
But this has harshed my mellow
And just not helped a lot

We're bunch of simple geeks
Who get together every year
We aren't demanding, I don't think
Our simple needs are clear

I don't believe that I could run
Your enterprise right here
It's difficult, and operations
Aren't my thing I fear

So please, don't take this badly
But you've really disappointed
Which is why a kindly soul like me
Has made remarks so pointed

We will help you if we can
We know you pay a lot for bits
But I have to know if web sites
Are receiving any hits

You've cut me off, I'm blind
And so I hope there's nothing funky
Happening to my servers
While I'm sat here getting skunky

Enough, I've made my point
So I must stop before I'm rude
The Internet's my meat and drink
You've left me without food.

trying-to-help-while-disappointed-ly yr's  - steve