Social Media and Immortality

I suppose everyone who uses social networks like Facebook eventually comes up against the situation when that network presents a dead friend in some context that only makes sense about someone who is alive and actively maintaining a social media presence. I just did, again.

In this particular case it was triggered by the suggestion from LinkedIn that I might like to add a fellow Learning Tree instructor to my roster of friends. He died, quite young and to most of his colleagues' surprise, about fifteen months ago (if my memory can be relied upon, which I wouldn't necessarily recommend as a strategy). I've seen similar reports on Twitter from other friends.

Now, I'm just a guy who chose to eschew the corporate career ladder and work on small systems that do demonstrable good, so I freely admit that the young devops turks of today are able to develop far more capable systems that I could have conceived of at their age. That's just the nature of technological progress. At the same time, I have to wonder why nobody appears to have asked the question "Should we take special actions (or at least avoid taking regular ones) for users who haven't logged in in over a year?"

Do they have no business analysts? Must we geeks be responsible for avoiding even the most predictable social gaffes?

Sidebar: I once designed the database for a system that monitored the repayment of student support funds by those who had accepted assistance from the federal government to train in teaching disadvantaged students. There were certain valid reasons for deferring repayment (such as military service), and of course these deferrals had to be recorded. I remember feeling very satisfied that all I had to do was associate the null value with the deferral duration for "Dead" as the referral reason to have everything work perfectly well.

The answer to my question of two paragraphs above, by the way, would be “yes”. This will be the last time I give free advice to the social media companies, so Twitter, Facebook, LinkedIn, and the rest, I hope you can find some benefit in this advice. Anything further will cost you dearly. (I should be so lucky).

Quite separately from the above speculations on human frailty, I can't help wondering what kind of immortality a continued existence on these platforms represents (even though this will probably lead to hate mail from all kinds of people the concept offends). I had an email from Google a couple of days ago asking me to log in with a particular identity* within a month or have the account go inactive. That's a necessary second step to whatever palliative actions you choose to take when presenting the account to others. Google, for all their execrable support,** get that you have to log in now and again just to assert your continued existence.

It strikes me this is a reasonably humane way to proceed. If you want to keep someone's memory alive on a social media platform then you must know them at least well enough to log in to their account, after which it's basically your shrine to them if you want it to be. I really don't like to think about what kind of complications the lawyers will dream up about this, though. Otherwise, well, we are after all all born to die (Ray Kurzweil notwithstanding).


*Note to the Google identity nazis: no, of course I was joking, I only have one identity
** Hint re Google customer service: if you aren't paying you aren't a customer, so expecting service might seem presumptuous

Practical Python (1)

Note: this blog post is the first I am undertaking with the IPython Notebook. I am still playing with formatting and so on, so please bear with me if the content doesn't seem as easy to read as it should. The notebook itself can be found as a gist file on Github and you can alternatively view it using the online Notebook viewer.

I want to discuss a typical bit of Python, taken from a program sent me by a colleague (whether it's his code or someone else's I don't know, and it hardly matters). It's the kind of stuff we all do every day in Python, and despite the Zen of Python's advice that “there should be one, and preferably only one, obvious way to do it” there are many choices one could make that can impact the structure of the code.

This started out as a way to make the code more readable (I suspect it may have been written by somebody more accustomed to a language like C), but I thought it might be interesting to look at some timings as well.

In order to be able to run the code without providing various dependencies I have taken the liberty of defining a dummy Button function and various other “mock” objects to allow the code to run (they implement just enough to avoid exceptions being raised)*. This in turn means we can use IPython's %%timeit cell magic to determine whether my “improvements” actually help the execution speed.

Note that each timed cell is preceded by a garbage collection to try as far as possible to run the samples on a level playing field**.

In [1]:

import gc

class MockFrame():
    def grid(self, row, column, sticky):
        pass
mock_frame = MockFrame()

def Button(frame, text=None, fg=None, width=None, command=None, column=None, sticky=None):
    return mock_frame

class Mock():
    pass

self = Mock()
self.buttonRed, self.buttonBlue, self.buttonGreen, self.buttonBlack, self.buttonOpen = (None, )*5

f4 = Mock()
f4.columnconfigure = lambda c, weight: None

ALL = Mock()

The code in this next cell is extracted from the original code to avoid repetition - all loop implementations are written to use the same data.

In [2]:

button = ["Red", "Blue", "Green", "Black", "Open"]  
color = ["red", "blue", "green", "black", "black"]  
commands = [self.buttonRed, self.buttonBlue, self.buttonGreen,
            self.buttonBlack, self.buttonOpen]  

So here's the original piece of code:

In [3]:

g = gc.collect()

In [4]:

%%timeit
# Benchmark 1, the original code
for c in range(5):  
    f4.columnconfigure(c, weight=1)
    Button(f4, text=button[c], fg=color[c], width=5,
               command=commands[c]).grid(row=0, column=c, sticky=ALL)

100000 loops, best of 3: 4.45 µs per loop

You might suspect, as I did, that there are better ways to perform this loop.

The most obvious is simply to create a single list to iterate over, using unpacking assignment in the for loop to assign the individual elements to local variables. This certainly renders the loop body a little more readably. We do still need the column number, so we can use the enumerate() function to provide it.

In [5]:

g = gc.collect()

In [6]:

%%timeit
for c, (btn, col, cmd) in enumerate(zip(button, color, commands)):  
    f4.columnconfigure(c, weight=1)
    Button(f4, text=btn, fg=col, width=5, command=cmd). \
               grid(row=0, column=c, sticky=ALL)
    pass

100000 loops, best of 3: 4.26 µs per loop

Unfortunately any speed advantage appears insignificant. These timings aren't very repeatable under the conditions I have run them, so really any difference is lost in the noise - what you see depends on the results when this notebook was run (and therefore also on which computer), and it would be unwise of me to make any predictions about the conditions under which you read it.

We can avoid the use of enumerate() by maintaining a loop counter, but from an esthetic point of view this is almost as bad (some would say worse) than iterating over the range of indices. In CPython it usually just comes out ahead, but at the cost of a certain amount of Pythonicity. It therefore makes the program a little less comprehensible.

In [7]:

g = gc.collect()

In [8]:

%%timeit
c = 0
for (btn, col, cmd) in zip(button, color, commands):  
    f4.columnconfigure(c, weight=1)
    Button(f4, text=btn, fg=col, width=5, command=cmd). \
               grid(row=0, column=c, sticky=ALL)
    c += 1
    pass

100000 loops, best of 3: 4.05 µs per loop

The next two cells repeat the same timings without the loop body, and this merely emphasises the speed gain of ditching the call to enumerate(). At this level of simplicity, though, it's difficult to tell how much optimization is taking place since the loop content is effectively null. I suspect PyPy would optimize this code out of existence. Who knows what CPython is actually measuring here.

In [9]:

g = gc.collect()

In [10]:

%%timeit
for c, (btn, col, cmd) in enumerate(zip(button, color, commands)):  
    pass

1000000 loops, best of 3: 1.18 µs per loop

In [11]:

g = gc.collect()

In [12]:

%%timeit
c = 0
for btn, col, cmd in zip(button, color, commands):
    pass
    c += 1

1000000 loops, best of 3: 854 ns per loop

Somewhat irritatingly, manual maintenance of an index variable appears to have a predictable slight edge over use of enumerate(), and naive programmers might therefore rush to convert all their code to this paradigm. Before they do so, though, they should consider that code's environment. In this particular example the whole piece of code is simply setup, executed once only at the start of the program execution as a GUI is being created. Optimization at this level woud not therefore be a sensible step: to optimize you should look first at the code inside the most deeply-nested and oft-executed loops.

If the timed code were to be executed billions of times inside two levels of nesting then one might, in production, consider using such an optimization if (and hopefully only if) there were a real need to extract every last ounce of speed from the hardware. In this case, since the program uses a graphical user interface and so user delays will use orders of magnitude more time than actual computing, it would be unwise to reduce the readability of the code, for which reason I prefer the enumerate()-based solution.

With many loops the body's processing time is likely to dominate in real cases, however, and that again supportus using enumerate(). If loop overhead accounts for 5% of each iteration and you reduce your loop control time by 30% you are still only reducing your total loop run time by 1.5%. So keep your program readable and Pythonically idiomatic.

Besides which, who knows, some Python dev might come along and change implementations to alter the relative time advantage, and then wouldn't you feel silly changing all that code back again?

---

* If you have a serious need for mock objects in testing, you really should look at the mock module, part of the standard library since Python 3.3. Thanks to Michael Foord for his valiant efforts. Please help him by not using mock in production.

** An interesting issue here. Originally I wrote the above code to create a new MockFrame object for each call to Button(), and I consistently saw the result of the second test as three orders of magnitude slower than the first (i.e. ms, not µs). It took me a while to understand why timeit was running so many iterations for such a long test, adding further to the elapsed time. It turned out the second test was paying the price of collecting the garbage from the first, and that without garbage collections in between runs the GC overhead would distort the timings.

Blip.tv Deletes Python Content

There's been some disturbance in the Python ecosphere because Blip.tv has removed a lot of Python content - for a long time, Next Day Video used Blip as their preferred hosting service (I don't know whether they still do or not, but after this I should hope not) and PyCon video was hosted there by default. According to their announcement

After many years of being an open platform, we’re now taking our mission to bring the best original web series to our audience more seriously.

So I wrote to them to ask if it would be possible to get copies of the content they had made unavailable:

I understand that many Python-related videos are no longer available on your service.

This is to ask whether you can make the original media available to us for re-hosting, since there is definite demand for some of the video content you have removed.

Their reply was unequivocal:

Unfortunately this content is no longer available on Blip and we are not able to provide it to you. The original content owner may re-upload their source files to another hosting provider as an alternative.

So if you were thinking about using Blip's services, you might want to think again. They have just deleted all this stuff, as far as I can see without giving much notice to the people who posted it in the first place. They have made no friends in the Python world as a result, and I can only imagine who else they have pissed off with their apparently heavy-handed actions.

Open Source and Money

David Heinemeier Hansson, a name well-known in the Ruby on Rails world, recently wrote a blog post entitled “The perils of mixing open source and money.” In it he argues that the ability of open source contributors to raise funding through sources like Kickstarter threatens open source, though I find his arguments unpersuasive.

First he suggests that fundraising represents a one-time “cashing in on goodwill earned,” whereas I suspect that if a funded project is successful that would increase the likelihood of receiving funding for future projects and wold actually increase the goodwill directed towards the fundraiser. Second he indirectly suggests that being compensated for writing software will lead to needless embellishment, whereas I should have thought that community pressures in any decent open source developer community would lead to negative code reviews and decisions not to include needless bloat.

Hansson then goes on to suggest that working for community donations causes people to work to keep the donations coming in rather than to improve the software. The fact that many Kickstarter software projects have apparently succeeded appears to make no difference to his opinion. Sadly it seems to me that in closing he reveals that the whole piece is indeed just opinion when he says

It's against this fantastic success of social norms that we should be extraordinary careful before we let market norms corrupt the ecosystem. Like a coral reef, it's more sensitive than you think, and it's how to underestimate the beauty that's unwittingly at stake. Please tread with care.

 Doesn't Hansson know that many people who work in open source do so principally because their employers pay them to do so? And yet their acceptance of the corporate shilling is apparently not in danger of perverting the course of open source development, while people with good ideas that others are prepared to fund apparently don't qualify to receive support because they put the whole ecosystem in danger.

Either I misunderstood something or what Hansson wrote doesn't make sense. The fact of the matter is that the best open source projects don't include contributions because they have been funded, they include them because they are valuable to the project. As long as these values remain in place then the injection of money into open source projects is both desirable and useful.

MacOS Migration: If I Have to Shave One More Yak ...

So I decided it was time to update my laptop, the current Macbook Air being three years old and having a tendency to forget that the screen is present. The triggering event was a minor spillage which stopped several crucial keys from working as the beer dried out (yes, I know ...).

Clearly the first thing to do was take the Air off the 'Net and back it up. Connecting the backup disk and starting a Time Machine backup saw it identify about 75,000 files of the 1,500,000 on y hard disk needed backing up. When the backup phase started it took about five minutes to get to "3k of 4.5GB backed up" and I realized I had a problem.

StackOverflow suggested I consider repairing the Time Machine disk. That's another hour I shall never see again, but at least Disk Utility gave the volume a clean bill of health, so I restarted the Time Machine backup and this time (hallelujah!) it ran. I then ran the Migration Assistant on the new (receiving) Mac and told it to restore my account, settings, applications - the whole caboodle. Since the Assistant kindly estimated this would take at least two hours and fifteen minutes I decided this would be an appropriate point at which to go for a couple of drinks with my buddy Kirby.

I found the Apple Migration Assistant to be quite friendly (disclaimer: I have always previously used manual migration procedures to switch computers and so cannot say whether Windows offers similar friendliness; if not, it should). When I returned about 90 minutes later I was delighted to find that the restore was complete (I hadn't waited around for Migration Assistant to upgrade its estimate), so I was able to log in to my newly-restored account on the updated Air.

My joy lasted as long as it took me to run a terminal session, when I saw the following delight:

Last login: Tue Nov 19 01:27:18 on tty??
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Library/Python/2.7/site-packages/virtualenvwrapper/hook_loader.py", line 16, in 
    from stevedore import ExtensionManager
  File "/Library/Python/2.7/site-packages/stevedore/__init__.py", line 3, in 
    from .extension import ExtensionManager
  File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 4, in 
    import pkg_resources
ImportError: No module named pkg_resources
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenv has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python and that PATH is
set properly.
AirHead:~ sholden$

Not exactly what you want to see when you log in, but at least indicating the the error was probably with my virtual environments. Fortunately it was quite easy to fix this daunting error message by doing a brew install python after uninstalling the previously installed version (thereby achieving an upgrade from 2.7.2 to 2.7.5). Since my account is set to prefer /usr/local/bin to /usr/bin it finds the updated python immediately. I then upgraded pip to the latest version and I appear to be good to go (though I am sure I shall find many other bumps in the road).

A very pleasant surprise was that the Mac knew about all my printers and was happy to try and use them for me. I won't know whether they work until I get back to the network to which they are connected, but if they do I will be both surprised and delighted. This is clearly superior technology where the user experience has been considered relatively carefully.

Unfortunately the user before me had already started using brew, so quite a lot of reowning was required to make my account the owner of essential locations before I could start to access the brew package again. Once I could start to think about doing that I discovered that brew update would not work because "The following untracked working tree files would be overwritten by merge". Thanks to this very helpful web page I updated by brew installation and then the update went ahead just fine. After which a brew upgrade decided that 35 packages needed updating (gulp!) and so I am currently waiting to see if the following packages upgrade without issues:

ack 2.10, atk 2.10.0, cairo 1.12.16, cmake 2.8.12.1, erlang R16B02, fontconfig 2.11.0, gdk-pixbuf 2.30.1, gettext 0.18.3.1, gfortran 4.8.2, git 1.8.4.3, glib 2.38.2, gmp 5.1.3, gtk+ 2.24.22, harfbuzz 0.9.24, icu4c 52.1, libmemcached 1.0.17, libxml2 2.9.1, mercurial 2.8, mplayer 1.1.1, ncdu 1.10, nmap 6.40, pango 1.36.1, pixman 0.32.2, postgresql 9.3.1, pv 1.4.6, pypy 2.2.0, python 2.7.6, python3 3.3.2, redis 2.6.16, sphinx 2.1.3, sqlite 3.8.1, xz 5.0.5, zeromq 3.2.4

I'll let you know how it goes! In the meantime, though my virtual environments still need some attention I am able to run Pythons 2 and 3 and the IPython Notebook. Kudos to Apple, this migration has left me much happier than I would have imagined.

What Do You Care?

I was saddened to discover from this blog post that Nadezhda Tolokonnikova is currently (and, most likely, call me paranoid if you will, deliberately) unaccounted for in the Russian prison system. I have never met this woman, but I know she has a family, and friends, who love her and need to know she will be kept safe. I also know that she tried, with her fellow band-members, to bring to the world's attention the fact that their homeland might not be ideal. For which they are all being punished.

If you won't just bring this one tiny injustice to your world's attention, what do you expect other people to do for you when you need help?

The Russian state needs to understand that unless it chooses to give up any pretense of listening to public opinion then it must get used to the fact that it too is being watched. If you can't find time to help Nadezhda, who should find time to help you?

Of course if you'd rather just write computer programs then this is just another annoying whiney post in a long series. Whose number is, by now, doubtless legion, sorry about that. But when you need to find your significant other in the TSA miscreants' repository don't come round asking me for help.

You think this could never happen? I so hope you are right.

Bradley Manning's Post-Sentencing Statement

This statement from Bradley Manning was read after hos 35-year sentence was passed by his lawyer, David Coombs.

The decisions that I made in 2010 were made out of a concern for my country and the world that we live in. Since the tragic events of 9/11, our country has been at war. We’ve been at war with an enemy that chooses not to meet us on any traditional battlefield, and due to this fact we’ve had to alter our methods of combating the risks posed to us and our way of life. 

I initially agreed with these methods and chose to volunteer to help defend my country. It was not until I was in Iraq and reading secret military reports on a daily basis that I started to question the morality of what we were doing. It was at this time I realized in our efforts to meet this risk posed to us by the enemy, we have forgotten our humanity. We consciously elected to devalue human life both in Iraq and Afghanistan. When we engaged those that we perceived were the enemy, we sometimes killed innocent civilians. Whenever we killed innocent civilians, instead of accepting responsibility for our conduct, we elected to hide behind the veil of national security and classified information in order to avoid any public accountability. 

In our zeal to kill the enemy, we internally debated the definition of torture. We held individuals at Guantanamo for years without due process. We inexplicably turned a blind eye to torture and executions by the Iraqi government. And we stomached countless other acts in the name of our war on terror. 

Patriotism is often the cry extolled when morally questionable acts are advocated by those in power. When these cries of patriotism drown our any logically based intentions [unclear], it is usually an American soldier that is ordered to carry out some ill-conceived mission. 

Our nation has had similar dark moments for the virtues of democracy—the Trail of Tears, the Dred Scott decision, McCarthyism, the Japanese-American internment camps—to name a few. I am confident that many of our actions since 9/11 will one day be viewed in a similar light. 

As the late Howard Zinn once said, “There is not a flag large enough to cover the shame of killing innocent people.” 

I understand that my actions violated the law, and I regret if my actions hurt anyone or harmed the United States. It was never my intention to hurt anyone. I only wanted to help people. When I chose to disclose classified information, I did so out of a love for my country and a sense of duty to others. 

If you deny my request for a pardon, I will serve my time knowing that sometimes you have to pay a heavy price to live in a free society. I will gladly pay that price if it means we could have country that is truly conceived in liberty and dedicated to the proposition that all women and men are created equal.

One might wish that President Obama could put up such a principled defense for his scurrilous conduct in handing over the security of the American people to secret courts and universal surveillance.

Getting Back to IT

Life has become much more interesting of late. With two or three interns in The Open Bastion office periodically I am revisiting basic software development practices to try and explain them. I am using it as an opportunity to bring my own skill-set up to date, and it's immense fun to actually get to grips with, even though quite a challenge in the face of intelligent (and sometimes daunting) questioning from the interns. We are all learning a lot, me included.

Because we want to be able to build flexible web systems we have decided to use django CMS as our base system. The editing interface may not represent the absolute zenith of design (though the 3.0 release, currently in Beta, looks rather better), but it appears to be a solid piece of technology and it is capable (when applied by a team including design skills) of producing attractive and easily-edited web content. Even with a relatively puny SQLite database and with the Django debug toolbar switched on you can see how people with little deep understanding of information technology could nevertheless use it to build their own web resources fairly quickly.

The original attraction of django-cms was its ability to maintain flexible content via its flexibly extensible plugin system. This is a valuable part of keeping content separate from presentation—and, of course, there's nothing to stop you writing plugins to use the same data as other Django code that's already part of your site. For a crowd of tech-heads the most difficult part will be the presentation and styling. Who knows, perhaps a design intern will want to get in on the ground floor.

Our first task was to get to grips with a solid but simple workflow. We have now settled on the simple but effective git branching model proposed in Vincent Driessen's excellent paper. Of course since everyone is learning git we are making all the usual mistakes and stumbling along at times. It's reassuring that whatever mistakes you do make are usually recoverable. Nowadays I sometimes even remember to checkout a new branch before committing development changes.

We are currently at the stage of creating a solid build and learning how to work with plugins. I have managed to extend the tables plugin to allow drop-down style selection (albeit with a somewhat inflexible mechanism) and gained much insight thereby. We still face the challenges of testing and deployment

Sadly now I have to go back to writing the code to create the slots for the DjangoCon schedule, but I hope to write more later.

OSCON Community Leadership Summit

I've just come back enthused from having spent my entire weekend working.

Of course anyone who works in open source will recognize the effect: someone gives you a chance to learn from and inform a couple of hundred like-minded individuals and there goes the weekend. Didn't even go to the PyLadies informal Saturday gathering, normally a fixed point of the weekend when I'm at home (a sadly somewhat too rare occurrence still).

OSCON is celebrating its fifteenth anniversary this year, and four years ago Jono Bacon of Canonical engineered the first Community Leadership Summit. I participated in this fifth run more fully than in previous years, and have benefited from much collective wisdom.

As is my practice at conferences I spent a lot of time in the hallway track. I acquired this habit from years of organizing PyCon, when I would frequently be waylaid on my way to some session and end up having an amazingly interesting and informative discussion, meeting new people, broadening my knowledge of Python and the open source world generally. If I could hire all the people I know in the open source world (which I couldn't, even with unlimited capital, since many of them are pursuing their own dreams) I would be able to build an unbeatable distributed information systems engineering and operations team.

A couple of interesting facts appeared as the weekend went by. First of all, perhaps fifteen to twenty-five percent of those attending had no direct connection with the open source development world. Some of these people were enthusiastic users of open source projects, and others were community managers who have started to realize that the way the open source world uses a diverse distributed open toolkit to achieve its goals could be useful to them too.

Secondly, and perhaps largely as a result of the first, the open source crowd began to realize that as the community continues to become more diverse (yay, diversity!) we have to stop talking in jargon about things like "cloning git repositories" if we don't want these brave spirits who are here to learn about the social and organizational side of open source to be completely turned off by our inability to speak plain English*.

We might also need to consult with UX teams to ensure that our current technologies can be used as the basis of a layer that makes sense to people who don't have our intimate knowledge of the technologies but are nevertheless keen to learn and use our methods. That could represent a substantial challenge.

The lightning talks were uniformly inspiring. Jono was kind enough to say he really enjoyed mine, which was called Making Small Positive Differences. The tradition at the CLS is to set the talk times to accommodate everyone who signed up. I didn't know this, so like many of the other speakers I entered the room expecting a five-minute slot.

Fortunately I was fifth on the list, so I had time to go through my slides (did I mention there was no projector? another surprise, but what the hey, this is mostly an unconference) and extract the salient points into notes for a three-minute talk in my extremely convenient (and free) O'Reilly notebook.

Some of the gems of the lightnings for me included:

• Kevin Johnson's appeal to share our technology to the benefit of all
• Britta (?)'s suggestion that barriers to entry weren't always a bad thing, with examples of how they could work in a community's favor (without discouraging diversity?)
• Josh Hibbet's talk about CityCamp, which could become a nationwide (worldwide?) series of events to promote citizen participation and government transparency
• Gribble, an IRC bot built by SourceForge to be friendly and welcoming to new IRC community members
• Selena Deckleman's invitation to go and speak in a K-12 classroom
• Aaron Wolf's promotion of the Snowdrift Co-op
• A talk by someone [whose name I need to research so I can introduce him to a colleague with a keen interest in such matters] who has developed a way (if I understood correctly) to codify the law and represent legal entities engaging in contractually-bound mutual obligations.

I also attended sessions on how to attract a more diverse skill set into the open source world (something I have been banging on about for a while now in keynotes up and down), and discussions about how, when you are building teams with people who have never worked together before, it's difficult to manage projects because things can get blocked when person A doesn't even know that person B cannot proceed until A's part of the task is complete.

This last discussion raised many interesting aspects of how people working on project teams communicate, and we realized that training was one important way that people with diverse skills can learn to work together effectively and harmoniously.

All in all this was an excellent start to OSCON, and I feel more in touch with the hot-button issues of the collected open source communities as the convention proper begins (tutorials are the main activities today and tomorrow). I'm looking forward to a great week, and hope to round it off with a reprise of last year's OSCON Survivors' Breakfast. Drop me a line if you'd like an invite.

* No holier than thou here. The best I can do at short notice is:

If you just need to use project X, go to [some web page] and click the "Install" button. Then follow the instructions on that page to make sure the installation has succeeded, and the follow the [tutorial web page] or browse [web search for currently best-rated and/or most popular blog posts about Projct X]

If you are a project X user who finds it unsatisfactory in some way we would love to hear from you. Only by being keenly attentive to feedback from our dedicated users can we ensure that project X continues to become more useful to a wider range of people every time.

If you would like project X to do things it currently doesn't please let us have your ideas on [ideas@projectX.hostingsite.example.com]. We will always attempt to acknowledge input and respond to feedback.

If you would like to help project X become more capable then please ask us how you can support the open source communities who built and help to maintain it. We are always happy to welcome new project members, who can provide the one thing without which open source could not be possible: the willingness to build software to raise the tide and level the playing field [see what I just did there]. 

By creating new infrastructure open to all we hope to improve humanity's lot more effectively.  Project X is available under the [link to OSI-approved license] and its documentation is available under the [some sort of Creative Commons license] on Read The Docs.

 

A Statement from Adria Richards

Having remained silent since PyCon, Adria Richards has now published a statement reflecting on those events, though sensibly without going into specifics. I think it's important that her retrospective view gets some air, so I am reproducing it here with my own comments.

Those who know me well in the the developer and tech community recognize that I have always tried to conduct myself in a way that builds bridges for everyone. My central aim is to do everything I can to help create new, inclusive inroads for all, no matter who they are, where they come from or what they believe. Development is about innovation, creativity, and in a grand sense, the betterment of human society through technology. So, it stands to reason that everyone should have a seat at the table, and everyone involved in this vital community should feel welcome, safe and respected. In essence, the worldwide community of developers can and should function as a reflection of what our wider society strives to be.

I don't know Adria, so I take her opening statement at face value. Those who disagree may do so elsewhere. Garann Means recently made a fairly good case that the denizens of the technology world don't act in a way that inspires respect from other communities. I think it's fair to say (and some of the comments in forums like Hacker News amply demonstrate) that there is plenty of room for improvement in what's considered acceptable conduct in the technology world.

I cannot comment at this time on the specifics of what occurred at PyCon on March 17, and the subsequent events of the following days, but I can offer some general thoughts. I don’t think anyone who was part of what happened at PyCon that day could possibly have imagined how this issue would have exploded into the public consciousness the way it has. I certainly did not, and now that the severest of consequences have manifested, all I wish to do is find the good in what has been one of the most challenging weeks of my life.

This positive attitude is to be admired. I have been through similarly shitty situations in my own life (as most of us have at one time or another) and while it's good to move on I like to extract lessons that will help me try to avoid making the same mistakes. If I can find those I can feel more positive about the experience.

And I do believe there is good to be found in this situation. Debate and recrimination can and must give way to dialog that explores the root causes of these issues in the tech industry. As developers and members of the startup community, we can welcome newcomers, women and people of color who, as of now, are under-represented in our ranks. And, all of us can learn a great deal from those who are well-established in the field. We can solidify the values of our workplaces (yes, conference spaces are workplaces!), and set new, positive and inclusive examples for other professional disciplines.

This is a good idea. PyCon was started to try and provide an inclusive space where everyone with interests in Python can come together for the betterment of all. I would be suspicious of those who don't support the above as an ideal. We surely have to accept that there's still along way to go to get there. So dialog is a good idea.

What happened at PyCon has cast a spotlight on a range of deep issues and problems in the developer world. As ugly as this situation has become, all of these issues have reasonable, and, I think, easily reached solutions that will help us cast conflict aside and construct a more cohesive and welcoming professional environment based on respect, trust and open communication. I do not, at this time, wish to concentrate on the fallout of the last several days. Instead, I want to be an integral part of a diverse, core group of individuals that comes together in a spirit of healing and openness to devise answers to the many questions that have arisen in the last week. Together, we can work to make the tech world a better place to work for everyone, and in doing so, we make the wider world a better place for all.

It's interesting that this episode happened at PyCon. When we started to talk seriously about Codes of Conduct in the Python world there was a lot of kick-back, and people said things like "but this is the Python community" (implying that we are such nice people that we won't have the same issues) and "but that's just normal behavior" (not even implying, but specifically stating, that anyone who needed a Code of Conduct to condition their behavior shouldn't be going to conferences anyway).

We pressed ahead none the less, refining the code to improve its expression of our ideals of openness and inclusivity. When issues had to be addressed (and you can read the reports in the PyCon blog if you need to refresh your memory) the code of conduct was applied, and to my mind it worked. The fact that two people lost their jobs as a result of the fall-out was nothing to do with the code of conduct, and yet there were many who felt that the blame should be laid at PyCon's door. I shudder to think what the result would have been at an event with no explicit code.

I hope that Adria can come back to PyCon without being subjected to any hostility. She expresses a great ideal, one that we should all be striving for. Now we just have to work out how to achieve it. But that's an implementation detail, right? I wish!

So Long, and Thanks for All the Fish

A notable transition point in my life, by any measure. Interesting things are happening in all directions, many of them (I am delighted to day) heavily involving Python. So this is definitely not goodbye. My time on the Foundation's board has been a wonderful and interesting ride. I hope I have lived up to Tim O'Reilly's ethic of creating more value than I extract.

Why Tuples?

A frequent question from new and even not-so-new Python programmers is "why does the language have both tuples (which, if you know Python, you will recall are immutable) and lists?" You might almost say there are two kinds of Python programmers, those who know what tuples are for and those whose mathematical education has been limited. I know this sounds like an awfully snobbish thing to say, but it isn't meant that way. The fact is that I learned most of my programming in eight years of working experience before I started my degree studies, and I am therefore of the very definite opinion that people with little mathematical background can be excellent programmers.

It's just that I myself only came across the word tuple when I started studying college-level mathematics and had to come to terms with things like

An NFA is represented formally by a 5-tuple, (Q, Σ, Δ, q₀, F), consisting of

• a finite set of states Q
• a finite set of input symbols Σ
• a transition relation Δ : Q × Σ → P(Q).
• an initial (or start) state q₀ ∈ Q
• a set of states F distinguished as accepting (or final) states F ⊆ Q.

Now this has a very definite meaning. It tells us that an (ordered - i.e. positionally-identified) set of five things is sufficient to define the behavior of a specific type of object (in this case a non-deterministic finite state automaton (NFA), though for all you need to know about those I might just as well be describing a flux capacitor).

If we wanted to encapsulate an NFA as a Python data structure, then we might at some point in our code write in our Python code something like

    nfa = (states, symbols, transition, initial, accepting_states)

though in actual fact you would be more likely to want to incorporate behavior in a class and so write instead

    nfa = NFA(states, symbols, transition, initial, accepting_states)

Now even though you may not know what an NFA is, you will surely perceive that the set of its possible states is a very different thing from the function that determines how a current state and a set of inputs are mapped into new states. So there is simply no way that it would be meaningful to write

    for thing in (states, symbols, transition, initial, accepting_states):
        do_something_with(thing)

unless you were, for example, trying to save its state during a pickling operation or some more obscure book-keeping metacode.

And that, best beloved, is what tuples are for: they are ordered collections of objects, and each of the objects has, according to its position, a specific meaning (sometimes referred to as its semantics). If no behaviors are required then a tuple is "about the simplest thing that could work."

This is why you often hear people informally say tuples are for "collections of things you don't need to iterate over" or "tuples are for sequences of dissimilar objects". My advice would be to stay away from such discussions. One might reasonably argue that including them in the language discouraged people from using objects with named attributes, which are always easier to deal with.

The problem with the tuple is that once we have constructed our NFA the only way to refer to the states in code is with the rather unedifying expression

    nfa[0]

which doesn't actually tell the reader much about the programmer's intention. The sequence nature of the tuple means that our accesses to the elements are difficult to imbue with meaning in our code. This latterly became such an obvious shortcoming that it prompted Raymond Hettinger to create the namedtuple object that allows you to easily specify a tuple whose elements can also be referred to by name.

It would be interesting to see whether users of namedtuple objects actually use them as tuples at all. I would guess that the sequence behaviors are rarely used, in which case perhaps it's time to either remove namedtuple's __getitem__() and similar methods or implement a similar object without the sequence behaviors.

Gun Nuts 1

In the wake of the most recent school slaying I thought it might be relevant to revisit the writings of Eric S Raymond. Within my world he is a well-known and self-described "gun nut." His two principal writings on he right to bear arms appear to be The Parable of the Sheep and Ethics Through the Barrel of a Gun. I will look at the latter in a separate post. There is nothing I can do for the misery of the bereaved families. I can only hope to help in the longer run.

The Parable of the Sheep

This is a tale of a flock of sheep. The dog cannot keep the wolves away. Some sheep take the claws and fangs from dead wolves. They fight back, reducing the carnage. Eventually the wolves mostly leave sheep alone. They do not know which sheep are armed. Many sheep are terrified of the weapons. They ban the armed sheep from to the pasture. They post signs at the edges forbidding hidden weapons. The wolves, seeing this, return. Once again they inflict carnage on the flock. They are only beaten off by the armed sheep. The flock remain afraid of the weapons. They barely tolerate their protectors, the armed sheep. The armed sheep altruistically continue to protect the flock. Raymond describes the situation thus:

The bold sheep knew that the fangs and claws they possessed had not changed them. They still grazed like other sheep, and raised their lambs in the spring, and greeted their friend the dog as he walked among them. But they could not quell the terror of the flock, which rose in them like some ancient dark smoky spirit and could not be damped by reason, nor dispelled by the light of day.

The parable omits many refinements of the true situation. That is forgivable for expository purposes. A school shooting represents something outside the parable. There is a genuine reason for the terror of the unarmed sheep. Occasionally the armed sheep start acting like wolves. Of course the unarmed sheep are then the inevitable target.

The statistics for murder and armed robbery are quite alarming. Those for child murder are truly chilling. Almost 63% involved a firearm in 2011. Well over a thousand kids blown away each year. To suggest that we all arm to protect ourselves seems unbalanced. It's like the 1950's "defense" policy of Mutually Assured Destruction. Never was an acronym more appropriate. The terrorists aren't over there, they are among us. A school massacre is an act of terror.

Here the Japanese strategy of many approaches might be fruitful. It is unlikely there is a single switch we can turn to stop this happening. Surely it is better to try to understand these acts of atrocity. Consider them a societal ill and seek its cure. Perhaps we could try to build a world where people are better in touch with their fellows. Extremes of behavior can (most of the time) be recognized. If someone is heading in a dangerous direction, perhaps it can be deflected by sympathetic attention.

I imagine* that most mass murderers are completely in control most of the time. They are indistinguishable  from "normal" people. The classic quote after an apparently psychotic shooting episode? "He seemed like a normal guy." Let's leave the fact it's usually a man to another time. It's not like they are landmines, liable to go off at any touch. I am a pretty placid person most of the time. Occasionally some perfect storm of circumstances hits. Shortage of sleep is often involved. Physical exhaustion never helps. Emotional distress is very rarely positive. I "go off on one" in an uncharacteristic way. We all have extremes of behavior, often in response to extreme circumstances. Some people are more extreme than others.

Some people, sadly, are so extreme that they must be incarcerated to protect the rest of us. I would, in the USA, prefer it if the government invested its funding in long-term plans to make the country a better place to live than lock up predominantly black predominantly drug-offending adults.** Now private companies are running prisons, vested interests are powerful.

So while Raymond's simplified argument seems reasonable enough, that's only because it is simplified. There are inevitable outliers in a flock of over three hundred million. Mental stability and degree of moral control are two important dimensions. At that scale the sheep would appear to have a good reason for caution. Anyone proposing to use fangs and claws should surely be assessed as a responsible individual. Mistakes will still be made, but (presumably) it is not beyond the wit of sheep to concoct a scheme that reduces the number of massacre incidents.

In network parlance, the parable doesn't scale to larger flocks. Neither does it scale to bigger and better teeth and claws. The reductio ad absurdum of many gun nut arguments is that we should all carry personal tactical nuclear weapons. What's reasonable, and what's not? Why are such deadly automatic weapons allowed outside gun clubs and the armed forces? I don't mind people enjoying handling and using firearms. Hunting, preferably for food, for example, is a perfectly legitimate use. So is fun at a gun club shooting range. Mowing down kids, not so much. We have to strike a balance.

One final note: please let's not imagine there is an instant cure. All we can do is lay, and maintain to, long term plans. These should aim to reduce the availability and access to weapons, particularly those capable of facilitating mass murder.

* Admittedly if this were a Wikipedia article someone would have at least added a "citation needed" flag
** Who are then unable to vote

A Thank You to Eric Sterling

Offered for the attention of those who attended Eric's recent keynote at DjangoCon and, most especially for Eric himself. I still remember the talk and the panel session he took the time to contribute to fondly, and I particularly enjoyed the unique flavor he added to the lunch he attended.

I'm Sorry

The one-eyed snake

It appears I've been that guy (again). I've seen those "sorry if you were offended" apologies, and they are bullshit. So let me begin by unreservedly apologizing for my inappropriate behavior. Then perhaps I can tell you what this is all about. I do so in the doleful expectation that this post will generate more heat than light, as is typical for this subject matter in geek circles, and will be the cause of a large amount of ill-informed, unhelpful and possibly personally damaging commentary. Since it gives me the opportunity to discuss a few issues of importance I accept that as the price.

Last This year, at PyCon, I had a Square, a device to accept credit card payments through my cellphone, and inter alia was soliciting donations to the Python Software Foundation. As a part of my schtick I was offering to take photographs, and offered the option of a prop, a little Beanie Babies™ bean-stuffed Python which my dog, when a  puppy, had once got hold of and played with so enthusiastically that she tore off (and probably swallowed) the glued-on bead that represented its left eye. I've had this snake almost ever since I started using Python (almost twenty years now) and I'm very fond of it. It often travels with me as a talisman, since I usually travel alone.

Because I am so familiar with it, in retrospect it seems a little odd that I had never particularly dwelt on the fact that it is, literally, my "one-eyed snake" though I had used the phrase at home with family and friends as a joke. [For readers not familiar with colloquial English and American, I am obliged to point out at this stage that "one-eyed snake" is one of many euphemisms for the penis]. I was amused enough, on that day, by this fatuous coincidence that I offered to include "the chairman's one-eyed snake" in the photographs I was taking, of men as well as women.

People appeared (in my opinion, not actually the opinion that matters as it happens) to tolerate my risqué attempt at humor, feeble as it was, and took it in good part although certainly not everyone asked for the snake to be included. The conference ran its course, I went on my way, unaware that a storm was brewing.

The storm's gestation in fact took another event to trigger it, a recent motion by the PSF board that we would require any conference we funded to implement a code of conduct. The board collectively felt that this would be a forward step, and would actually be helpful to conference organizers who had not yet addressed this issue. PyCon has has a code of conduct for several years, and since the conferences I run professionally also have codes of conduct, I had no hesitation in voting for the motion.

It's difficult to describe this situation in anonymous terms, since verbatim quotes are required in order to allow me to properly discuss the way this situation has turned out. But those who wish to know the identities of the proponents must do so by using {{search_engine_of_choice}}. What happened when news of the motion came out is that someone posted a comment on Hacker News which included the following content

From what I understand, the code was approved by the Board of Directors of the PSF, and not the PSF as a whole. Please, correct me if I am wrong. This is ironic since one of the Board members was walking around the conference last year with a damaged stuffed python toy asking, "Would you like to see my one eyed snake?"

This was said to one of my female colleagues. I asked her if she would like me to say something and she replied, "No, it is just creepy, but I'm an adult."

I'd like you to particularly note two things: first, the quote is inaccurate, I believe (if I uttered those words I would have to accept I would have gone over the line with anyone but a close friend); second, the colleague herself chose not to complain. This is significant because of what happened next. Clearly the comment had received some attention, since shortly thereafter a second person, one I think it is fair to say is known not to be well-disposed towards the PSF, tweeted (in reply to a third party who may for all I know have been a fifth or twelfth party)

Wait, I didn't read this comment closely enough. What fucktard was walking around asking women to pet his one-eyed snake?

Note again: the already inaccurate quotation has been through the mangler once more. So now we (that is me and the PSF, though in this post I must put my director's hat aside and accept personal culpability) have a situation. This is not something that would have worried me hugely in my personal capacity, but the fact is that it has brought a lot of pressure to bear on someone who, through his heroic efforts to improve on last year's incredible PyCon, is already under quite enough pressure. So it is necessary for me to write this explanation lest his efforts to bring in sponsorship and promote diversity of all kinds in the Python community are damaged by my bad behavior.

Now to the several points I should like to make about the outcomes so far. I hesitate to think of the fallout to my professional life.

1. I am lucky to be a man, since any woman posting about similar issues of female sensitivity is likely to be harangued and harassed until, unless she is of extraordinary character, she either allows herself to be silenced or leaves the Internet altogether (yes, well-known women have received threats that they felt obliged them to do this).
2. Context is important. Although my behavior may not have been appropriate, it was taking place in broad daylight in a well-populated area so nobody felt especially threatened by it. The same offer (let alone either of the two misquoted ones above) made to a woman who is the only other occupant of an elevator at 1:30 in the morning would, I think, be reasonably interpreted as a possible if not an actual threat. Anyone who doesn't understand that isn't really fit to be out alone.
3. It isn't up to me to decide whether my behavior is appropriate, but those with whom I interact. I teach professionally, have taken (and passed) sensitivity training classes, and am accustomed to behave in a professional manner that does not run the risk of attaching stigma to those I represent.  Because I feel that many members of the Python community are my friends, I perhaps overstepped the boundaries of professionalism, believing that nobody would mind.
4. The fall-out of righteous indignation about events like this often lands on the wrong person. There is now heavy pressure on Jesse Noller as PyCon chair for not taking action about this. There have been definite suggestions that Jesse is being hypocritical in promoting a code of conduct for PyCon while condoning inappropriate behavior by a fellow director. There are two things worthy of note about this. 
5. Firstly, since the issue only came to his and my attention eight months after the conference (and, indeed, after both he and I voted for the motion requiring codes of conduct) it is difficult to know what he could have been expected to do at the time. He has been heroically silent (for him).
6. Secondly, the code of conduct exists precisely to reassure people that harassing actions can be reported, and will be dealt with, no matter what the position of the offender in the community. So I think anyone who is gunning for Jesse about this should give him a pass. If he had come to me because he had received a complaint I cannot say what action I would have felt correct, but at the least I would have expected to make a public apology. I do not expect that Jesse would have felt able to let a transgression on my part go without action: he has far too much integrity for that. I like to think I have enough integrity to accept being told when I have transgressed (hence my dislike of the "if anyone was offended, sorry" stuff).
7. Circumstances alter cases. The code of conduct is not expected to protect people in a vacuum. If you find someone's behavior offensive your first recourse should be to say so. Hopefully an apology should be forthcoming, or at least a civilized discussion about your differences in standards. If the issue cannot be resolved in person then is the time to invoke the code of conduct. This is what makes variations in behavioral standards acceptable: when you are with people you know you are unlikely to go wrong; when you are not you should take others' input as defining acceptable limits.
8. It is unreasonable to expect perfection. We all slip from time to time, and a slip should not necessarily lead to a fall. I noted with a parenthetical "again" in the opening paragraph that this is not the first time I have behaved inappropriately, and sadly it may not be the last. I can say, though, that on the few occasions my behavior towards an individual was definitely capable of causing offense I have apologized sincerely and without reservation, usually before further contact from the other party (i.e. I am capable of recognizing bad behavior on my own part and voluntarily apologizing for it). I am happy to say that I count some of those to whom I have had to apologize as friends even now, and believe those feelings are reciprocated.
9. I specifically do not want the support of men weighing in with some equivalent of "what the hell, man, you didn't do anything wrong, what are the women complaining about?" Perhaps when your knuckles stop dragging along the ground we can talk. I very much would like the support of men who are equally liable to do stupid things (and have done) and yet still remain open to the possibility that one can be equally civil and courteous to members of all sexes. It's easy for men to say "we've all been that guy," but I hope that women don't imagine that guy conversation is laced with tales about how we were "that guy". We don't talk about our goofs even among ourselves (unless there's some club I'm not a member of). Men who are sensitive to the need for gender diversity find these mistakes painful to remember, and only dwell on them in private (this has not been an easy post to write). Men who are not sensitive to that need, please re-read the opening sentence of this paragraph. Support from others will also be more than welcome. 
10. Diversity is not just gender diversity. Because I speak fluent English it is easy for Americans to imagine that I have absorbed their culture fully, though even after fifteen years I often have to stop to clarify cultural referents common to most natives. They forget that I am, in fact, an immigrant who was brought up with different standards. I am not trying to suggest that I should not be subject to the values of the USA, but would hope that when I overstep the bounds some leniency might be shown. I in my turn sometimes forget that standards are different over here.

There is more that I could write*, but I feel that this post is already long enough. I will close by remarking that it turns out the female colleague of the original poster is personally known to me, and in our several meetings both personal and professional since then she has not mentioned this issue (unless my memory has failed me). Naturally I intend to apologize to her at the first available opportunity, and to ensure that in the future I do nothing to make her feel uncomfortable (the basis on which I believed I was conducting the relationship). If I was not confident that people would feel comfortable in my presence I would not feel able to attend PyCon, since that would exclude others.

As PyCon's founder I genuinely want to see it address diversity issues properly. For eight years as a director and three years as chairman I have worked to ensure that the PSF is seen to take these issues seriously. I am mortified that I ran the risk of bringing either into disrepute. I hope that by making this apology I continue to lead by example.

* I might, say, discuss the appropriateness of using the word "fucktard" in public discourse on such serious matters .

Python for Data Analysis

Python for Data Analysis; Wes McKinney. O'Reilly Media, October 2012
[Review copy kindly provided gratis by O'Reilly Media]

This book is a welcome addition to the Python canon, and anyone who is at all concerned with data analysis would do well to read it. From the opening chapter it is a clear and concise explanation of how to deploy Python for such tasks. Early on the author makes the point that Python should be thought of primarily as a “glue” language, so the book makes no attempt to show you how to program analytical methods in Python but rather demonstrates through practical examples how to use the existing extremely effective tool chain already available.

After a few brief motivating examples you will read one of the clearest expositions of the benefits of iPython I have seen. Since reading this chapter I have become convinced of the benefits of iPython Notebook and have opened an account with NotebookCloud*, which I feel will be a real boon in my teaching work.

The next chapter covers Numpy basics, followed by an introduction to the pandas library. You are then taken in detail, and with copious practical examples, through data cleanup and various important transformations, plotting and visualization, data aggregation and grouping, time series and finally financial applications.

The final chapter on advanced Numpy provides the kind of look under the hood that will help those interested to extract maximum efficiency from the package and understanding how best to take advantage of more advanced features. The closing Appendix, “Python Language Essentials,” is a concise introduction to the language that should provide even inexperienced Python users with sufficient information to understand the examples.

The writing is clear throughout, with numerous examples that enliven and illuminate the discussions. For those whose mantra is "show me the code," the code is here in spades. Flipping through the book you find very few pages that do not contain Python code (and many of them are displaying the output from Python code).

Wes McKinney took the sensible decision to focus on Python 2.7, which is what is currently deployed by the vast majority of the scientific and analytical communities. The quality of the writing and the recent welcome news that matplotlib** is now ported to Python 3 will, I am sure, make a Python 3 update a popular choice in coming years.

If you want to understand how you can use Python as an analytical tool then I heartily recommend this book.

* An Amazon Web Services account is required to use this service, and will be charged for your computations
** This means the standard "Scientific tool chain" is now available for Python 3

Building New Communities

[This post is based on a keynote talk to the PyCon DE conference on November 1, 2012]

[UPDATE: Nov 17, 2012: The video of this talk is at http://pyvideo.org/video/1479/building-new-communities]

Good afternoon, and thank you for inviting me. It’s always intimidating to talk to an audience which is so technically capable. Today, though, I am hoping that we can look at a non-technical topic that has implications for everyone who is interested in Python: how do we promote the Python language and improve the development process?

DO EXISTING COMMUNITIES WORK?

Let me begin by saying that I have invested huge amounts of time in the Python community, and I believe that it has a well-deserved reputation as one of the more welcoming open source communities. So today I am talking more broadly than just Python. What I say applies, I believe, to many open source communities.

This talk is being given at an unprecedented time in human history. Clearly these things are a matter of interpretation, but nowadays it’s hard to find someone that doesn’t agree that the modern world is failing to meet the needs of many of its inhabitants.

Historically, we are living in strange times, The world human population reached 7 billion, depending on which figures you choose to believe, between October 2011 and March 2012, only twelve years after it reached 6 billion. According to one UN estimate it’s possible that population will reach ten billion before reaching any kind of stability, and that population in Africa could triple in this century. However you interpret these figures it is obvious that most governments are going to be struggling with population growth in a world where they are already finding it difficult to manage things to everyone’s satisfaction.

So I have a lot of questions, and not many answers. Being a geek, my approach attempts rationality—we have to take into account what we know about human behavior, differences between the various national and political cultures in which people live, and so on. In an increasingly crowded world it is easy to feel threatened by governmental and economic systems that are posited on continuous growth.

When I was young an organization called the Club of Rome produced a report called “The Limits to Growth,” positing that the finite nature of many resources could, if growth were not checked, lead to cataclysmic failures in supply of the very things that keep us alive: clean air, clean water, and nutritious food. This is where climate change deniers got their start, and yet from recent work it appears that the report was, within its limitations, a very accurate projection of certain futures.

My own vision of the future tends, I must admit, towards the catstrophic. I think it is entirely possible that the human race will wipe itself out simply by continuing to foul its own nest and ignoring the toxic environment it is creating. So I see it as important to try and bring this to people’s attention, and to do what I can to ensure that we (the human race) build a sustainable future for ourselves and the other species we have to share this huge ball of mud we call Earth with.

HEY, ISN'T THIS PYCON?

All of this might seem a very long way from our little open source world. I can already hear people thinking that I am abusing my position as a keynote speaker to stand on a political soap box and peddle my socialistic ideas. I admit I am a socialist, and those who know me also know that I am happy to stand on a soap box at times. But this is not one of those times.

Context, however, is everything. So I want to place my remarks today in the context of a world that isn’t working, that isn’t equitable, and in which ever larger divisions are opening up: between the poor and the rich in almost any country you like to name, between poor countries and rich countries, and so on. Huge amounts of resource are being wasted on machinery to wipe each other out. This is particularly noticeable in my current home country, the USA, which has 11% of the world’s population and yet is responsible for over 40% of all military expenditures, more than the next 14 top-spending countries put together. One might wonder what they are afraid of, but these expenditures certainly give them clout in the geopolitical sphere.

Open source communities, the Python community included, have growth problems of their own. According to Armit Deshpande and Dirk Riehle the number of lines of open source code is growing exponentially. This is a management problem of some magnitude, which is not being addressed at the moment. They suggest that open source applications generated revenues of $1.8 billion in 2006, and certainly growth has continued unimpeded since. Of course that $1.8bn represented only about three-quarters of one percent of total software revenues, so there is still a long way to go.

Tim O'Reilly made a sensible case this year that open source is responsible for increasing small business revenues of a single hosting company’s user base by $180 billion  And, of course,  many proprietary software products nowadays contain significant open source components.

GET PAID TO BUILD STUFF?

I have said many times that the emergence of open source represents a new economic phenomenon. It’s as though, when railways were needed in the United States, a bunch of people had said “You know, that’s a great idea. Let’s go build a railway system,” taken up their picks and shovels and headed off to work. Of course you can’t build a railway without access to heavy engineering, which is arguably difficult to open source. Similarly, most individuals would not find it cost-effective to make their own computers from scratch, or to interconnect them in the way that the infrastructure of the Internet does.

Just the same, the emergence of Linux as a powerful force in the software world, and the somewhat anarchic, less-than formal management of the whole Internet (despite the belief of many US politicians that the Internet should be managed by the US as a divine right) should convince us that it’s possible for individual and group initiatives to take hold and prosper in today’s rather strained commercial environments.

I understand that open source is not the exclusive domain of unpaid experts. Nowadays a lot of companies are paying programmers and other staff to assist with the creation of open source software projects. Many different models are used to try and recoup the investments made, but the fact remains: it is now commonly accepted that there is value to creating common infrastructure whose presence benefits a large proportion of the community rather than a single individual or organization. Of course such ideas often seem, particularly in the USA, socialistic and therefore suspect, but it is still true that large companies like HP, IBM and Microsoft are enthusiastically involved in myriad open source projects. They aren't generally thought of as having socialist tendencies.

HOW IS SOFTWARE BUILT?

http://www.multunus.com/2011/02/software-development-as-a-balance

Here’s a fairly typical example of the “software development lifecycle.” It lists a large number of activities and requirements, which deserve to be looked at in detail.

Whether all these activities are strictly necessary or not could be a pleasant discussion over drinks, but I get the definite impression that many open source projects aren’t engaged in any kind of formal process. This freedom, of course, is one of the things that attracts many people into the open source world, and I would be the last person to suggest that unnecessary work should be involved in software production. Nevertheless, open source projects are predominantly run vby programmers, and often all project members are programmers.

WHO BUILDS SOFTWARE?

http://www.ambysoft.com/essays/agileRoles.html

This is a typical conception of the way software is developed in the agile development world. There are many different ways to think about the development process. Arguably, none of them is specifically right or wrong. But when you think about commercial projects the size of Python, there will be many more people involved than just programmers. So Python, like most other open source projects, has to press developers into service to fill the other roles, or leave them unfilled.

WE need to think about whether this works to the ultimate advantage of our projects, or whether it is really a rather inefficient way to produce software.

SOME DEVELOPER ROLES

Above are just some of the roles that a large-ish software development requires. I would argue that lots of the time projects focus largely or even solely on the programming task, with some design work. Many of the other roles do not offer attractive work for developers, who therefore ignore the need to fill them, often to the project’s great detriment.

A prime example here is the Python documentation. Good as it is in comparison with many other open source projects (and it is good, because there has been a lot of hard work put in on it), it still sucks. Its organization is mostly based on a fifteen-year-old structure, back when Python was a smaller and somewhat different language. I still have no idea why, when I want documentation on the Python standard types (int, list, dict, and so on) I have to look in the Standard Library documentation. They are available without any need for an import statement, so why are they documented with the library?

Of course anyone who has used Python seriously for more than a year knows these things, and can usually find out where to look for the information they need, but this is entirely the wrong emphasis. The first need for documentation is when you are learning a language, and then the information you need should be readily available. Who is the advocate for a rewrite or reorganization of the documentation? Nobody in the current development team relishes such a task, because we don’t have technical authors and editors as a part of our community. And yet this issue causes learners way more pain than it should, and even turns people away from the language.

This brings me to my central thesis: if we don’t broaden our community to include such people we are unlikely to achieve the world domination we all jokingly feel that Python could achieve. I know from my work over the last eighteen years, and from visiting Python gatherings like this one all over the world, that Python is poised for massive growth. It would be possible, if a concerted effort was made, to make Python the first programming language of choice for literate students. The world has woken up to the fact that computational thinking is an important skill in today’s complex societies, and some feel that Python is its ideal expression.

So, do we want to slam the doors, say “No, we like things the way they are, we aren’t going to change”? I suspect that some developers will feel that way. But as a PSF director I have to be mindful of our duty to promote the development of the language and grow the international community, and so I have to force myself to look for deficiencies than can be rectified. This means looking outwards, and trying to include people who can fill the necessary roles.

The alternative, of course, is either to ignore the need for a role to be filled or to have someone fill it whose time would be better spent writing code. Neither is an entirely satisfactory solution.

BEING MORE INCLUSIVE

Sarah Novotny, one of the OSCON conference chairs, wrote of her experience at the 2011 conference: "We must challenge the idea that if someone really wants to use a piece of software, he or she will be willing to slog through half-written documentation, the actual code base and an unkind user interface." 

This is an approach that I have tried to promote to Python developers, with rather limited success. I can think of several reasons why this should be so. Here are some classic signs that the status quo is being defended:

• Considering the cost of switching before you consider the benefits
• Highlighting the pain to a few instead of the benefits for the many
• Focusing on short-term costs instead of long-term benefits, because the short-term is more vivid for you
• Embracing sunk costs

We need to try and broaden our development community to include people with the necessary skills, even though we aren't necessarily very comfortable getting close to them. This makes sense to me as a historical perspective: back in "the old days", using an open source project was highly likely to involve you as a contributor as well. Nowadays there are many other roles for contributors to fill: bug reporters, technical writers, bloggers, testers, and I am sure all you core Python developers have your own ideas for roles that could reduce your workload.

David Eaves of Mozilla corporation has done a lot of work on how software communities can use metrics to improve their development process and monitor community involvement. He asks relevant questions like “why are bugs in this section of the code taking twice as long to be reviewed?” and “who has contributed consistently over the last 18 months, but not in the last 30 days?”

If we are prepared to answer questions like that we can take the information, both qualitative and quantitative, and then use it to continue to improve our communities.

INVOLVING THE COMMUNITY

We must accept that not every member of the community wants to be involved outside a fairly narrow specialism. This is fine with me: I would much rather have contented community members contributing their skills than seeing members pressed to leave their comfort zone.

At the same time, there is considerable evidence that an inability to contribute is a source of frustration to would-be active members. So if we want to grow our community we need to be welcoming and, dare I say, open. We need to make sure that the barriers to entry into our community are as low as possible, and that people whose ideas may not yet be well-informed do not find a hostile environment when they join.

The forthcoming revision of the python.org web site is an attempt to democratize the content and lower the barriers to making a contribution. The intention is to allow contributors to directly affect the content in their area of competence, without requiring mediation from a "webmaster" team who are familiar with the gory internal technical details of the site. More such initiatives are required.

REACHING OUT

So, it's important to reach out to the people with the skills to make our community, and our software, better: more user-friendly, more robust, better documented, and so on. The existing community does a fantastic job, but think how much better things could be if we added new skills by growing the community outside its current demographic. While one of the advantages of open source, to many participants, is its lack of corporate rules and regulations, we nevertheless should accept that we do face many of the same problems that businesses do. If we can find new and creative solutions to those problems then that's great. If we can't, however, we shouldn't necessarily ignore the solutions that have already been developed in more traditional environments.

Besides wanting to increase the size of the existing Python community, I would also like to see the Python world reaching out more determinedly to groups supporting specific applications areas such as accounting, law, the conference industry, the print industry, the transport industry, the aerospace industry, and so on.

The scientific community is a stellar example of the sort of collaborations I envisage. There, the community members are mostly not interested in Python for its abstract beauty, and don't have any ambition or desire to contribute to the core. They use Python because it has been demonstrated to be an effective tool for solving their problems. We need to remember that's the motivation of almost all users outside the development community.

In my secret life as a conference organizer I am using Python in the shape of Eldarion's Symposion project. While it does currently have many failings, my approach is to use it intensively, and feed back the problems to the developers for fixes. As funds become available we intend to invest in improvements that will make our job as conference organizers easier and more effective. I am hoping that eventually, once we have together built a project that non-specialist users can be happy with, Eldarion and The Open Bastion will find a market among the small to medium-sized conferences, and another industry will start to fall under Python's sway.

Repeating such efforts across many industries is, in my opinion, the best way to make Python popular, and its many existing successes are a stable platform on which to base this work.  This will involve effort, since we have to explain open source to people who are not as familiar as we are with the milieu, and whose interest in it is limited to the areas in which it can help them get their jobs done.

TAKING RESPONSIBILITY

Python is a mature technology, and it's time to consider it, if not exactly a finished work, at least as something that can be presented as a feature-complete tool for use in many different application areas. The UK government, having accepted that its computer education has gone through a period in the "dark ages," has accepted that computational thinking is a core skill in today's world. It looks as though Python will be adopted aggressively as a teaching language. I hope that success will be mirrored in many other countries.

If we are prepared to engage others with different skill sets from our own, and listen to their needs, we can build new communities who will complement our existing skill set and help us promote Python world wide in a multitude of application areas. The benefits for the Python community, though, are only a tiny portion of the gains to be made.

I mentioned at the start of my talk that the world is demonstrably broken. Central governments have neither the resources nor the imagination nor the flair to apply global solutions to local problems. It seems self-evident to me that the best way to solve local problems is locally, and so we need to persuade these governments to release some of their control and their funding to the communities with the power to solve those problems.

Eric Sterling gave a very engaging keynote at DjangoCon this year in which he suggested that it was important for a much broader cross-section of the population to make their needs known. As a group who only have to see an itch to start thinking about how to scratch it, I think that open source devotees are in a position to provide some valuable answers to the question of how we fix some of today's complex problems. This means leaving their comfort zone, reaching out and engaging with government and commerce at all levels, and I want to persuade open source communities to use their considerable skills to do so. The world is broken, but I think with determination and goodwill we can help to fix it.

This is an exciting time to be a Python programmer!