MacOS Migration: If I Have to Shave One More Yak ...

So I decided it was time to update my laptop, the current Macbook Air being three years old and having a tendency to forget that the screen is present. The triggering event was a minor spillage which stopped several crucial keys from working as the beer dried out (yes, I know ...).

Clearly the first thing to do was take the Air off the 'Net and back it up. Connecting the backup disk and starting a Time Machine backup saw it identify about 75,000 files of the 1,500,000 on y hard disk needed backing up. When the backup phase started it took about five minutes to get to "3k of 4.5GB backed up" and I realized I had a problem.

StackOverflow suggested I consider repairing the Time Machine disk. That's another hour I shall never see again, but at least Disk Utility gave the volume a clean bill of health, so I restarted the Time Machine backup and this time (hallelujah!) it ran. I then ran the Migration Assistant on the new (receiving) Mac and told it to restore my account, settings, applications - the whole caboodle. Since the Assistant kindly estimated this would take at least two hours and fifteen minutes I decided this would be an appropriate point at which to go for a couple of drinks with my buddy Kirby.

I found the Apple Migration Assistant to be quite friendly (disclaimer: I have always previously used manual migration procedures to switch computers and so cannot say whether Windows offers similar friendliness; if not, it should). When I returned about 90 minutes later I was delighted to find that the restore was complete (I hadn't waited around for Migration Assistant to upgrade its estimate), so I was able to log in to my newly-restored account on the updated Air.

My joy lasted as long as it took me to run a terminal session, when I saw the following delight:

Last login: Tue Nov 19 01:27:18 on tty??
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.4/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Library/Python/2.7/site-packages/virtualenvwrapper/hook_loader.py", line 16, in 
    from stevedore import ExtensionManager
  File "/Library/Python/2.7/site-packages/stevedore/__init__.py", line 3, in 
    from .extension import ExtensionManager
  File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 4, in 
    import pkg_resources
ImportError: No module named pkg_resources
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenv has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python and that PATH is
set properly.
AirHead:~ sholden$

Not exactly what you want to see when you log in, but at least indicating the the error was probably with my virtual environments. Fortunately it was quite easy to fix this daunting error message by doing a brew install python after uninstalling the previously installed version (thereby achieving an upgrade from 2.7.2 to 2.7.5). Since my account is set to prefer /usr/local/bin to /usr/bin it finds the updated python immediately. I then upgraded pip to the latest version and I appear to be good to go (though I am sure I shall find many other bumps in the road).

A very pleasant surprise was that the Mac knew about all my printers and was happy to try and use them for me. I won't know whether they work until I get back to the network to which they are connected, but if they do I will be both surprised and delighted. This is clearly superior technology where the user experience has been considered relatively carefully.

Unfortunately the user before me had already started using brew, so quite a lot of reowning was required to make my account the owner of essential locations before I could start to access the brew package again. Once I could start to think about doing that I discovered that brew update would not work because "The following untracked working tree files would be overwritten by merge". Thanks to this very helpful web page I updated by brew installation and then the update went ahead just fine. After which a brew upgrade decided that 35 packages needed updating (gulp!) and so I am currently waiting to see if the following packages upgrade without issues:

ack 2.10, atk 2.10.0, cairo 1.12.16, cmake 2.8.12.1, erlang R16B02, fontconfig 2.11.0, gdk-pixbuf 2.30.1, gettext 0.18.3.1, gfortran 4.8.2, git 1.8.4.3, glib 2.38.2, gmp 5.1.3, gtk+ 2.24.22, harfbuzz 0.9.24, icu4c 52.1, libmemcached 1.0.17, libxml2 2.9.1, mercurial 2.8, mplayer 1.1.1, ncdu 1.10, nmap 6.40, pango 1.36.1, pixman 0.32.2, postgresql 9.3.1, pv 1.4.6, pypy 2.2.0, python 2.7.6, python3 3.3.2, redis 2.6.16, sphinx 2.1.3, sqlite 3.8.1, xz 5.0.5, zeromq 3.2.4

I'll let you know how it goes! In the meantime, though my virtual environments still need some attention I am able to run Pythons 2 and 3 and the IPython Notebook. Kudos to Apple, this migration has left me much happier than I would have imagined.

What Do You Care?

I was saddened to discover from this blog post that Nadezhda Tolokonnikova is currently (and, most likely, call me paranoid if you will, deliberately) unaccounted for in the Russian prison system. I have never met this woman, but I know she has a family, and friends, who love her and need to know she will be kept safe. I also know that she tried, with her fellow band-members, to bring to the world's attention the fact that their homeland might not be ideal. For which they are all being punished.

If you won't just bring this one tiny injustice to your world's attention, what do you expect other people to do for you when you need help?

The Russian state needs to understand that unless it chooses to give up any pretense of listening to public opinion then it must get used to the fact that it too is being watched. If you can't find time to help Nadezhda, who should find time to help you?

Of course if you'd rather just write computer programs then this is just another annoying whiney post in a long series. Whose number is, by now, doubtless legion, sorry about that. But when you need to find your significant other in the TSA miscreants' repository don't come round asking me for help.

You think this could never happen? I so hope you are right.

Bradley Manning's Post-Sentencing Statement

This statement from Bradley Manning was read after hos 35-year sentence was passed by his lawyer, David Coombs.

The decisions that I made in 2010 were made out of a concern for my country and the world that we live in. Since the tragic events of 9/11, our country has been at war. We’ve been at war with an enemy that chooses not to meet us on any traditional battlefield, and due to this fact we’ve had to alter our methods of combating the risks posed to us and our way of life. 

I initially agreed with these methods and chose to volunteer to help defend my country. It was not until I was in Iraq and reading secret military reports on a daily basis that I started to question the morality of what we were doing. It was at this time I realized in our efforts to meet this risk posed to us by the enemy, we have forgotten our humanity. We consciously elected to devalue human life both in Iraq and Afghanistan. When we engaged those that we perceived were the enemy, we sometimes killed innocent civilians. Whenever we killed innocent civilians, instead of accepting responsibility for our conduct, we elected to hide behind the veil of national security and classified information in order to avoid any public accountability. 

In our zeal to kill the enemy, we internally debated the definition of torture. We held individuals at Guantanamo for years without due process. We inexplicably turned a blind eye to torture and executions by the Iraqi government. And we stomached countless other acts in the name of our war on terror. 

Patriotism is often the cry extolled when morally questionable acts are advocated by those in power. When these cries of patriotism drown our any logically based intentions [unclear], it is usually an American soldier that is ordered to carry out some ill-conceived mission. 

Our nation has had similar dark moments for the virtues of democracy—the Trail of Tears, the Dred Scott decision, McCarthyism, the Japanese-American internment camps—to name a few. I am confident that many of our actions since 9/11 will one day be viewed in a similar light. 

As the late Howard Zinn once said, “There is not a flag large enough to cover the shame of killing innocent people.” 

I understand that my actions violated the law, and I regret if my actions hurt anyone or harmed the United States. It was never my intention to hurt anyone. I only wanted to help people. When I chose to disclose classified information, I did so out of a love for my country and a sense of duty to others. 

If you deny my request for a pardon, I will serve my time knowing that sometimes you have to pay a heavy price to live in a free society. I will gladly pay that price if it means we could have country that is truly conceived in liberty and dedicated to the proposition that all women and men are created equal.

One might wish that President Obama could put up such a principled defense for his scurrilous conduct in handing over the security of the American people to secret courts and universal surveillance.

Getting Back to IT

Life has become much more interesting of late. With two or three interns in The Open Bastion office periodically I am revisiting basic software development practices to try and explain them. I am using it as an opportunity to bring my own skill-set up to date, and it's immense fun to actually get to grips with, even though quite a challenge in the face of intelligent (and sometimes daunting) questioning from the interns. We are all learning a lot, me included.

Because we want to be able to build flexible web systems we have decided to use django CMS as our base system. The editing interface may not represent the absolute zenith of design (though the 3.0 release, currently in Beta, looks rather better), but it appears to be a solid piece of technology and it is capable (when applied by a team including design skills) of producing attractive and easily-edited web content. Even with a relatively puny SQLite database and with the Django debug toolbar switched on you can see how people with little deep understanding of information technology could nevertheless use it to build their own web resources fairly quickly.

The original attraction of django-cms was its ability to maintain flexible content via its flexibly extensible plugin system. This is a valuable part of keeping content separate from presentation—and, of course, there's nothing to stop you writing plugins to use the same data as other Django code that's already part of your site. For a crowd of tech-heads the most difficult part will be the presentation and styling. Who knows, perhaps a design intern will want to get in on the ground floor.

Our first task was to get to grips with a solid but simple workflow. We have now settled on the simple but effective git branching model proposed in Vincent Driessen's excellent paper. Of course since everyone is learning git we are making all the usual mistakes and stumbling along at times. It's reassuring that whatever mistakes you do make are usually recoverable. Nowadays I sometimes even remember to checkout a new branch before committing development changes.

We are currently at the stage of creating a solid build and learning how to work with plugins. I have managed to extend the tables plugin to allow drop-down style selection (albeit with a somewhat inflexible mechanism) and gained much insight thereby. We still face the challenges of testing and deployment

Sadly now I have to go back to writing the code to create the slots for the DjangoCon schedule, but I hope to write more later.

OSCON Community Leadership Summit

I've just come back enthused from having spent my entire weekend working.

Of course anyone who works in open source will recognize the effect: someone gives you a chance to learn from and inform a couple of hundred like-minded individuals and there goes the weekend. Didn't even go to the PyLadies informal Saturday gathering, normally a fixed point of the weekend when I'm at home (a sadly somewhat too rare occurrence still).

OSCON is celebrating its fifteenth anniversary this year, and four years ago Jono Bacon of Canonical engineered the first Community Leadership Summit. I participated in this fifth run more fully than in previous years, and have benefited from much collective wisdom.

As is my practice at conferences I spent a lot of time in the hallway track. I acquired this habit from years of organizing PyCon, when I would frequently be waylaid on my way to some session and end up having an amazingly interesting and informative discussion, meeting new people, broadening my knowledge of Python and the open source world generally. If I could hire all the people I know in the open source world (which I couldn't, even with unlimited capital, since many of them are pursuing their own dreams) I would be able to build an unbeatable distributed information systems engineering and operations team.

A couple of interesting facts appeared as the weekend went by. First of all, perhaps fifteen to twenty-five percent of those attending had no direct connection with the open source development world. Some of these people were enthusiastic users of open source projects, and others were community managers who have started to realize that the way the open source world uses a diverse distributed open toolkit to achieve its goals could be useful to them too.

Secondly, and perhaps largely as a result of the first, the open source crowd began to realize that as the community continues to become more diverse (yay, diversity!) we have to stop talking in jargon about things like "cloning git repositories" if we don't want these brave spirits who are here to learn about the social and organizational side of open source to be completely turned off by our inability to speak plain English*.

We might also need to consult with UX teams to ensure that our current technologies can be used as the basis of a layer that makes sense to people who don't have our intimate knowledge of the technologies but are nevertheless keen to learn and use our methods. That could represent a substantial challenge.

The lightning talks were uniformly inspiring. Jono was kind enough to say he really enjoyed mine, which was called Making Small Positive Differences. The tradition at the CLS is to set the talk times to accommodate everyone who signed up. I didn't know this, so like many of the other speakers I entered the room expecting a five-minute slot.

Fortunately I was fifth on the list, so I had time to go through my slides (did I mention there was no projector? another surprise, but what the hey, this is mostly an unconference) and extract the salient points into notes for a three-minute talk in my extremely convenient (and free) O'Reilly notebook.

Some of the gems of the lightnings for me included:

• Kevin Johnson's appeal to share our technology to the benefit of all
• Britta (?)'s suggestion that barriers to entry weren't always a bad thing, with examples of how they could work in a community's favor (without discouraging diversity?)
• Josh Hibbet's talk about CityCamp, which could become a nationwide (worldwide?) series of events to promote citizen participation and government transparency
• Gribble, an IRC bot built by SourceForge to be friendly and welcoming to new IRC community members
• Selena Deckleman's invitation to go and speak in a K-12 classroom
• Aaron Wolf's promotion of the Snowdrift Co-op
• A talk by someone [whose name I need to research so I can introduce him to a colleague with a keen interest in such matters] who has developed a way (if I understood correctly) to codify the law and represent legal entities engaging in contractually-bound mutual obligations.

I also attended sessions on how to attract a more diverse skill set into the open source world (something I have been banging on about for a while now in keynotes up and down), and discussions about how, when you are building teams with people who have never worked together before, it's difficult to manage projects because things can get blocked when person A doesn't even know that person B cannot proceed until A's part of the task is complete.

This last discussion raised many interesting aspects of how people working on project teams communicate, and we realized that training was one important way that people with diverse skills can learn to work together effectively and harmoniously.

All in all this was an excellent start to OSCON, and I feel more in touch with the hot-button issues of the collected open source communities as the convention proper begins (tutorials are the main activities today and tomorrow). I'm looking forward to a great week, and hope to round it off with a reprise of last year's OSCON Survivors' Breakfast. Drop me a line if you'd like an invite.

* No holier than thou here. The best I can do at short notice is:

If you just need to use project X, go to [some web page] and click the "Install" button. Then follow the instructions on that page to make sure the installation has succeeded, and the follow the [tutorial web page] or browse [web search for currently best-rated and/or most popular blog posts about Projct X]

If you are a project X user who finds it unsatisfactory in some way we would love to hear from you. Only by being keenly attentive to feedback from our dedicated users can we ensure that project X continues to become more useful to a wider range of people every time.

If you would like project X to do things it currently doesn't please let us have your ideas on [ideas@projectX.hostingsite.example.com]. We will always attempt to acknowledge input and respond to feedback.

If you would like to help project X become more capable then please ask us how you can support the open source communities who built and help to maintain it. We are always happy to welcome new project members, who can provide the one thing without which open source could not be possible: the willingness to build software to raise the tide and level the playing field [see what I just did there]. 

By creating new infrastructure open to all we hope to improve humanity's lot more effectively.  Project X is available under the [link to OSI-approved license] and its documentation is available under the [some sort of Creative Commons license] on Read The Docs.

 

A Statement from Adria Richards

Having remained silent since PyCon, Adria Richards has now published a statement reflecting on those events, though sensibly without going into specifics. I think it's important that her retrospective view gets some air, so I am reproducing it here with my own comments.

Those who know me well in the the developer and tech community recognize that I have always tried to conduct myself in a way that builds bridges for everyone. My central aim is to do everything I can to help create new, inclusive inroads for all, no matter who they are, where they come from or what they believe. Development is about innovation, creativity, and in a grand sense, the betterment of human society through technology. So, it stands to reason that everyone should have a seat at the table, and everyone involved in this vital community should feel welcome, safe and respected. In essence, the worldwide community of developers can and should function as a reflection of what our wider society strives to be.

I don't know Adria, so I take her opening statement at face value. Those who disagree may do so elsewhere. Garann Means recently made a fairly good case that the denizens of the technology world don't act in a way that inspires respect from other communities. I think it's fair to say (and some of the comments in forums like Hacker News amply demonstrate) that there is plenty of room for improvement in what's considered acceptable conduct in the technology world.

I cannot comment at this time on the specifics of what occurred at PyCon on March 17, and the subsequent events of the following days, but I can offer some general thoughts. I don’t think anyone who was part of what happened at PyCon that day could possibly have imagined how this issue would have exploded into the public consciousness the way it has. I certainly did not, and now that the severest of consequences have manifested, all I wish to do is find the good in what has been one of the most challenging weeks of my life.

This positive attitude is to be admired. I have been through similarly shitty situations in my own life (as most of us have at one time or another) and while it's good to move on I like to extract lessons that will help me try to avoid making the same mistakes. If I can find those I can feel more positive about the experience.

And I do believe there is good to be found in this situation. Debate and recrimination can and must give way to dialog that explores the root causes of these issues in the tech industry. As developers and members of the startup community, we can welcome newcomers, women and people of color who, as of now, are under-represented in our ranks. And, all of us can learn a great deal from those who are well-established in the field. We can solidify the values of our workplaces (yes, conference spaces are workplaces!), and set new, positive and inclusive examples for other professional disciplines.

This is a good idea. PyCon was started to try and provide an inclusive space where everyone with interests in Python can come together for the betterment of all. I would be suspicious of those who don't support the above as an ideal. We surely have to accept that there's still along way to go to get there. So dialog is a good idea.

What happened at PyCon has cast a spotlight on a range of deep issues and problems in the developer world. As ugly as this situation has become, all of these issues have reasonable, and, I think, easily reached solutions that will help us cast conflict aside and construct a more cohesive and welcoming professional environment based on respect, trust and open communication. I do not, at this time, wish to concentrate on the fallout of the last several days. Instead, I want to be an integral part of a diverse, core group of individuals that comes together in a spirit of healing and openness to devise answers to the many questions that have arisen in the last week. Together, we can work to make the tech world a better place to work for everyone, and in doing so, we make the wider world a better place for all.

It's interesting that this episode happened at PyCon. When we started to talk seriously about Codes of Conduct in the Python world there was a lot of kick-back, and people said things like "but this is the Python community" (implying that we are such nice people that we won't have the same issues) and "but that's just normal behavior" (not even implying, but specifically stating, that anyone who needed a Code of Conduct to condition their behavior shouldn't be going to conferences anyway).

We pressed ahead none the less, refining the code to improve its expression of our ideals of openness and inclusivity. When issues had to be addressed (and you can read the reports in the PyCon blog if you need to refresh your memory) the code of conduct was applied, and to my mind it worked. The fact that two people lost their jobs as a result of the fall-out was nothing to do with the code of conduct, and yet there were many who felt that the blame should be laid at PyCon's door. I shudder to think what the result would have been at an event with no explicit code.

I hope that Adria can come back to PyCon without being subjected to any hostility. She expresses a great ideal, one that we should all be striving for. Now we just have to work out how to achieve it. But that's an implementation detail, right? I wish!

So Long, and Thanks for All the Fish

A notable transition point in my life, by any measure. Interesting things are happening in all directions, many of them (I am delighted to day) heavily involving Python. So this is definitely not goodbye. My time on the Foundation's board has been a wonderful and interesting ride. I hope I have lived up to Tim O'Reilly's ethic of creating more value than I extract.

Why Tuples?

A frequent question from new and even not-so-new Python programmers is "why does the language have both tuples (which, if you know Python, you will recall are immutable) and lists?" You might almost say there are two kinds of Python programmers, those who know what tuples are for and those whose mathematical education has been limited. I know this sounds like an awfully snobbish thing to say, but it isn't meant that way. The fact is that I learned most of my programming in eight years of working experience before I started my degree studies, and I am therefore of the very definite opinion that people with little mathematical background can be excellent programmers.

It's just that I myself only came across the word tuple when I started studying college-level mathematics and had to come to terms with things like

An NFA is represented formally by a 5-tuple, (Q, Σ, Δ, q₀, F), consisting of

• a finite set of states Q
• a finite set of input symbols Σ
• a transition relation Δ : Q × Σ → P(Q).
• an initial (or start) state q₀ ∈ Q
• a set of states F distinguished as accepting (or final) states F ⊆ Q.

Now this has a very definite meaning. It tells us that an (ordered - i.e. positionally-identified) set of five things is sufficient to define the behavior of a specific type of object (in this case a non-deterministic finite state automaton (NFA), though for all you need to know about those I might just as well be describing a flux capacitor).

If we wanted to encapsulate an NFA as a Python data structure, then we might at some point in our code write in our Python code something like

    nfa = (states, symbols, transition, initial, accepting_states)

though in actual fact you would be more likely to want to incorporate behavior in a class and so write instead

    nfa = NFA(states, symbols, transition, initial, accepting_states)

Now even though you may not know what an NFA is, you will surely perceive that the set of its possible states is a very different thing from the function that determines how a current state and a set of inputs are mapped into new states. So there is simply no way that it would be meaningful to write

    for thing in (states, symbols, transition, initial, accepting_states):
        do_something_with(thing)

unless you were, for example, trying to save its state during a pickling operation or some more obscure book-keeping metacode.

And that, best beloved, is what tuples are for: they are ordered collections of objects, and each of the objects has, according to its position, a specific meaning (sometimes referred to as its semantics). If no behaviors are required then a tuple is "about the simplest thing that could work."

This is why you often hear people informally say tuples are for "collections of things you don't need to iterate over" or "tuples are for sequences of dissimilar objects". My advice would be to stay away from such discussions. One might reasonably argue that including them in the language discouraged people from using objects with named attributes, which are always easier to deal with.

The problem with the tuple is that once we have constructed our NFA the only way to refer to the states in code is with the rather unedifying expression

    nfa[0]

which doesn't actually tell the reader much about the programmer's intention. The sequence nature of the tuple means that our accesses to the elements are difficult to imbue with meaning in our code. This latterly became such an obvious shortcoming that it prompted Raymond Hettinger to create the namedtuple object that allows you to easily specify a tuple whose elements can also be referred to by name.

It would be interesting to see whether users of namedtuple objects actually use them as tuples at all. I would guess that the sequence behaviors are rarely used, in which case perhaps it's time to either remove namedtuple's __getitem__() and similar methods or implement a similar object without the sequence behaviors.