Carl Trachte: Mining Engineer Turned Geek

Another #LABHR post.

I first met Carl when he came to a training session for new PyCon speakers. He was very nervous, and concerned that his topic would not be of sufficient interest to his audience. I made a point of attending the beginning of his talk, and by the time I left there was no doubt in my mind that Carl would do well in the Python community.

Over the years he has done invaluable work for the PSF, both in policing potential abuses of the Foundation's trademark for the Python programming language, and also in collecting sample descriptions of Python in many languages and alphabets to try and make Python's web presence more welcoming to those from diverse linguistic communities.

He has been a PyCon volunteer for a while now, helping others to find what they need and demonstrating by example the helping spirit of Python and open source generally.

Carl has done all this with no thought of return, and in doing them has helped uncountable numbers of Python users. Well done, Carl, and thanks!

Intermediate Python: An Open Source Documentation Project

There is a huge demand for Python training materials, and there are many people who just don't have the spare cash to buy books or videos. That's one reason why, in conjunction with a new Intermediate Python video series I have just recorded for O'Reilly Media I am launching a new, open source, documentation project.

My intention in recording the videos was to produce a broad set of materials (the linked O'Reilly page contains a good summary of the contents). I figure that most Python programmers will receive excellent value for money even if they already know 75% of the content. Those to whom more is new will see a correspondingly greater benefit. But I was also keenly aware that many Python learners, particularly those in less-developed economies, would find the price of the videos prohibitive.

With O'Reilly's contractual approval the code that I used in the video modules, in IPython Notebooks, is going up on Github under a Creative Commons license [EDIT: The initial repository is now available and I very much look forward to hearing from readers and potential contributors - it's perfectly OK if you just want to read the notebooks, but any comments yuu have about your experiences will be read and responded to as time is available]. Some of it already contains markdown annotations among the code, other notebooks have little or no commentary. My intention is that ultimately the content will become more comprehensive than the videos, since I am using the video scripts as a starting point.

I hope that both learner programmers and experienced hands will help me turn it into a resource that groups and individuals all over the world can use to learn more about Python with no fees required. The current repository has to be brought up to date after a rapid spate of editing during the three-day recording session. It should go without saying that viewer input will be very welcome, since the most valuable opinions and information comes from those who have actually tried to use the videos to help them learn.

I hope this will also be a project that sees contributions from documentation professionals (and beginners they can help train), so I will be asking the WriteTheDocs NA team how we can lure some of those bright minds in.

Sadly it's unlikely I will be able to see their talented array of speakers as I will still be recovering from surgery. But a small party one evening or a brunch at the office might be possible. Knowing them it will likely involve sponsorship or beer. Or both. We shall see.

I think it's a worthwhile goal to have free intermediate-level Python sample code available, and I can't think of a better way for a relative beginner to get into an open source project. I also like the idea that two communities can come together over it and learn from each other. Suffice it to say, if there are enough people with a hundred bucks* in their pocket for a six-hour video training I am happy to use part of my share in the profits to support this project to some degree.

[DISCLOSURE: The author will receive a proportion of any profit from the O'Reilly Intermediate Python video series]

* This figure was plucked from the air before publication, and is still a good guideline, though as PyCon opened (Apr 11) a special deal was available on a package of both Jessica McKellar's Introduction to Python and my Intermediate Python.

OSCON Community Leadership Summit

I've just come back enthused from having spent my entire weekend working.

Of course anyone who works in open source will recognize the effect: someone gives you a chance to learn from and inform a couple of hundred like-minded individuals and there goes the weekend. Didn't even go to the PyLadies informal Saturday gathering, normally a fixed point of the weekend when I'm at home (a sadly somewhat too rare occurrence still).

OSCON is celebrating its fifteenth anniversary this year, and four years ago Jono Bacon of Canonical engineered the first Community Leadership Summit. I participated in this fifth run more fully than in previous years, and have benefited from much collective wisdom.

As is my practice at conferences I spent a lot of time in the hallway track. I acquired this habit from years of organizing PyCon, when I would frequently be waylaid on my way to some session and end up having an amazingly interesting and informative discussion, meeting new people, broadening my knowledge of Python and the open source world generally. If I could hire all the people I know in the open source world (which I couldn't, even with unlimited capital, since many of them are pursuing their own dreams) I would be able to build an unbeatable distributed information systems engineering and operations team.

A couple of interesting facts appeared as the weekend went by. First of all, perhaps fifteen to twenty-five percent of those attending had no direct connection with the open source development world. Some of these people were enthusiastic users of open source projects, and others were community managers who have started to realize that the way the open source world uses a diverse distributed open toolkit to achieve its goals could be useful to them too.

Secondly, and perhaps largely as a result of the first, the open source crowd began to realize that as the community continues to become more diverse (yay, diversity!) we have to stop talking in jargon about things like "cloning git repositories" if we don't want these brave spirits who are here to learn about the social and organizational side of open source to be completely turned off by our inability to speak plain English*.

We might also need to consult with UX teams to ensure that our current technologies can be used as the basis of a layer that makes sense to people who don't have our intimate knowledge of the technologies but are nevertheless keen to learn and use our methods. That could represent a substantial challenge.

The lightning talks were uniformly inspiring. Jono was kind enough to say he really enjoyed mine, which was called Making Small Positive Differences. The tradition at the CLS is to set the talk times to accommodate everyone who signed up. I didn't know this, so like many of the other speakers I entered the room expecting a five-minute slot.

Fortunately I was fifth on the list, so I had time to go through my slides (did I mention there was no projector? another surprise, but what the hey, this is mostly an unconference) and extract the salient points into notes for a three-minute talk in my extremely convenient (and free) O'Reilly notebook.

Some of the gems of the lightnings for me included:

• Kevin Johnson's appeal to share our technology to the benefit of all
• Britta (?)'s suggestion that barriers to entry weren't always a bad thing, with examples of how they could work in a community's favor (without discouraging diversity?)
• Josh Hibbet's talk about CityCamp, which could become a nationwide (worldwide?) series of events to promote citizen participation and government transparency
• Gribble, an IRC bot built by SourceForge to be friendly and welcoming to new IRC community members
• Selena Deckleman's invitation to go and speak in a K-12 classroom
• Aaron Wolf's promotion of the Snowdrift Co-op
• A talk by someone [whose name I need to research so I can introduce him to a colleague with a keen interest in such matters] who has developed a way (if I understood correctly) to codify the law and represent legal entities engaging in contractually-bound mutual obligations.

I also attended sessions on how to attract a more diverse skill set into the open source world (something I have been banging on about for a while now in keynotes up and down), and discussions about how, when you are building teams with people who have never worked together before, it's difficult to manage projects because things can get blocked when person A doesn't even know that person B cannot proceed until A's part of the task is complete.

This last discussion raised many interesting aspects of how people working on project teams communicate, and we realized that training was one important way that people with diverse skills can learn to work together effectively and harmoniously.

All in all this was an excellent start to OSCON, and I feel more in touch with the hot-button issues of the collected open source communities as the convention proper begins (tutorials are the main activities today and tomorrow). I'm looking forward to a great week, and hope to round it off with a reprise of last year's OSCON Survivors' Breakfast. Drop me a line if you'd like an invite.

* No holier than thou here. The best I can do at short notice is:

If you just need to use project X, go to [some web page] and click the "Install" button. Then follow the instructions on that page to make sure the installation has succeeded, and the follow the [tutorial web page] or browse [web search for currently best-rated and/or most popular blog posts about Projct X]

If you are a project X user who finds it unsatisfactory in some way we would love to hear from you. Only by being keenly attentive to feedback from our dedicated users can we ensure that project X continues to become more useful to a wider range of people every time.

If you would like project X to do things it currently doesn't please let us have your ideas on [ideas@projectX.hostingsite.example.com]. We will always attempt to acknowledge input and respond to feedback.

If you would like to help project X become more capable then please ask us how you can support the open source communities who built and help to maintain it. We are always happy to welcome new project members, who can provide the one thing without which open source could not be possible: the willingness to build software to raise the tide and level the playing field [see what I just did there]. 

By creating new infrastructure open to all we hope to improve humanity's lot more effectively.  Project X is available under the [link to OSI-approved license] and its documentation is available under the [some sort of Creative Commons license] on Read The Docs.

 

Building New Communities

[This post is based on a keynote talk to the PyCon DE conference on November 1, 2012]

[UPDATE: Nov 17, 2012: The video of this talk is at http://pyvideo.org/video/1479/building-new-communities]

Good afternoon, and thank you for inviting me. It’s always intimidating to talk to an audience which is so technically capable. Today, though, I am hoping that we can look at a non-technical topic that has implications for everyone who is interested in Python: how do we promote the Python language and improve the development process?

DO EXISTING COMMUNITIES WORK?

Let me begin by saying that I have invested huge amounts of time in the Python community, and I believe that it has a well-deserved reputation as one of the more welcoming open source communities. So today I am talking more broadly than just Python. What I say applies, I believe, to many open source communities.

This talk is being given at an unprecedented time in human history. Clearly these things are a matter of interpretation, but nowadays it’s hard to find someone that doesn’t agree that the modern world is failing to meet the needs of many of its inhabitants.

Historically, we are living in strange times, The world human population reached 7 billion, depending on which figures you choose to believe, between October 2011 and March 2012, only twelve years after it reached 6 billion. According to one UN estimate it’s possible that population will reach ten billion before reaching any kind of stability, and that population in Africa could triple in this century. However you interpret these figures it is obvious that most governments are going to be struggling with population growth in a world where they are already finding it difficult to manage things to everyone’s satisfaction.

So I have a lot of questions, and not many answers. Being a geek, my approach attempts rationality—we have to take into account what we know about human behavior, differences between the various national and political cultures in which people live, and so on. In an increasingly crowded world it is easy to feel threatened by governmental and economic systems that are posited on continuous growth.

When I was young an organization called the Club of Rome produced a report called “The Limits to Growth,” positing that the finite nature of many resources could, if growth were not checked, lead to cataclysmic failures in supply of the very things that keep us alive: clean air, clean water, and nutritious food. This is where climate change deniers got their start, and yet from recent work it appears that the report was, within its limitations, a very accurate projection of certain futures.

My own vision of the future tends, I must admit, towards the catstrophic. I think it is entirely possible that the human race will wipe itself out simply by continuing to foul its own nest and ignoring the toxic environment it is creating. So I see it as important to try and bring this to people’s attention, and to do what I can to ensure that we (the human race) build a sustainable future for ourselves and the other species we have to share this huge ball of mud we call Earth with.

HEY, ISN'T THIS PYCON?

All of this might seem a very long way from our little open source world. I can already hear people thinking that I am abusing my position as a keynote speaker to stand on a political soap box and peddle my socialistic ideas. I admit I am a socialist, and those who know me also know that I am happy to stand on a soap box at times. But this is not one of those times.

Context, however, is everything. So I want to place my remarks today in the context of a world that isn’t working, that isn’t equitable, and in which ever larger divisions are opening up: between the poor and the rich in almost any country you like to name, between poor countries and rich countries, and so on. Huge amounts of resource are being wasted on machinery to wipe each other out. This is particularly noticeable in my current home country, the USA, which has 11% of the world’s population and yet is responsible for over 40% of all military expenditures, more than the next 14 top-spending countries put together. One might wonder what they are afraid of, but these expenditures certainly give them clout in the geopolitical sphere.

Open source communities, the Python community included, have growth problems of their own. According to Armit Deshpande and Dirk Riehle the number of lines of open source code is growing exponentially. This is a management problem of some magnitude, which is not being addressed at the moment. They suggest that open source applications generated revenues of $1.8 billion in 2006, and certainly growth has continued unimpeded since. Of course that $1.8bn represented only about three-quarters of one percent of total software revenues, so there is still a long way to go.

Tim O'Reilly made a sensible case this year that open source is responsible for increasing small business revenues of a single hosting company’s user base by $180 billion  And, of course,  many proprietary software products nowadays contain significant open source components.

GET PAID TO BUILD STUFF?

I have said many times that the emergence of open source represents a new economic phenomenon. It’s as though, when railways were needed in the United States, a bunch of people had said “You know, that’s a great idea. Let’s go build a railway system,” taken up their picks and shovels and headed off to work. Of course you can’t build a railway without access to heavy engineering, which is arguably difficult to open source. Similarly, most individuals would not find it cost-effective to make their own computers from scratch, or to interconnect them in the way that the infrastructure of the Internet does.

Just the same, the emergence of Linux as a powerful force in the software world, and the somewhat anarchic, less-than formal management of the whole Internet (despite the belief of many US politicians that the Internet should be managed by the US as a divine right) should convince us that it’s possible for individual and group initiatives to take hold and prosper in today’s rather strained commercial environments.

I understand that open source is not the exclusive domain of unpaid experts. Nowadays a lot of companies are paying programmers and other staff to assist with the creation of open source software projects. Many different models are used to try and recoup the investments made, but the fact remains: it is now commonly accepted that there is value to creating common infrastructure whose presence benefits a large proportion of the community rather than a single individual or organization. Of course such ideas often seem, particularly in the USA, socialistic and therefore suspect, but it is still true that large companies like HP, IBM and Microsoft are enthusiastically involved in myriad open source projects. They aren't generally thought of as having socialist tendencies.

HOW IS SOFTWARE BUILT?

http://www.multunus.com/2011/02/software-development-as-a-balance

Here’s a fairly typical example of the “software development lifecycle.” It lists a large number of activities and requirements, which deserve to be looked at in detail.

Whether all these activities are strictly necessary or not could be a pleasant discussion over drinks, but I get the definite impression that many open source projects aren’t engaged in any kind of formal process. This freedom, of course, is one of the things that attracts many people into the open source world, and I would be the last person to suggest that unnecessary work should be involved in software production. Nevertheless, open source projects are predominantly run vby programmers, and often all project members are programmers.

WHO BUILDS SOFTWARE?

http://www.ambysoft.com/essays/agileRoles.html

This is a typical conception of the way software is developed in the agile development world. There are many different ways to think about the development process. Arguably, none of them is specifically right or wrong. But when you think about commercial projects the size of Python, there will be many more people involved than just programmers. So Python, like most other open source projects, has to press developers into service to fill the other roles, or leave them unfilled.

WE need to think about whether this works to the ultimate advantage of our projects, or whether it is really a rather inefficient way to produce software.

SOME DEVELOPER ROLES

Above are just some of the roles that a large-ish software development requires. I would argue that lots of the time projects focus largely or even solely on the programming task, with some design work. Many of the other roles do not offer attractive work for developers, who therefore ignore the need to fill them, often to the project’s great detriment.

A prime example here is the Python documentation. Good as it is in comparison with many other open source projects (and it is good, because there has been a lot of hard work put in on it), it still sucks. Its organization is mostly based on a fifteen-year-old structure, back when Python was a smaller and somewhat different language. I still have no idea why, when I want documentation on the Python standard types (int, list, dict, and so on) I have to look in the Standard Library documentation. They are available without any need for an import statement, so why are they documented with the library?

Of course anyone who has used Python seriously for more than a year knows these things, and can usually find out where to look for the information they need, but this is entirely the wrong emphasis. The first need for documentation is when you are learning a language, and then the information you need should be readily available. Who is the advocate for a rewrite or reorganization of the documentation? Nobody in the current development team relishes such a task, because we don’t have technical authors and editors as a part of our community. And yet this issue causes learners way more pain than it should, and even turns people away from the language.

This brings me to my central thesis: if we don’t broaden our community to include such people we are unlikely to achieve the world domination we all jokingly feel that Python could achieve. I know from my work over the last eighteen years, and from visiting Python gatherings like this one all over the world, that Python is poised for massive growth. It would be possible, if a concerted effort was made, to make Python the first programming language of choice for literate students. The world has woken up to the fact that computational thinking is an important skill in today’s complex societies, and some feel that Python is its ideal expression.

So, do we want to slam the doors, say “No, we like things the way they are, we aren’t going to change”? I suspect that some developers will feel that way. But as a PSF director I have to be mindful of our duty to promote the development of the language and grow the international community, and so I have to force myself to look for deficiencies than can be rectified. This means looking outwards, and trying to include people who can fill the necessary roles.

The alternative, of course, is either to ignore the need for a role to be filled or to have someone fill it whose time would be better spent writing code. Neither is an entirely satisfactory solution.

BEING MORE INCLUSIVE

Sarah Novotny, one of the OSCON conference chairs, wrote of her experience at the 2011 conference: "We must challenge the idea that if someone really wants to use a piece of software, he or she will be willing to slog through half-written documentation, the actual code base and an unkind user interface." 

This is an approach that I have tried to promote to Python developers, with rather limited success. I can think of several reasons why this should be so. Here are some classic signs that the status quo is being defended:

• Considering the cost of switching before you consider the benefits
• Highlighting the pain to a few instead of the benefits for the many
• Focusing on short-term costs instead of long-term benefits, because the short-term is more vivid for you
• Embracing sunk costs

We need to try and broaden our development community to include people with the necessary skills, even though we aren't necessarily very comfortable getting close to them. This makes sense to me as a historical perspective: back in "the old days", using an open source project was highly likely to involve you as a contributor as well. Nowadays there are many other roles for contributors to fill: bug reporters, technical writers, bloggers, testers, and I am sure all you core Python developers have your own ideas for roles that could reduce your workload.

David Eaves of Mozilla corporation has done a lot of work on how software communities can use metrics to improve their development process and monitor community involvement. He asks relevant questions like “why are bugs in this section of the code taking twice as long to be reviewed?” and “who has contributed consistently over the last 18 months, but not in the last 30 days?”

If we are prepared to answer questions like that we can take the information, both qualitative and quantitative, and then use it to continue to improve our communities.

INVOLVING THE COMMUNITY

We must accept that not every member of the community wants to be involved outside a fairly narrow specialism. This is fine with me: I would much rather have contented community members contributing their skills than seeing members pressed to leave their comfort zone.

At the same time, there is considerable evidence that an inability to contribute is a source of frustration to would-be active members. So if we want to grow our community we need to be welcoming and, dare I say, open. We need to make sure that the barriers to entry into our community are as low as possible, and that people whose ideas may not yet be well-informed do not find a hostile environment when they join.

The forthcoming revision of the python.org web site is an attempt to democratize the content and lower the barriers to making a contribution. The intention is to allow contributors to directly affect the content in their area of competence, without requiring mediation from a "webmaster" team who are familiar with the gory internal technical details of the site. More such initiatives are required.

REACHING OUT

So, it's important to reach out to the people with the skills to make our community, and our software, better: more user-friendly, more robust, better documented, and so on. The existing community does a fantastic job, but think how much better things could be if we added new skills by growing the community outside its current demographic. While one of the advantages of open source, to many participants, is its lack of corporate rules and regulations, we nevertheless should accept that we do face many of the same problems that businesses do. If we can find new and creative solutions to those problems then that's great. If we can't, however, we shouldn't necessarily ignore the solutions that have already been developed in more traditional environments.

Besides wanting to increase the size of the existing Python community, I would also like to see the Python world reaching out more determinedly to groups supporting specific applications areas such as accounting, law, the conference industry, the print industry, the transport industry, the aerospace industry, and so on.

The scientific community is a stellar example of the sort of collaborations I envisage. There, the community members are mostly not interested in Python for its abstract beauty, and don't have any ambition or desire to contribute to the core. They use Python because it has been demonstrated to be an effective tool for solving their problems. We need to remember that's the motivation of almost all users outside the development community.

In my secret life as a conference organizer I am using Python in the shape of Eldarion's Symposion project. While it does currently have many failings, my approach is to use it intensively, and feed back the problems to the developers for fixes. As funds become available we intend to invest in improvements that will make our job as conference organizers easier and more effective. I am hoping that eventually, once we have together built a project that non-specialist users can be happy with, Eldarion and The Open Bastion will find a market among the small to medium-sized conferences, and another industry will start to fall under Python's sway.

Repeating such efforts across many industries is, in my opinion, the best way to make Python popular, and its many existing successes are a stable platform on which to base this work.  This will involve effort, since we have to explain open source to people who are not as familiar as we are with the milieu, and whose interest in it is limited to the areas in which it can help them get their jobs done.

TAKING RESPONSIBILITY

Python is a mature technology, and it's time to consider it, if not exactly a finished work, at least as something that can be presented as a feature-complete tool for use in many different application areas. The UK government, having accepted that its computer education has gone through a period in the "dark ages," has accepted that computational thinking is a core skill in today's world. It looks as though Python will be adopted aggressively as a teaching language. I hope that success will be mirrored in many other countries.

If we are prepared to engage others with different skill sets from our own, and listen to their needs, we can build new communities who will complement our existing skill set and help us promote Python world wide in a multitude of application areas. The benefits for the Python community, though, are only a tiny portion of the gains to be made.

I mentioned at the start of my talk that the world is demonstrably broken. Central governments have neither the resources nor the imagination nor the flair to apply global solutions to local problems. It seems self-evident to me that the best way to solve local problems is locally, and so we need to persuade these governments to release some of their control and their funding to the communities with the power to solve those problems.

Eric Sterling gave a very engaging keynote at DjangoCon this year in which he suggested that it was important for a much broader cross-section of the population to make their needs known. As a group who only have to see an itch to start thinking about how to scratch it, I think that open source devotees are in a position to provide some valuable answers to the question of how we fix some of today's complex problems. This means leaving their comfort zone, reaching out and engaging with government and commerce at all levels, and I want to persuade open source communities to use their considerable skills to do so. The world is broken, but I think with determination and goodwill we can help to fix it.

This is an exciting time to be a Python programmer!

Fun and Games at OSCON

Once again I spent a week in Portland with over 3,000 Open source enthusiasts, developers, business people and community managers at the largest US open source convention. Once again the hallway track was supremely interesting, and there was much of interest in the sessions.

For me there was a little more business than usual, as a new client of The Open Bastion was present and this was an opportunity to learn more about their team, meet people, form impressions, find out how they work and so on. Also the Python language had a stand in the Open Source Pavilion (alongside many other famous open source names), and I took my Raspberry Pi along. It was amazing how many people had heard of the Pi, and very few actually had one. The commonest comment was “I’m still waiting for mine.”

Alas I again didn't have very much time for the excellent Community Leadership Summit which now traditionally precedes OSCON, though I did manage to take part in one session about conflict resolution, and heard much useful stuff in plenary talks. If you are the leader of any open source community group I would definitely recommend going to this (free) event. It's well worth the cost of an extra couple of night's accommodation. Monday was taken up with clerical work, but on Tuesday I did make it to several sessions of the Business Leadership Day tutorial, and found them sound to excellent.

Tuesday evening Camp OSCON was held—indoors, due to threatened thunder storms which in the end never transpired. This was a traditional O'Reilly party, open bars, many different types of food in all-you-can-eat quantities, airbrushed fake tattoos, the whole bit. I managed to score four party tickets for a friend who came with his wife and two children. They were excited beyond belief, and it was delightful to see how much pleasure a two-minute stop at the registration desk had created.

Among the many activities was a dunking stool, allowing three pitches in return for a $10 donation. My major part in the evening's activities was to be (as I thought) the final target of the event. I took my place, and was duly dunked several times. At one point a guy who looked a lot like Tim O'Reilly took a shot. Good heavens, it was Tim O'Reilly. After failing to unseat me with any of his three pitches he moved closer to the target. Saying “How does this work?” (something I suspect he was not in need of information about) he pressed the target and promptly subjected me to my fifteenth ducking of the evening.

Tim O'Reilly in the dunking tank
Photo from Flickr by aaronparecki

When I emerged from the tank I thought it was all over, but then I saw Tim getting into it. Even though I was still wet I couldn't resist, and promptly made the necessary donation to put me at the front of the line. I am happy to say by the flukiest luck I managed to turn him from dry to wet with my first pitch. Success! My two remaining pitches failed to hit the target, so I went off to the men's room to dry and change. When I emerged I couldn't resist going up to the target, saying “How does this work?” and subjecting him to a second dunking.


I went around back to pack my wet clothes and he insisted on reaching over to shake my hand with a big smile, which reassured me that there were no hard feelings and that the cancellation of my O'Reilly School of Technology classes was not imminent. Tim O'Reilly is a good sport.

Wednesday's keynotes were thought-provoking. I particularly like the fact that the open source world appears to be becoming less shy about the value that it creates in the economy. Tim O'Reilly suggested that successful organizations create more value than they capture, and was very scathing about the short-term thinking and transparent absence of business ethics on Wall Street.

He did some very interesting sums suggesting that open source is a huge net plus to the economy. He likened the situation to that in energy: if we use a clothes dryer to dry our clothes that energy is metered and measured and accounted for. If we dry our clothes on the line using solar power nothing ever gets recorded. Similarly the benefits of open source software have not been sufficiently trumpeted to the world, and so are taken for granted by the commercial enterprises that profit from them because they do not perceive any significant costs. He ended by thanking those of us who have been working in open source for a long time for our forward thinking and our contributions.

Much of the rest of the day I spent talking to people either on the Python stand or in the hallway track. It is always impressive to learn about the range of applications to which open source is being put. The medical faction are much in evidence. The most surprising discovery I made is that at the core of the award-winning open source VistA system is a MUMPS database, technology whose roots must go back over forty years. I sure hope there's a Python API.

Thursday morning's keynotes were more practical in nature, my favorite being the talk from Canonical's Mark Shuttleworth. I'm a sucker for technology, and his explanation of how the juju system lets you build and model complex, scalable systems on the desktop and then easily migrate them to the cloud for testing or production was quite something. The practical demonstration of migrating a service from Amazon to the H-P Cloud service was impressive* (though I was amused to see he did at one point get bitten by a minor gremlin).

There was a frankly disappointing interview with Microsoft's Senior Director for Open Source Communities, Gianugo Rabellino. Not only did it sound as though it might have been scripted by Microsoft's public relations department, it seemed to be mostly "sound and fury, signifying nothing." I am aware that Microsoft** are now a significant contributor to (for example) the Linux kernel and other open source projects, but it's difficult to have a company's representatives talking about open source in "snake oil" terms one year and then see them hold out the olive branch the next.

There is still a substantial mistrust of Microsoft's motivations and purposes among a part of the open source community, as evidenced by a subtle change in the audience's mood. They will have to work hard to prove that they deserve a place in the open source community. This is a pity, because Microsoft employ many smart technical people who fully "get" open source, but it is taking the corporate side a long time to come to terms with it.

The Python stand was very busy again, as were those of the other open source projects. The day went quickly, and I got to listen to a couple of track talks, but again a lot of my time was spent in the hallways. I also spent some time observing the O'Reilly and Convention Center staff, who appeared for the most part to be a model of curtesy and efficiency. In the evening I was invited to the chairmens' party thrown by co-chairs Edd Dumbill and Sarah Novotny. This was, as is often the case, a fairly low-key affair attended by many likable people, and I was surprised to see that it was after midnight when we left. Time does indeed fly in good company.

This meant I had to hustle and get my beauty sleep as I was a small part of Friday's keynotes, announcing a well-deserved Frank Willison Memorial Award for Jesse Noller. Jesse is the classic busy person you should ask to get things done. His work rate is phenomenal, and he held down a full-time job while putting together the most amazing PyCon ever and going through a good deal of personal disturbance. I can't think of anyone who deserves it more, and the PSF Board was in full agreement that Jesse should be the recipient.

It was fascinating to spend a little time backstage and see the technology and the production values that now support OSCON. The show is professional from its head to the tip of its toes. The keynote session was followed by more hallway track and meetings while the sessions continued, then back for the closing plenary session at which Open Source Awards were handed out and the final entertainment was provided by Paul Fenwick, always worth listening to even when his topic is light.

After the formal end of the conference I had a couple of business meetings and went home (using public transport, because I live in Portland, hooray!) for a well-earned nap. Feeling much better-rested the next day I was happy to host the inaugural OSCON Survivors' Brunch at the Tabor Hill Cafe, a local place I make sure to patronize while I am at home. This was kind of an ad hoc event, attended by about a dozen of my favorite open source people, and I will repeat it in a more organized way next year.

OSCON always leaves you with plenty of food for thought, and this year I was much more encouraged about the prospects of open source entering (or even, dare I suggest, becoming) the mainstream. Mark Shuttleworth revealed OEM connections this year and claimed that next year 5% of all computers will ship with Ubuntu pre-loaded. I'd recommend buying your ticket for next year's OSCON as soon as they come on sale.

* H-P were a major OSCON sponsor this year; this was a shrewd move
** Microsoft also sponsored OSCON

[Disclosure: Steve Holden's business runs technical events and conferences,
and Steve profits from the sale of O'Reilly School Python classes]

Amazing Mozilla Annual Report Video

I've just been looking at the Mozilla 2010 Annual Report. It makes terrific reading. But most of all I really love the video that they have brought out to accompany it. To say it's inspirational to the open source world doesn't do it justice. I have rarely seen so clear an exposition of the ideals that have led me to serve the Python community and the Python Software Foundation for the last few years.

Sure, Mozilla had problems when they inherited the Netscape code base. They have matured as an organization, and are currently clearly demonstrating the meaning of release early, release often with high-quality feature releases at frequent intervals. It makes me wish I could offer such inspiration to the Python community.

Community or Fanbois?

Regular readers will know that I place a lot of stock in community. I started PyCon, the US Python community conference, which has taken off worldwide. In earlier incarnations I was chairman of the Sun UK User Group and Treasurer of DECUS UK. As chairman of the Python Software Foundation I spend a lot of time thinking about how to engender community spirit and encourage people to contribute to the Python language in any way they can. This is an ongoing battle, but I do think that the Board is showing signs of understanding how to involve people, and also that the Foundation relies on the involvement of third parties in order to achieve most of what supports its mission.

In my own life, I have recently decided to become (even) more involved in open source, hopefully to the extent that I can make a better living at it - the PSF chairmanship is an honorary position and takes time away from business matters. So I have moved to Portland (whose natives frequently refer to it by its PDX airport code), and recently hosted a reception to get PDX open source people together with elected representatives and business people with the intention of starting new conversations about how each can benefit the other. I am hoping that the move will allow me to work in a more sympathetic environment, and one in which the potential of open source is more clearly perceived.

So anyone who cares to look should be able to discern that I am at least fairly serious about Portland and its open source community, and working towards improving things for the open source community (on the theory that a rising tide lifts all boats). I have found in the past that it's generally possible to share plans with open source community members and have them respect the sensitive nature of the information you have shared with them. This is one of the things I like about working in the open source community: generally speaking (and with the occasional unavoidable exception) people are willing to respect your concerns, and are generally much more concerned about producing good software than scooping each other on news of features and the like.

The people I meet in the open source world are generally responsive to new ideas and quite willing to discuss them. Generally speaking people are both willing and able to discuss the work they are doing - after all, it i going to be published, so there is little point in secrecy. Contrast this, however, with the brouhaha that arose today about the latest version of Apple's OS X operating system. It seems that Apple shared a beta version with some developers, who have naughtily (and anonymously) disclosed what purports to be real information about what Apple rather grandiosely term "the world's most advanced operating system". To which  my reply is a snort of derision, since I think that Mac OS X is actually in some ways inferior to Windows.

It turns out that they really don't like it when engineers who are given access to a pre-release (in this case OS X Lion, the forthcoming 10.7 release) copy and then promptly spill the beans to all and sundry. But frankly these look like small beans indeed. It seems as though Apple has tweaked quite a bit, but hasn't introduced any fundamentally new features into the operating system.  It's all very well for Apple to delight in being different, but the radical differences between Apple's GUI and everyone else's just don't seem to actually make using the computer any easier, and leave me wondering whether Apple is really heading in the right direction.

Whatever else they may have going for them they certainly don't seem to have engendered a lot of loyalty in their fan base. Or maybe this was just a few bad apples (so to speak).,

Pony Outlook Fine, Say Wu and Holden

There's been some reportage about the presence of Congressman David Wu at DjangoCon this year (to which the most I could do yesterday was briefly allude). After my Labor Day post I think I can safely say that Chairman Wu's presence was reassuring to the many at DjangoCon who might have been seriously inconvenienced by the pony shortage, which has been discussed in panels at the conference.

Of course it would have been terribly unfair to the Congressman to have asked public questions, and as he is in a busy campaign I know you would not have wanted me to delay him in your name. After a breakfast meeting this morning I am happy to be able to tell you, though, that his staff have convinced me there really is no sign of a pony shortage (and I doubt many reading this will have bought more ponies than I of late). In fact, had I known that I could have procured the ponies locally (Oregon price: 50c less than manufacturer's) I might have saved the cost of carriage, too. Next time I'll just call the store in advance.

If further reassurance is needed, I would like to note that after a third meeting with Congressman Wu I am happy to endorse him as a two-pony family man. Those ponies will be in safe hands, and the people of the First District of Oregon would be ably represented by a Congressman of such integrity.

The Python MiroCommunity

Well, I have been wondering what to do with python.tv for a while, but I think it's pretty clear now that a lot of redirection to the Python Miro Community is in order. This is some seriously helpful stuff for anyone who wants to know more about Python.

If I were Steve Jobs (and I'm sure he will tell you I'm not) I might want to say it was fantastically amazingly stupendously great, but in fact it's simply the result of some solid work by an individual and a foundation.

So, a tip o' the hat to Will Kahn-Greene, the currently volunteer curator of the collection (sorry the PSF couldn't help you with funding, Will, but the amazing job you have done without it surely tells potential sponsors that your channel will represent amazing value to them) and to the Participatory Culture Foundation, Miro's originators.

Oh, and then of course there's the hundreds of people involved in creating the videos and the conferences and user group meetings at which they were made. Including a small core centered around Carl Karsten who have worked for years to get to the stage where we are publishing video while the conference is still happening. Pretty amazing stuff, really. Lots of Python in the mix, naturally, and pretty all of it open source unless I mistake my guess.

Hmmm, python.tv ...

Comments or Not? Public or Private? Relevant or Irrelevant?

Recently I heard that some people aren't happy about changes made to PyPi, the Python Package Index. As far as I am aware there is only one person who has been working on that application recently, and that's Martin von Loewis. Martin, last year's winner of the Frank Willison Award, is a Director of the PSF and someone who works tirelessly at various aspects of Python -- many of them not particularly rewarding. As with many open source activities it is (I assume) a labor of love.

Recently Martin updated the Package Index to allow users to leave comments, and it appears that this innovation has been contentious. As a result of the rumblings Martin created a poll to determine whether the feature should continue in its present form or be modified in various ways. Here are the results as at the time of this writing:

Allow ratings and comments on all packages (status quo)					223
Allow package owners to disallow comments (ratings unmodified)					137
Allow comments, but only send them to package owners (ratings unmodified)					33
Disallow comments (ratings unmodified)					24
Disallow ratings and comments (status three months ago)					88

This is all very well, but unfortunately it appears that PyPi is now boxed into a corner. Even if the most popular option is implemented (retaining the current situation, where the newly-added rating and comment feature is allowed on all packages) this guarantees that a majority of those voting will have their favored option rejected. I suppose this demonstrates that you can give people too many choices.

My own discomfort with PyPi goes rather deeper. While I think that it's great that we have a central repository to support setuptools (even though release 0.6 is now three years into its release cycle and onto its eleventh release candidate) and now distribute, I would like to see it become much more usable than it currently is. It would be easy to see this as an attack on the implementers and maintainers (which it is not intended to be: the maintainers of all the software I have mentioned have done valuable work that I could not). Honestly, it isn't.

In reality I think it would be good if they had more help. Particularly the kind of help that let them package the facilities this excellent tool PyPi provides, in a much more obvious way. Even if this means complaining about the way things currently are.

Almost as a side note, I ended up following a twisty little maze of (web) passages all alike which finally led me to the Python issue tracker. Since it showed an apparently remembered login name and password I assumed all I had to do was click the Login button and all would be well, but apparently not. So I did what any user would do, and followed the Lost your login? link.

Alas, neither my email address nor my user name was recognized on that page, so I decided the only thing I could do was to register an account (even though I know I have submitted bugs in the past). So I went through the registration process only to be presented with the following unhelpful message:


OK, so what the heck is all that about? How am I now supposed to proceed if by chance I have burning information about a bug in the Python system? I have said before, and I will say again, that as an interface between Python's users and its developers the issue tracker sucks. I am sure it is very useful for the developers, but as an input collection mechanism it seems to be only slightly less valuable than a customer service desk staffed only by a notice reading "go away" (I am exaggerating for dramatic effect here).

[Edit: this was apparently a bug, which has now been fixed at least for the Python bug tracker.]

I suspect that what is needed here is the e-mail equivalent of a help desk, where people with no knowledge of the infrastructure can exchange messages with a team of real human beings who know what the score is and can make any necessary inputs to the issue tracking system on their behalf. Call them user proxies, if you like. I am aware that in the high-tech world of open source this may be seen as a heresy, but people still have their uses, dammit, and they clamor to be useful even as the capitalist world declares them redundant by the hundreds of thousands.

Now I know before I post it that some people are not going to like this article: they will either say that I shouldn't be complaining if I'm not prepared to fix what I'm complaining about, or that I should not be making a noise about something that third parties will use as evidence that the Python world is in some sort of disarray. Frankly I don't buy either of those arguments.

The Python world has recently gone through a long-drawn-out and extremely energetic discussion about increasing the diversity of the community. As an existing community we are fighting an uphill battle, because it's even more difficult to change the constituency of existing communities than it is to recruit a diverse mix to new ones. Just the same this has had some very positive results, not least the publication of a diversity statement that I think the Python Software Foundation has every right to be proud of -- it might seem like a simple piece of text, but it was a hard-won development that even cost us the resignation of a member.

Yet despite all that work, we apparently haven't yet got to the stage where the Python community includes people who can look at a less-than-optimal interface and say out loud "we need to do something about this". I don't know if this is because we are too close to PyPi to be able to acknowledge its faults, or because people fear hostile responses if they make negative comments about the infrastructure, or (perhaps most likely) because they don't want to offend those who have invested their time and effort into producing something, at least, which is more than most of us do. I do know that it frustrates the hell out of me.

So let me put a public stake in the ground here. I have visited a number of local Python user groups in the past year (the first PSF chairman to do so, as far as I can tell). Almost everyone I have spoken to along the above lines has been enthusiastic about making things better, and willing to volunteer to help make the necessary improvements. So now I need to hear from the broader Python community about what's "wrong" (less pejoratively: what we should change to make things better). This means you.

The Python Software Foundation is currently looking quite critically at next year's activities and the budget to support those activities. If we are going to make a real difference to the perception of Python and to its adoption as a serious IT solution to a broad range of problems then we need broad involvement from the whole community, not just the PSF membership. I am investigating a number of ways in which the PSF could encourage a broader involvement, and it would be helpful at this point if there were general evidence of a desire by non-members to get more involved in Python: not just its development, but its community.

If this piece isn't enough to get a decent discussion going then I suppose I should just resign as PSF chairman and look elsewhere for a community that gives a damn. I honestly don't think that will be what I need to do. I'm really hoping you guys don't let me down here.

Community Spirit

If the Python community (or the whole open source community, for that matter) could learn to work together for a common goal there's almost nothing it couldn't achieve. Here's an example of what single-minded determination can do.

Can You Do Better?

As a graphic artist I'm a pretty good combine harvester operator, so there is no subtlety about the banner graphic I came up with for (maybe) the python.org homepage. But I was glad that Andrew Kuchling asked, because PyCon should be better known. If Python is the technical secret weapon then I believe PyCon is truly the social secret weapon. I know other projects are having good success with community conferences, and would like to have the time to visit some of our sibling foundations' events.

If you attend such an event you will be doing the Python community a service by writing it up for Planet Python. It's not hard to get your blog on the Planet if you do a few good Python posts, and we hear a lot more about the technical than the social normally.

Wow, next year it will be PyCon number eight. Three years after that will be the tenth anniversary of the very first PyCon. It would be terrific if we could go back to DC for that conference. What about it, DC metro area?

Seth Godin Says A Mouthful

You will probably be aware if you are a regular reader that while I'm not wild about marketing generally, I do see value in some things some marketers do and say. Seth Godin had a blog post recently called You Matter that describes quite well the kind of spirit I want to foster in the Python community.

As tummy.com said on their PyCon sticker this year: "Python will save the world! I don't know how, but it will." Isn't it up to us all to help Python do that?

PyCon Proves Its Worth

Here's a great quote from Catherine Devlin's blog post Five Minutes at PyCon Changes Everything reporting the unexpected recruitment of a high-powered development team after giving a lightning talk:

If I'd had $1 million of startup funding to hire a staff to work on sqlpython, I couldn't have gotten a team that large or that talented. I figure that gives me better than a 1000-to-1 return on my PyCon investment. :)

It's also been extremely gratifying to discover that the dip in numbers (initial assessment says we were maybe 10% down on 2009) represents an amazing result. Apparently many conferences have seen attendance at 50% of last year's numbers, and some have simply canceled because the cancellation fees were a smaller that their expected loss.

This really vindicates the low-cost community-based approach that PyCon has always used. Several people told me that they booked to attend the conference without knowing whether they could get company approval, because they knew the conference was great value and they could afford to attend on their own dime. PyCon is a very special conference indeed, and the Python community makes remarkable things happen there every year.

PyCon Friday: the Calm Before the Storm

Late to bed last night, and clearly too sleepy to set the alarm properly, so woke up today feeling a little groggy rather later than I had intended. Laptop behaving so far, fingers crossed.

Breakfast and the keynotes first. For the morning session the only two talks I want to hear clash with each other! PSF members' meeting at lunchtime, then chairing a session in the afternoon (a good way to make sure you get to hear some talks you want to). Trying to leave the evening open for "social business" activities.

I'm asking all the newcomers I meet what made them come to PyCon this year in an attempt to understand the explosive growth. It seems that a lot more people are starting to use Python now, vindicating my ten-year master plan to be involved with a significant technology by the end of this decade. Wonder whether it will do me any good.

Alex Martelli says I should consider writing another book, and wants me to contact a publisher. Whenever anyone says this I always remember the $2.50 royalty check I got once for Python Web Programming, and this brings me back to reality. I told him I might consider it if he'd be my technical editor again. His encyclopedic programming knowledge was terrifically useful last time.

Talking to Glyph Lefkowitz in the bar about books, he asked me how sales were going and I told him it is still in print but I am waiting for it to be remaindered (though it's hardly Books-a-Million material). He was nice enough to say that compared with other six-year-old books on web technologies there was still a lot that was relevant, which is definitely a complement. I did try hard to include explanations of the fundamentals, and I guess that counts for something.

Enough, I'm getting hungry.

PyCon is International

Various national conferences have started to use the PyCon name, and this has now been formalized by making the various national sites available under the pycon.org domain. So now you can go to:

• br.pycon.org
• eu.pycon.org
• fr.pycon.org
• it.pycon.org
• uk.pycon.org

That's a long way for a community conference to spread in five years! The eu conference is a redirection to EuroPython, which is a trans-national conference and (if I am not mistaken) was actually started the year before PyCon. But it's nice to have them in the family.

Thanks to Marc-Andre Lemburg for maintaining the domain. Let's hope to see more PyCons as the Python's influence spreads further and its popularity continues to increase.

PyCon Registration Going Great Guns

With over five weeks still to go until PyCon 2008 in Chicago registrations already number over 180, almost three-quarters of the total registrations for the first PyCon in 2003 in Washington DC. The tutorials, running for the third year, are also popular and many of them already have enough registered students to guarantee that they will run.

I'm delighted to see that my own Python 101 for Programmers is a popular offering—it's intended for people who have had little or no previous exposure to the language, and I am trying to cover all the essentials in three hours! Let's hope that it brings some new members to the Python community.

If you have never been to PyCon, I would like to recommend you try it. It's definitely not a stuffy atmosphere, people go there to have fun as well as do productive work, you can learn huge amounts from both the tutorials and the regular papers, and the sprints are a great way to get into a new project and learn from the developers themselves.

It's a unique event, besides being terrific value for money.

HOPping All Over the Place

Several regular Python bloggers have already noted the Google Highly Open Project (GHOP), which aims to introduce secondary school students to the open source world. The team managing the Python effort for the project have been pleasantly surprised by the competence of the student contributors, and they are just getting ready to add another batch of projects. If there's a project that you think would improve the Python ecology, take a look at the Python project page and get in touch with the organizers from there.

The project has just received an offer of hardware to help it along. This will broaden the scope of the activities, and the team has already started brainstorming about uses for the new kit.

So, the reason for this post is mostly to point out what an awesome job the guys who are organizing the project have been doing. Titus Brown took the lead and came up with a stupendous list of projects in extremely short order. He has been ably assisted by Andre Roberge, Doug Hellman and Georg Brandl pretty much right from the start, and an increasing number of others have joined in to suggest projects and review student submissions, offering guidance and generally helping things along.

Well done, all of you. The whole Python community owes you thanks for the amazing efforts you continue to make on their behalf on GHOP.

Jeff Rush Wins "Most Popular Video" on ShowMeDo

Congratulations to the PSF's Advocacy Coordinator, Jeff Rush. Jeff has just been awarded ShowMeDo's second (September) most popular video award for his The 'IPython' Interactive Shell - Part 1 video. Well done, Jeff, and good luck as you move on from the Coordinator position to other challenges. The Forrester report in scripting languages might have been far less favorable to Python had it not been for your efforts!

The Python Community

This page is clearly seditious, as it claims that there are activities that are "more fun" than programming in Python*.

It does start to give you some idea of the breadth of the Python community. Strange as it may seem, after 40 years working with information technology I like the Python community as a place I don't necessarily have to do technical stuff. So I have tended to organize conferences rather than present at them.

With PyCon and EuroPython, and now individual conferences all over Europe and South America starting up under the PyCon [the page is a bit out of date] banner, Python has a truly international community.

Through its various electronic media, both interactive (newsgroups/mailing lists, Wiki, IRC) and with groupings based on geography and technical interests, everyone gets a chance to contribute no matter where in the world they are. The jobs list is also worldwide.

The Python Software Foundation's Advocacy Coordinator (Jeff Rush) certainly has his hands full!

* ObGeek: Outrageous suggestion.