DRM Saga Continues: No Surprises

Well, the content protection debacle grinds on. The users continue to be unimpressed with digital rights management, and the hackers and crackers continue to break the harebrained industry schemes with monotonous regularity. Digg temporarily tried to censor material as a result of a blanket cease-and-desist threat, got it in the neck from their users, and apparently learned their lesson.

Favorite quotes from the latest discussion:

As Joe Rogan's character on Newsradio once quite accurately quipped, "Dude, you can't take something off the Internet.. that's like trying to take pee out of a swimming pool." The content providers have attempted to do exactly that, remove pee from the proverbial swimming pool that is the Internet and, as we've witnessed so many times before, they've failed miserably. [1]

and

There isn't a single known DRM system worth cracking that hasn't been cracked, multiple times; AACS will likely be no different. [2]

When will the recording and film industries learn that the "lost income" from content pillaging by dishonest consumers isn't "lost" at all? It's what the retail trade calls "wastage" (some things arrive spoiled, some get spoiled, some are stolen). Their revenues are what their revenues are, and the overall level of dishonesty is what it is. The prospect of additional billions is illusory, because people who don't pay for content will simply stop consuming it if they can't get it for nothing.

So the joke is that if the industry ever achieved their total-protection nirvana they will have shot themselves in the foot, because even bootleg copies are positive marketing. Sometimes greed is so sad. Were I a stockholder in RIAA member companies I would be furious at the waste of effort. I wonder if the senior staff of studios actually pay for the DVDs they take home. If not, aren't they too stealing from their stockholders?

I am Smalltalk

Well, apparently I am Smalltalk. According to this highly reputable quiz I found on the Internet :-)


Which Programming Language are You?

Given that I was responsible for an early UK port of Smalltalk-80 (the heavy lifting being done by Mario Wolczko) I don't particularly mind this verdict, but I don't do Smalltalk any more. It's wrong about the objectification, though.

Other People's Time

Would you rather work with a contractor who gives you a due date and sticks to it or one you could never be entirely certain of? If the project slips, how good will communications be?

Seth Godin here relates why it's important to keep public events to a schedule. Conferences aren't about the muckety-mucks, they're about the hard-working stiffs who've forked over their cash. Well, the conferences I organize are, anyway.

If a meeting's not about the participants, why are they there?

Python Acknowledged at IEEE

The May/June edition of the prestigious Computing in Science and Engineering (CiSE) magazine is a single-subject issue on the topic of Python. This is surely a recognition of the language's value to scientists, and a vindication of the work put in by scientists all over the world to apply Python to scientific tasks.

CiSE is a joint publication of the IEEE Computer Society and the American Institute of Physics. Unfortunately not all articles are available electronically to non-members.

Bet Your Business on Python

It is encouraging to see that increasingly businesses are depending on Python to give them a strategic advantage in their marketplaces. The most recent comment of note comes from yet another PyCon sponsor, ITA Software, who use Twisted quite heavily in airline scheduling applications. They are a new phenomenon to the airline business: a company that provides responsive service without the use of mainframe heavy-iron. You can read the whole eWeek article to get more context about ITA's use of Python, but the quote I liked best was from Dan Kelley, ITA's director of application integration:

"So we think, absolutely, that it's ready for prime time. What we're doing is saying to a billion-dollar business, 'Yes, we can write components in this particular programming language, and they will keep your airline running.'"

Note that ITA don't use Python exclusively, and they see one of the advantages of distributed systems as being able to choose appropriate languages for each component.. I don't believe anyone thinks Python is perfect - it's a waste of time to seek perfection in a programming language because the concept is so subjective. Just the same it's nice to see real businesses coming out with this kind of informed opinion about Python in fairly mainstream publications. Quite apart from the general advocacy benefits it is useful, among other things, as ammunition against the dynamic language bigots should any such ammunition be needed.

MSDN Webmasters: Read Peter Norvig!

Peter Norvig's an interesting chap. I mean, head of research for Google, that's got to be an interesting job, right? One of the blogs in Planet Python (I can't remember whose, unfortunately, because the "How'd I get here" button in Firefox can't tell me what Thunderbird was looking at when I clicked a link (oh, all right, three minutes research tells me it was Peter Bengtsson)) recently pointed me at a recent Norvig gem, the spelling corrector in 21 lines of Python.

I don't believe I've previously mentioned in these pages that I am dabbling with ASP.NET. Shrieks of horror resound around the halls of Holden Web. It's been interesting in the sense of the Chinese curse that dooms you to live in "interesting times". The learning curve is steep (and any regular readers of comp.lang.python will know that that metaphor has been lately subjected to the Belgian logic chopper, to nobody's very great illumination). But I digress.

One of the things I have spent a lot of time doing, given the large-ish nature of the framework I am trying to learn, is look things up in the MSDN documentation. Anyone who has corresponded with me will know that the standard of my typing is roughly the same as if you were to pay a squirrel to dance on the keyboard, which is to say I make typographical errors with a regularity that makes monotony seem interesting - the Secret Service characterize my typing style as "frequent use of backspace key". Consequently I have, in the last 24 hours, entered "sqldatasourec reference" into the MSDN search page no less than three separate times (see, when I find a way to bollix a word up I stick with it - I am only half a touch typist, so short entries not for human consumption are even more frequently wrong).

Three times I have been confronted with the helpful message "We're sorry, we couldn't find any results containing sqldatasourec reference". Now this is exceedingly strange. In trying to verify this behavior I made yet another error, entering the search string as "sqldatasource referenec", and while I saw the same(-ish) error message there was above it the helpful, dare I say Googlesque, question "Did you mean: sqldatasource reference" - which, of course, I did. Perhaps the correction corpus should be extended to include the subject material?

Just for devilment I entered the same two search phrases into Google's engine and sure enough it corrected them both. The obvious conclusion must be that Google know more about the content of MSDN than Microsoft do, but surely it wouldn't be fair to say that. Or maybe it would ...

Help Wanted!

Another post from Doug Napoleone says (among many other things) "I need to take the very clean and and elegant code of Steve Holden and stuff in into the somewhat organic and undocumented PyCon-Tech code."

It's strange how other people's perceptions of our code can be so different from our own. I am conscious of several areas where that code (published, by the way, as a part of my PyCon 2007 tutorial) really needs revision. But anyway, thanks for the tip of the hat, Doug.

Anyone wanting to help to improve the software support for PyCon should sign up for the PyCon Tech project. It would be a great way to get started in Django in a supportive environment.

Python is Not a Religion

The title of this piece is a quote from a post by Doug Napoleone on the subject of Python advocacy. Another quote: "Advocating Python for the sole purpose of promotion is idiocy." I quite agree with that one too. I don't really understand quite why one of my blog entries is referenced as ironically hypocritical, but I suppose I can live with that.

Nor am I sure who Doug is referring to as those who feel there is no need for Python advocacy. I certainly can agree, though, that there is absolutely no need for religious wars when it comes to choice of programming language, and that the appropriate question to ask is "is Python a good way to solve my problems". Anyone who sees Python as the solution to all programming problems is perceiving the language as a hammer and the problems as nails. There are many things that Python is good at (and many more it could be good at if someone wrote the right applications), but it's no more a panacea than any other language.

Doug's blog post appears to be a call for enthusiasts ("passionate people" is what he calls them) to come to the aid of the language. The sad fact is that the majority of the users of any technology (or any religion, come to that) - even users who profit vastly from its adoption - much prefer to be passive consumers than advocates, and this is the way of the world. Passion is all very well, but ultimately it needs to be directed to a goal, and shouldn't the goal of Python advocacy be helping people to solve technical problems more effectively?

While passionate people will help, I think advocacy needs planning as well as passion, and it's only when the two go together that language advocacy really helps. Support for the existing community is important, but so is increasing awareness of Python outside that community. I'm not convinced that most community members have the stomach for that task. It remains to be seen whether Python users are the best advocates for the language, but rightly or wrongly the Advocacy Coordinator position is currently planned as follows:

• User groups how-to and content resources (40%)
• Web site content showcasing Python capabilities (40%)
• User group infrasructure support (5%)
• Reactive and proactive response to queries and support requests (15%)

I am sure that the Coordinator, Jeff Rush, would love to have passionate people knocking on his door and asking how they can help him. Then he would have more time for other efforts that the PSF has asked him to back-pedal on for the moment. Ask not what Python can do for you ...

Buildbot Machines Wanted

I know there are quite a few hosting companies now who represent themselves as Python-friendly. Rather than trawl through the Wiki pages, however, and email each one separately I am sending this request to the comp.lang.python list (and putting it on my blog) in the hope of attracting those companies who are more involved with the Python community.

There is a particular problem, highlighted recently by this comment from Dennis Lee Beiber:

Too many 3rd-party modules still aren't available in 2.5 versions for my tastes...

This applies particularly (though not exclusively) to the Windows platform, for various reasons -- the most common one is that Linux developers frequently don't have a Windows machine available to help them test their builds and ensure that distributions are available.

I am trying to address this problem, initially by making hosted Windows machines available for use as buildbots. I already have agreement from Grig Gheorghiu (who maintains the PSF's buildbots) to try and support these efforts, and from Microsoft to consider providing appropriate software.

What I don't have is hosting companies offering me space on machines in their racks. If anyone reading this can help out I'd appreciate it if they would get in touch with me (replying to the newsgroup post or commenting on the blog entry should do it). These machines would need remote desktop access so they could be managed without physical presence.

The intention is to try and shorten the "version lag" so that new versions of Python can be better supported more quickly. I don't guarantee that this will happen overnight, but I'd like to make a start.

Give Me a Break, Microsoft

Well, it's going to be quite a long goodbye, but today I finally and irrevocably decided to stop buying Windows. As with most decisions of its kind this hasn't happened overnight. There have been the little annoyances, the bloat as more and more bits and pieces clamor for run time and screen real-estate, the ever-increasing startup time, but what's finally pushed me over the top is the Windows Genuine Advantage (or, as I like to think of it, the Make Microsoft Richer) program. I've been saying for over ten years now that Microsoft will become "the first IBM of the twenty-first century" and this program, coupled with the digital rights management (DRM) approach embodied in Windows Vista, has finally made me realize I am sick of this snake oil.

Windows Genuine Advantage has been bugging me for a while, sitting there as an uninstalled update. Eventually, as it was designed to, the continued notification on restart became annoying enough that I decided "what the hell, this Windows came from Dell, why don't I just install it and stop this annoyance". So I did. Right at the end there was a check box that said "when I click finish show me the benefits of Windows Advantage" or some such, and rather than uncheck it I thought I'd see if Microsoft could persuade me that this effort had in some trivial way been worthwhile. Unfortunately the justifications seem mostly to be bogus in the extreme.

First example: "A recent report from the market research firm IDC found that, if the global software piracy rate was lowered just 10 percentage points over the next 4 years, this would contribute a total of 2.4 million new jobs and $400 billion in economic growth to the global economy." This, accompanied by a link (that would have opened several popups had Firefox been dumb enough to let it) to the Business Software Alliance web site and an IDC white paper on "Expanding the Frontiers of Our Digital Future - Reducing Software Piracy to Accelerate Global IT Benefits". For some reason they go to great lengths to hide the URL of that white paper, and there appear to be a family of these documents, each tailored to their specific audiences, so the link I've included might not be "your" content if you come from somewhere where the prinicpal economic impact of IT has been to reduce the amount that can be spent on things like clean drinking water.

On page 4 we see, under the heading Who Wins and Why the point that "Governments benefit from $67 billion in new tax revenues for needed services that could [according to a unstated OECD cost estimates for government services] be used to provide:

• 33 million computers for schools

• 45 million people with health care

• 6.6 million people with college educations

• 11 million children with schooling

• 435 million people with job training, or

• 132 million families with services like day care, maternity, or home help services
  

The snag here, of course, is in that little word could, since the same revenues could also be used to prosecute America's war in Iraq for a further eight months. When I ask myself which is more likely I don't conclude that Republican crocodile tears about "big government" will suddenly cause a sea-change in US political behavior.

On page 8 the authors try to convince us that "Lower Software Piracy Produces Higher IT Benefits". This section is accompanied by a wonderful graphic which I reproduce here as a classic example of how correlation is frequently taken to be causality. And yet I can find nothing anywhere in the paper that suggests the graphic might not equally well be entitled "Economic Growth Starts with High Piracy Rates" or "Poor People Steal Software Rather Than Do Without". The one thing IDC's methodology did not do was to look at single countries' change in "IT tax benefits" as the piracy rate changed, which it seems to me would be the only convincing way to demonstrate the benefits of such a change.

But I digress. In their discussions of the Genuine Software Initiative, Microsoft explain that downloading illegally-obtained software increases the chances of infection by malware and identity theft. Clearly these are things we would all like to avoid (and for my part it's why I tend not to buy anything from dodgy characters down back streets). They go on to say "In addition ... installing and using counterfeit software can prevent customers from obtaining some updates and premium add-ons." In other words, Microsoft will punish you still further for acquiring illegal software by refusing to provide updates to non-genuine copies. This seems fair enough, and it seems to me is the real thrust of Microsoft's policies: they don't care about your possible malware and identity theft issues (otherwise they would make their own products more secure), they care about whether they receive the revenue or not. The rest is just hokum designed to make people fearful about software theft, masquerading as a concern for the consumer.

I have to admit that Microsoft's update also bugged me by creating an automatic reboot event. This might not normally have been too troublesome (I had deferred it several times, and can only presume that the window appeared and consumed a random newline to gain permission to restart the system) had it not occurred during the installation of software. Aarrgghh!

The thing that really makes me want to run away from Windows, however, is Vista and its draconian approach to product registration and DRM issues. On the latter topic it's as though Microsoft were already a fully-fledged member of the Hollywood hegemony (which might give us clues about its media ambitions: I am sure that Microsoft is clever enough to sense the diminishing returns from software production). This paper by Peter Gutman enumerates some of the many ways in which DRM paranoia has caused Microsoft to choose to degrade content presentation and disable functionality that many users will desire.

With all the brains that Microsoft has at its disposal (and despite my occasional scathing criticism of the company and its products Microsoft does employ many very clever people), the best approach they could come up with for securing content in the 64-bit Vista environment was an insistence that drivers be digitally signed and approved by Microsoft. Fortunately it took about a month for NV Labs in India to design a system that completely subverted these protections. It's clear that Microsoft, like Hollywood, just doesn't understand that copy protection schemes are a busted flush. But they still continue to wrong-headedly pursue them because they know the majority of consumers will accept the technology unmodified and not even realize that in many legislations (i.e. those not subject to provisions similar to those of the USA's stupid Digital Millennium Copyright Act) their legal rights to fair-use copying are infringed by such systems.

One quite amusing result of Windows' present-day incorporation of DRM features is that in five or ten years' time Microsoft will have the content producers by the nuts. I can't say I'm sorry about that one. Microsoft like to paint themselves as subject to Hollywood's whims, but they are actually very cleverly positioning themselves to control the distribution channel. This would allow Microsoft to say "if you don't like our deal, tough, because it's the only game in town". This would be true if they weren't so busy throwing the desktop away. People are going to be really pissed when they find that they can't play their HD content on Vista, or that they have to repurchase after a disk crash.

The only systems I have left using Windows are laptops, and it's becoming increasingly easy to buy a laptop without a Windows you never plan to use. Unfortunately this won't stop me having to deal with people who can only provide me with low-resolution content because they can only talk to me over what Microsoft regard as an insecure (by which they mean non-protectable) channel. Microsoft say:

Digital distribution offers consumers a convenient way to access their favorite content at any time, and with content protected with Windows Media DRM 10 they'll have even greater flexibility and choice. Today consumers can choose from a variety of content service providers, a multitude of devices, and a variety of "purchase and download" payment options and subscription models. Windows Media DRM 10 ensures that consumers will be able to enjoy even greater flexibility and choice by allowing them to acquire and/or transfer their subscription content to the devices of their choosing.

I say this is complete tosh. The benefits of DRM are for the content owners and content providers, and Windows DRM allows them to provide copy-protected content that they can disable simply by adding it to a Certificate Revocation List. The "greater flexibiity and choice" is specious: Microsoft assume that content providers would choose not to make their content available digitally without DRM schemes, but in fact there has been no effective demonstration of this assertion. But I am used to corporate America pretending that their profit plans are really for my benefit.

Gutman says at the start of his paper "The Vista Content Protection specification could very well constitute the longest suicide note in history". It certainly puts them in the running to become a shadow of their former selves. Notice, by the way, that while IBM is also a shadow of its former self it is still a significant organization.It just no longer has the 75% of the world computer market that it used to. When Microsoft looks back thirty years hence I hope it will realize that this obsession with DRM was what lost it the desktop dominance that fueled its growth into the largest software company in the world. I just hope they still sponsor PyCon.

Just Another Dynamic Language Weenie?

I was mildly amused by a recent post on the Hacknot site. The author is clearly completely in love with himself (if this piece was written by a woman I'll buy a hat and eat it), and cannot even be bothered to get Stephan Diebel's name right. I really don't see the point of trying to put dynamic languages down - they exist, and those who don't want to use them don't have to. This is yet another boring case of some idiot fighting a religious war of their own imagining, and unfortunately it doesn't seem to contain much that usefully adds to the static vs. dynamic language debate.

About the most egregious example of false logic to be found is in the attempt to put down people who say "where's the harm - [dynamic languages] are just another item in your tool box, for use where appropriate". The author attempts to persuade us this argument is untenable because, he says, "... the same milksop argument is used by advocates of intelligent design ...". Because an argument is used irrationally by one misguided group of people doesn't make it any less valid in cases where it actually applies, however.

Overall I think the author is just looking for a group to refer to as weenies in a feeble attempt to prove (possibly to himself) that "I'm OK because you're not OK". The blog is output-only, with no apparent mechanism for reader feedback. Probably the author isn't interested in other people's opinions.

Science and Faith

While this post is probably a tiny bit cynical I think it addresses the behavior of the worst fundamendalists, choose any denomination you like.

Impressions of PyCon

Well, since a laptop fault meant I was without a computer for PyCon I wasn't able to do the instant blogging I did last year - sorry if you were expecting that. Instead I decided I'd try an experiment and turn my photo collection into a video on YouTube - too late to enter the photo competition anyway, so now for something completely different. Hope you like it.

Tutorial Is Ready

I wanted to make sure that the tutorial materials were available on the web before I started this year. Sean Reifschneider established network connnectivity earlier this evening, meaning that those students who can't read a thumb drive will be able to download the slides and sample code.

So can anyone else, come to that. I hope that the materials will be helpful to anyone who wants to learn to use databases from Python.

At PyCon!

I thought I was going to have a difficult journey when the airline announced that the flight was delayed for engineering reasons. Fortunately the delay was minimal and my flight to Fort Worth from Dulles actually arrived slightly ahead of time!

I met up with Andrew Kuchling, who's done a terrific job of organizing the last two PyCons (with able assistance from his co-chair Jeff Rush) and we shared a cab to the hotel. After we checked in we spoke to the sales staff about the accommodation issue. It now sounds like the relocation that some delegates have been forced to into should be for Thursday night only. This will mean a move after their first night, but at least they'll be able to spend the rest of the conference in the same hotel as everyone else.

I have to put the finishing touches to the tutorial presentation for tomorrow, and then we'll see how the rest of the evening goes.

Brown Bears Threatened by Builders

My son has just sent me details of a development in Northern Spain that threatens the brown bear population. Apparently one of the greatest problems suffered by the bears is insularization - separation of the habitat areas by removal of the corridors between them by building development. Apparently this particular development proposes a ski resort be built in an area of low snowfall across a migration corridor used by a threatened population of bears.

This is in contravention of EU law, in particular NATURA 2000. Clearly someone's narrow economic benefit is being put before the law, let alone the welfare of the bear population. If you value species diversity and want to help please sign this petition and request that the Spanish Minister for the environment enforces European law to preserve a threatened population in preference to a development of marginal utility.

Are Viacom Threatening You?

Have you received a notice from Viacom demanding that your remove YouTube content? Apparently they sent 100,000 take down notices about YouTube content in a single day.

Obviously each of these notices would be meticulously researched, which is why some of them were completely unconnected to the corporation or its content. If you received a notice you feel is unfair and you need help you can contact the Electronic Freedom Foundation for help.

It's time that media giants stopped abusing their position. Once they do that we can address the issues raised by their blatant manipulation of the political process in their favor. If they had their way there'd be no fair uses at all for their content other than paying for it before any use at all.

Five Years as a Blogger Tomorrow

I just read via Unofficial Planet Python fuzzyman wishing his Voidsapce blog a happy fourth birthday - congratulations, fuzz. This caused me to dig back into my own archives and realise that "... for Some Value of Magic" will be five years old tomorrow. So if four years is "practically ancient" I suppose this blog must go back into pre-history.

Why couldn't this wait until tomorrow? Because that'll be my last full day in the UK for some time - we are moving back to Virginia and I very much doubt I'll have the time to be writing blog entries as we have to get the cats to the airport and various other mundane but necessary tasks. I'm not looking forward to the joys of a 5:50 am flight out on Tuesday, but can't help being excited at the prospect of this return.

The Purpose of Advertising

A recent Seth Godin post about changes in the way advertising has been changing over the last quarter-century ends with an interesting quote:

"The winners, I think, are the agencies and the pundits and those that would like advertising to be more than it actually is."

He makes the point that advertisers are more interested in creating a short little movie that spreads - which if you think about it is exactly what everyone on YouTube is trying to create.

Bloggers are trying to create compelling writing that holds the interest of a (probably quite select) group of readers. Is this the same thing? Not really, in my view. Most bloggers aren't doing it for the money, but to promote their opinions and share their knowledge, but there's still the desire to promote a particular point of view - even if successful promotion doesn't result in the reader buying something.

While there's always a chance that a successful blog will attract enough readership to make advertising a money-earning proposition most blogs aren't going to be popular enough to attract advertisers. Yet still their dedicated authors keep pushing out the material. Just like I do, I suppose.

For some reason a number of posts were stuck in the blog as drafts. This one is from February 2007.

So You Want To Help the World?

When I was an academic, one of the problems I struggled with was relating information technology to the basic needs of populations in less-developed areas. In my M.Sc. classes I tried to explain the irrelevance of high-tech information systems to someone who was struggling to find clean drinking water and sufficient food. These explanations were somewhat academic (no pun intended) in nature because I haven't visited such areas or even been in particularly close contact with them.

If you would like to help others less fortunate than yourself PyCon gives you a great opportunity. I have been corresponding with Oouc, who says:

Thank you greatly for your interest. People in the disaster area, especially Nias, have had their old way of life destroyed. Many are now beggars because their type of employment before the tsunami and earthquake 3 months and 2 days later is gone. The March earthquake destroyed 80% of the buildings and killed many of the business leaders because they could afford to live in large heavy concrete houses.

Their primary need is jobs. But before the tsunami/earthquake the great majority were poorly educated tribal language only gatherers in the cocanut and rubber tree plantations. They need practical education in the national or international language if they are going to have a real life after all the results of their old way of life has been destroyed.

The Internet has a wealth of Public Domain materials which could help accomplish this education but it is not accessible to these people. I have an expenses only, no salary, grant to grab these Public Domain materials, put them on a local server and LiveCD's and make them available to Nias and Aceh residents in the original language and in translations to their national and tribal language. Decades ago, I was called "the impossible man" by a translations society because they were repeatedly told by computer scientists that something was impossible but I never failed to do what they wanted. Now most of my time is spent helping others and I cannot even do what I want to do. The sprinting or distant programming help of any Python programmer who can develop almost anything for education would be helpful.

I joined a sprint on a dial-up connection half way around the world last year and was able to benefit much more than my small contribution.

People who want to help should look at http://us.pycon.org/TX2007/EducationalSprint and the two links on that page. If they will give me their name or handle, abilities and interests; then eMail to me or leave their contact info on the wiki then I will plan to use their abilities and turn the sprint more towards their interests.

The tsunamis have left large communities without visible means of support: lines of communication have been cut, buildings have been destroyed, coastlines have changed (sometimes by over 50m/160 ft) and livings have disappeared. If you would like to help people who have been thrown into turmoil through no fault of their own then consider signing up for the sprint, or getting in touch with Oouc at the conference (as I intend to) and offering your help.

Lots of PyCon Good News

The PyCon blog announced today that pre-registered attendance has now reached 400. After a pause in growth last year (due to the change of venue?) I am happy to see that Python's continued increase in popularity is being reflected in PyCon's attendance figures. It is, after all, the primary US conference for anyone with an interest in the Python language.

Sponsorship is also on the increase, and I am delighted to have exceeded the target that the co-chairs set me when they put the budget together. This is the first year we have attracted five Platinum-level sponsors. Thanks to everyone who is supporting the conference in this most practical of ways. I hope all my readers will take the time to follow their links at the right-hand side of the PyCon web pages. It's also not too late to sign up as a sponsor - where else could you get your organisation's name on the web next to both Google and Microsoft?! The details are here.

On a personal note the database API tutorial is going ahead. The lower numbers this year should allow time for questions and personal tuition. I'll have to get working on the slides and examples now!

Things You Don't Expect

It's strange. I use the web an awful lot (the company is called Holden Web, after all) and so I am used to seeing server error messages appear in my browser. So I had to stop and wonder just why I found this one so fascinating:

The answer finally came to me after a minute's puzzling: it's the very first Google server error I have ever seen. Given the amount I use Google and the scale of the service they now provide I think that says a huge amount about the quality of their engineering.

Python Everywhere ...

In other news, a Swiss company that sells remote sensing systems for liquid petroleum gas tanks is buying Python-based sensing equipment from a US-based company. This is probably nothing new, as the Python in question is 1.5.2, but it reminds me just how pervasive Python is becoming.

The device in question is 1.62" x 1.24" x 0.14" (41.4 x 31.4 x 3.6 mm if you must) and it runs Python!

The news about Autodek's adoption of Python for Maya now also seems to be getting around - three more news pieces listed in my Google alerts today.

Python 3.0 Alarmism: Pay No Attention

I don't follow the python-dev list religiously, but I do like to keep my eye on (what seem to me) the more important threads. Recently there's been a "Proposed 3.0 compatibilty module" thread in which various luminaries have discussed how migration to the new version might be made easier for current users. The general conclusion, now the dust has settled, seems to be that no compromise is to be made on the 3.0 design, that it will clearly break compatibility with 2.5 (no surprise there), and that 2.6 should contain enough 3.0 features that onward migration with the help of an automated translation tool should not be a problem.

In an earlier thread on the proposed bytes type, I expressed to Guido my feeling that it's important the development of 3.0 doesn't fragment the community in the way suggested by theand asked him whether there were plans to discuss 3.0 with implementation teams from the versions other than CPython.

Yesterday there were some rather alarmist comments by Calvin Spealman, suggesting it's at least possible that the sky is falling and that forks are coming. One quote from Spealman's blog is revealing: "What few ideas I do have towards avoiding the problems are not well thought out, probably are missing vital information, or are simply stupid." You said it, Calvin, and I'm not convinced that your approach was the most helpful one you could have taken. Fortunately our technical blogs don't usually make the front pages of the Washington Times - I have been known to suffer from foot-in-mouth myself.

Now I see that Chas Whitacre has also read Spealman's blog and is looking for advice in whether to adopt IronPython. He says "So what would really help me would be posts demonstrating the respective market shares of the various Python implementations, and resources detailing the differences between them". This is a reasonable request, but I have a nasty feeling that these two blog posts between them will spark alarm, that those with an axe to grind will jump in gleefully, and that a shitstorm will result.

Guido replied off-list, so all I can say is that he has confirmed that the upcoming PyCon represents a great opportunity to address these issues. In fact, though, the situation today is already somewhat fragmented. Jython support appears to lag at least a release-and-a-half behind and IronPython and PyPy are currently (as far as I know) still targeting 2.4. Just the same they are all implementing Python, and that's the important fact to keep in mind.

I believe it's necessary to try to ensure that all implementations are encouraged to stay in step with the reference implementation (which at the moment remains CPython). Even if this proves difficult and they don't all make it to the promised land of 3.0 their users will still be better off running 2.3, or 2.4, or 2.5 than using other, less useful, languages. Microsoft managed to break compatibility on Visual Basic without the world coming to an end, and the user community involved there was enormous, so let's not pretend that this is any more than a storm in a teacup.

Of course the information that Chad Whitacre asked for would be both interesting and useful, so if anyone has information to share (preferably facts rather than opinions, but I suspect few can be discovered) I would love to see it posted, here or elsewhere. One fact I can give you is that Python's popularity continues to grow. As Python is now integrated into most popular Linux/*nix distributions (and indeed scripts many of their installation routines) a much larger percentage of the downloads are for Windows than was the case a year or two ago. H-P now installs Python on their Windows PCs to help script their installs, so the movement towards world domination continues. You can even run Python on the Nokia 700 web tablet. This is a fundamentally healthy situation.

Is Python becoming a standard for animation application scripting interfaces? Autodesk, originally well-known for their AutoCAD drafting software have diversified into animation with a product called Maya, which has just been upgraded to version 8.5. One of the new features trumpeted in a report of the announcement is Python scripting, which turns out to be very popular with the film industry.

Anders Langlands, R&D lead at The Moving Picture Co. (MPC) and Maya 8.5 beta tester, commented: "Having Python support available in Autodesk Maya means we can leverage many of our existing tools directly within Maya, rather than having to write glue code to bind Maya to our pipeline. This allows us to develop new node and command plug-ins in a fraction of the time it would normally take using other solutions."

I already knew that Python was popular with companies like ILM and Pixar, but it seems as though its value in animation is now an open secret.

No Database Tutorial?

Well, from the reports so far it looks as though I won't be giving my Using a Database in Python tutorial. It looks like the PyCon market for introductory database tutorials was saturated last year and there may be too few signups for this year. Unless there's a late surge it looks like it will be cancelled.

So, if you were planning to sign up later you should swell the numbers by doing it now. That would be a good idea anyway, as Early Bird registration is still in force for a few more hours - until the end of Monday! In fact, forget the tutorial -- if you want to go to PyCon then register now! It has to be the most cost-effective Python conference on the calendar -- and I'm not just saying that because of my own affiliations with the conference.

PyCon Sprints: Hotel Misinformation

Just in case you have been told, as a couple of people have already reported, that there are no rooms available at the conference hotel for the sprint days, please don't give up! This appears to be a glitch in the hotel's reservation system.

To the best of my knowledge the agreement with the hotel is that rooms will be bookable at the conference rate even after the PyCon guaranteed allocation is used up, and indeed as long as rooms continue to be available. So persist, and if necessary call 1-800-228-9290 to demand the room rates that PyCon delegates are entitled to.

Thanks to David Goodger for the appropriate extract from the contract. I wonder if Marriott need some help with their information systems? I bet they aren't written in Python!

Questions for Python Developers

Those who aren't heavily involved with the Python world may have missed the fact that PyCon 2007 will be held towards the end of February. There are already signs that the hotel is starting to fill up and early bird registration ends in less than two weeks, so if you are planning to attend now would be a good time to register.

Besides the database tutorial I am presenting on the Thursday before the conference proper, I am also acting as moderator for a panel where Python developers will answer questions, some of which will come from the audience. Is there anything you'd like to know about how Python is developed, what the general philosophy is, or what it takes to become a committer? The broader the scope of the discussions, the more interesting this session is likely to be. If you would like to add your own question you can do so on the PyCon web site, and I will be happy to include it.

If you are so technophobic that you aren't comfortable editing a wiki page then you can add a comment on this blog, but since that would involve clerical work on my part I can't guarantee that channel will operate as effectively.

It's interesting to see what an organised conference PyCon has become under Andrew Kuchling and Jeff Rush, this year's joint chairs. I still remember wondering whether the first one would cover its costs after I had casually predicted that the community could run a better conference than the commercial organisers who had run the IPC series before it. So far each PyCon has made positive contributions to the Python Software Foundation's coffers, thanks largely to the generous sponsorships we have attracted.

Since I am responsible for administering the sponsorships, and since there are a number of people whom I need to correspond with quite urgently I had beter get back to that task. If your organisation would like to help by sponsoring the conference, please visit this page or drop me a line.

The SuperVillain Test

Happy 2007, everyone!
Oh, no, what a way to start a new year. I am Lex Luthor

Lex Luthor	62%
Dr. Doom	57%
Kingpin	55%
Green Goblin	52%
Mr. Freeze	46%
The Joker	39%
Apocalypse	34%
Riddler	33%
Poison Ivy	30%
Catwoman	29%
Mystique	28%
Magneto	27%
Venom	21%
Dark Phoenix	12%
Juggernaut	12%
Two-Face	12%

A brilliant businessman on a quest for world domination and the self-proclaimed greatest criminal mind of our time!

Click here to take the "Which Super Villain are you?" quiz...

Then again, looking at the other options maybe Lex Luthor isn't so bad ... at least the guy has some charisma. Overall I suspect I'm just not really supervillain material.

The SuperHero Test

I saw this on Allessandro Iob's blog. I too am Iron Man. Maybe I should start using IronPython?


Your results:
You are Iron Man

Iron Man 80% Spider-Man 80% Robin 72% Hulk 70% Superman 65% The Flash 60% Supergirl 55% Catwoman 50% Wonder Woman 40% Green Lantern 40% Batman 35%

Inventor. Businessman. Genius.

Click here to take the "Which Superhero am I?" quiz...

W'appen?

I see Blogger have released their beta developments into production. I can't help wondering if that's why my previous posting doesn't seem to have made it through RSS feeds and into the places it usually appears. It's not every day something so noteworthy happens.

Noteworthy?

Well, thanks to coralpoetry for pointing out this has become a blog of note, presumably the last of 2006. I suppose this might be more humbling if there were published criteria for what makes a blog noteworthy, or if I had been aware of this feature before it linked to ... for Some Value of Magic. Another ten seconds of my fifteen minutes of fame?

Acknowledgements of any sort are encouraging, though. Like the old saying about bad publicity, for a minor blog there's no such thing as unwelcome atttention! So many thanks to whoever at Blogger (Google) put me on that list and to anyone else who happens to read this blog.

Because of the linkage I suspect that a small spike in readership is likely to occur, so let me take this opportunity to wish all fellow-members of the blogosphere (dreadful word) and all Holden Web customers a happy and prosperous new year. 2006 (and especially the latter part of it) has been somewhat traumatic for various reasons, so here's hoping for a smoother 2007.

Finally, while I have your attention, let me point you to Pass IT On, my favourite charity. Martha and Sandy's dedication to their goal of providing computer equipment to the disabled is single-minded, and I wish I could do more to help them. If Santa Claus has brought you a new computer please consider donating your old one. They also need cash support to help them become a free-standing organisation this year, so you can give them money if you don't have a computer to spare. Good luck to all at Pass IT On.

ZFS: "Probably more abuse in 20 seconds than you'd see in a lifetime"

I've been reading (thanks to Brett Cannon) about the Zettabyte File System (ZFS) - an open source project from Sun Microsystems that seems to offer real opportunities to take proper advantage of both SAN and virtualization technologies. If you want to know what a zettabyte is then as usual the Wikipedia is reliable on technical matters (and no, I had never run across the term before).

Brett was simply happy that Apple announced the inclusion of ZFS into the Leopard (OS X 10.5) distribution. I was interested because Sun developers were taking an interest in advanced filesystems even back when I used to work there - they produced the Translucent File System (TFS), which was a copy-on-write mechanism to allow users to share a common source pool and keep their own changes.

One of the problems with TFS was its need for kernel integration (in those days even the windowing system was integrated into the kernel). Another issue arose when a user base their TFS on another TFS which was based on a third TFS which ... effectively it implemented an copy-on-write inheritance mechanism, but efficiency could drop quite rapidly with multiple layers.

Unlike TFS, ZFS can run in either kernel or user mode, and the nightly testing appears to twist the filesystem in every possible direction simultaneously (hence the title of this blog entry). Here's a list of what goes on in parallel in each nightly test:

? Read, write, create, and delete files and directories
? Create and destroy entire filesystems and storage pools
? Turn compression on and off (while filesystem is active)
? Change checksum algorithm (while filesystem is active)
? Add and remove devices (while pool is active)
? Change I/O caching and scheduling policies (while pool is active)
? Scribble random garbage on one side of live mirror to test self-healing data
? Force violent crashes to simulate power loss, then verify pool integrity

So, it would appear that ZFS is a worthy successor to TFS and much more besides, as you will find out if you too read the slide show. I haven't looked at the license yet, but it does appear that under Johnathan Schwartz's direction Sun might be beginning to find its way in the open source world of the 21st century, despite the amusing graphic that Eric Bangeman produced when Scott McNealy resigned. Maybe I should buy some stock again. Maybe we all should.

Lies, Damned Lies and Baggage Claim Call Centres

I travel quite a lot, and up until now I have been lucky. Bags have been lost from time to time, but they have always made their way back to me, usually within 24 hours. I understand that bags can get lost occasionally, but the frustrating thing last time this happened was the complete absence of any reliable information.

I flew from Edinburgh to London Heathrow, then onwards to Washington Dulles where I was incorrectly instructed to enter the USA through the transit lounge. Since my flight to New York's La Guardia was on a separate reservation and my bag had only been checked through to Washington this put me in the wrong place to reclaim it, and trriggered what I would call a comedy of errors if there was anything the least bit funny about it. My bag and I were both in Dulles, but United were unable to reunite us. I was initially assured that the bag would join me on the flight to La Guardia but, inevitably, I arrived and it didn't.

So I registered a claim and went to my accommodation. It took forty-eight hours and ten 'phone calls before I was finally able to wear the clothes I had brought with me. For most of that time United's web site was telling me that the bag had been found and that it would be sent to New York on a non-existent flight shortly before I landed at Washington!

Talking to other travellers, the general impression is that baggage "help" lines are uniformly bad. My own experience was certainly terrible: I could easily have believe that there was no computer-based information system at all behind the scenes, as the different people I spoke to gave me completely different and inconsistent stories each time I called. The responses almost seemed calculated to inflame. I was particularly incensed by a conversation with one woman who, when I called back as requested after waiting two hours, insisted that I had only called thirty minutes previously, and that I should wait until next morning before calling again. It's not usually considered good customer service to call the customer a liar. The different stories I was told about my luggage might as well have been chosen at random.

Under these circumstances the airlines would do better to dispense with the call centres completely and replace them with a recording saying "Our staff are working to restore your lost bag. Thank you for your patience". This would be less irritating than the nonsense I had to put up with, would have taken up far less of my time, and would hopefully reduce costs and allow the airlines to invest a bit more in relocating and delivering lost bags.

And now, if you'll excuse me, I have to send a claim for reimbursement to United.

Blogger on a Nokia 770

Well, I find myself with only a Nokia 770 Internet tablet to access the 'Net, so I thought I'd see how easy it was to create a blog entry. The typing is less than terrific through the hunt-and-peck on-screen keyboard, but so far I haven't been able to get my snazzy laser-projection VKB virtual keyboard to work reliably with the Nokia, so I have to be satisfied with the standard mechanism..

One rather strange feature of the interface seen through the Nokia is the invisibility of the title field. I see in the postings list that I've somehow managed to call it JU, but I really can't see how to change it. Maybe I'll change it later.

I seem to remember that the AJAX functionality used to work, but that's not the case since I upgraded the operating system. Maybe there's some setting that I need to update, but if so it eludes me. Anyway, this has taken quite long enough so I'll save the post as a draft and put the title right before I post it.

Browser Statistics (Hello, Robots)

During the current statistics trawl I started to wonder which browser technologies dominated among the visitors to the Holden Web site. November was an interesting month because both Microsoft and the Mozilla team released new products.

Here is typical information for the top ten client programs. Unlike previous statistics, Firefox users account for almost a half of the traffic to the site - usually Internet Explorer leads the pack with about 30%, but in November it was relegated to third position.



Internet Explorer 7.0 currently accounts for 2% of traffic, the first time it's appeared in the stats. The thing I found most surprising was the fact that more Firefox users were using the recently-released 2.0 version than anything else. 2% of users still use Firefox 1.0.

Rather less delightful is the fact that 20% of site traffic is spidering (though how come Yahoo has to crawl so much more aggressively than anybody else is beyond me). Perhaps the real message is that I need to put some more compelling content on the site!

A Tip o' the Hat to Ron Stephens

Looking at my web site statistics the other day I discovered, as you might expect, that the majority of referrals were coming from Google. However the notable second source is Ron Stephens' Awaretek site. Ron has been quietly generating Python reference materials and producing a Python podcast for years now, with vary little in the way of public acknowledgement. So thanks, Ron, keep up the good work!

Database API Tutorial at PyCon TX 2007

The list of PyCon tutorial offerings has now been finalised, and I'm happy to say that my Database API proposal has been accepted. Once again there is a great line-up of tutorials. It must have been hard for the program committee to choose this year's line-up since most of the offerings were strong and from well-respected speakers.

This doesn't guarantee that any tutorial will run: there have to be enough sign-ups for that to happen. This year there is also a tutorial on SQLAlchemy, and it looks as though the schedule will allow people wnating to understand how to use databases in Python to go through the DB API in the morning and then learn SQL Alchemy in the afternoon. I'll be conferring with Jonathan Ellis to make sure that I cover any necessary background for those who want to attend both.

I'm hoping that my tutorial will be well-received — last year's filled the room, and helped me fund my attendance at PyCon. It's always good to have guidance from students in advance telling me what their "hot-button" items are. If you're thinking of attending you can use this page to provide guidance on the content you'd like to see.

A False Sense of Security?

Although I am primarily a technical person I have broader interests than just that, including business and marketing. Sometimes these interests intersect in areas that affect a broader cross-section of the population. This is not a technical post.

One of my sources of occasional reading is Seth Godin, author of several books including All Marketers are Liars. He also runs a personal blog, and in the wake of the Thanksgiving rush he wrote this TSA-inspired post about the stupidity of the air transport security rules and the insensitive way in which they are administered. This really rang bells with me.

Godin courageously risks being thought a whiner by ridiculing the rules that allow a full three-ounce container of gel through the screening but disallow a five-ounce container with only one ounce in it. I've spent a lot of time in US airport security lines, and people aren't generally welcoming of critical remarks. Unfortunately many of the "security" activities that we have to go through are what Bruce Schneier calls security theatre.

At the same time as the TSA are forcing even the airline staff to submit to security screenings there are airports where construction staff are allowed airside simply by swiping an identification card. So the terrorists will be targeting the construction workers, not trying to smuggle six ounces of gel through the passenger channels.

Now I know I'm not the only person to suggest that the US government's initial response to the events of 9/11 was off-target. Benjamin Franklin was spot on in saying "Those who sacrifice freedom for security deserve neither". It seems that the majority of American citizens appear to believe that the government has duty to protect them from all risk from the cradle to the grave.

Said citizens, of course, are oblivious to the fact that it is impossible to eliminate risk completely: the aim of security is to raise the cost of a breach to a level that makes the breach uneconomic. But enough is enough. Dulles International has got to the stage where "going through security" can add over an hour to the passenger processing time, and I'm sure others are as bad if not worse. If things go on the way they are, people will stop flying because of the security measures.

What percentage of the American population use the air transport system? How much is being spent every day on airport security theatre, and how many lives is it saving? How much should we be prepared to spend to save a single life? How quickly could we have fresh drinking water available for all of humanity if we spent the money towards that goal instead? How many lives would that save? Which expenditure would make the USA more secure? Isn't it perhaps time for a reassessment of priorities?

Movie Moguls Need Control

In yet another example of movie mogul insanity the MPAA, in the guise of Paramount Pictures, is suing a small business that is ripping DVDs to iPods for customers who buy both at the same time. This highlights exactly why the Digital Millenium Copywrong Act was a bad idea, and it shows American politics up exactly at its worst: as a way for wealthy influence groups to get their own way at the expense of the buying public who have made them rich.

Don't even get me started on software patents.

Corporate governance always makes such a huge fuss about how corporations are really responsible to their stockholders. Who are the stockholders of the MPAA members, and why aren't they telling their managements to start respecting the customers who earn them their living?

Thunderbird QA Not Slipping!

Well, for my sins I have become a Thunderbird 3.0a1 user, and I can't say I'm happy. Several times the program has threatened to update itself, and eventually the promised (threatened?) upgrade arrived. [NOTE: apparently the 3-series builds are intended only for developers].

The error that originally caused me to update to the latest version is even worse in 3.0a1. Message contents are now being mangled on a regular basis. I have no idea whether they can be recovered correctly or not, and I am beginniing to lose confidence in what has until now been a reasonably reliable piece of software. It's ironic that I am suffering this problem even worse than I was because I wanted to help the project by making an accurate bug report.

If many other users are suffering the same problems I am then this issue needs to be addressed before it starts to give open source a bad name.

Thunderbird: Can't Help for Hurting

I've been an enthusiastic Thunderbird user for a fairly long time, and have only noticed odd quirks from time to time. One such quirk is the fact that occasionally a newsgroup article will show completely the wrong content: like the content of some other article altogether. This happened again today, and my first thought was to report the bug.

Now I've reported bugs in the past, and been told that they were duplicates of existing reports. So this time I decided I'd do the right thing and follow the reporting protocol. After all the guys who put Thunderbird together deserve some consideration, and I'm sure many users are cluelessly inconsiderate of the fact they are one amongst milliions and that they haven't paid a cent for some pretty high-quality software. I can't tell whether there is any description of such a bug in Bugzilla -- it might be bug 239665 or it might not.

It turns out that one if the things you are supposed to do is install the latest nightly download in a blank folder with a new profile and see if the same error occurs. This is not a prospect I relish as it will involve a certain amount of setup, but I do the download (nothing much else to do in my hotel room) and start the install. Unfortunately it turns out there's an error in the install, and I see a dialog box showing

Microsoft Visual C++ Runtime Library error R6034: An application has made an attempt to load the C rutime library incorrectly. Please contact the application's support teamfor more information.

Lucky me: I thought I had one problem, but now I have two. I feel duty bound to report this new error under the "Installer" category. Fortunately the installer completes, and I can run the new Thunderbird to confirm the same error occurs (though I stupidlu omit to bring my RSS feeds forward into my new version). So now I can go ahead and add to the bug report for the original problem, since unfortunately it hasn't gone away. My motivation is severely diminished, however, so this may not actually happen. At least I reported the installer problem.

One step forward, at least only one step back.

Stackless Python Continues to Amaze

I recently started tracking the Stackless Python mailing list. I've been interested in Stackless, the brainchild of the fearsomely clever Christian Tismer, for some time now. I recently acquired a client with active interests in that area, so it behooves me to stay up to date.

Stackless allows you to organise your work into tasklets (which as far as I understand it have replaced the original microthreads as the unit for scheduling: this is all pretty new to me, so apologies if it's incorrect). An interesting thing about tasklets is that you can pickle them, pass the pickle to another computer, unpickle them and resume the tasklet in the new environment.

My attention was grabbed by one recent thread in which a user explains that he was trying to perform such interactions across a heterogeneous environment with Power PC, SPARC and Intel-based CPUs and found that it worked between SPARC and PPC, but the Intel architecture appeared to be failing -- presumably due to the different endianness.

After a couple of messages surmising that this was hardly surprising, Christian came into the thread to explain that there was no architecture-dependent code in the pickled thread resumption functionality. Lo and behold, the original poster came back explaining:

I must apologize, it turns out my problem was due to the fact that I was using a different version of stackless on the intel machine. In case anyone is curious, resuming pickled tasklets across architectures is easy. Thank you for your replies!

Let me repeat that, just in case you missed it: resuming pickled tasklets across architectures is easy. As far as I can see this gives Python an amazing capability to produce applications with highly-distributed architectures. It's going to be interesting to see where Stackless goes with this, but it should be shouted from the rooftops. This is an advanced feature that represents a real advantage for (Stackless) Python in a world where everyone is wondering how they can accommodate the new multicore processors.

With Stackless, your CPUs don't even need to have the same architecture, but the feature will work just as well in the homogeneous environment that multi-processor computers provide.

What, Me, Unique?

Hardly, as the box below shows. But ..

HowManyOfMe.com

There are: 48 people with my name in the U.S.A. How many have your name?

... just the same none of those others have

• Fathered my son (hi, Simon!)
  
• Chaired three PyCons
• Written a book about Python
• Taught almost 3,500 students about networking and security topics
• Helped comp.lang.python readers to use Python

So there are lots of things that distinguish me from the doubtless equally worthy 47 others. It's always interesting to see the uses to which the web will put information. Twenty years ago HowManyOfMe.com would only have been imaginable as an online application. Now it just merits a raised eyebrow and a passing "good idea". Who knows what tomorrow's kids will take for granted assuming the preceding generations don't destroy the world.

Oh well, that's enough cheerful for today.

Thunderbird Politically Incorrect?

Someone just posted a rude but amusing remark about a regular comp.lang.python contributor, so I replied off-list with a snigger. Imagine my surprise when Thunderbird's spilling chucker failed to recognise the word and suggested a change to "nigger"!

Nice to be back on the blog. I've been working my ass off lately.

Moin Moin - Introduction for Developers (Thomas Waldmann, Alexander Schremmer)

MoinMoin was initially a one-man project, but the original author is less frequently available now and there is a core team that is collectively responsible for the code. The project has a large user base, which grew significantly when Unicode processing was introduced and enabled internationalisation.

For the rest of this presentation you should see the published slides, which unfortunately not only accurately represent the content of the talk but also encapsulate its presentation. I suspect (though I do not know them) that the speakers were perhaps nervous, and not therefore confident enough to talk ad lib in a foreign language to what must sometimes seem like an intimidating audience.

I found this a little frustrating since the few departures from reading the slides that the presenters allowed themselves made it clear that they both had an excellent grasp of a very popular Python web application. With more time at the conference I know I would have liked to talk to them both about possible applications and developments.

Literate Testing with Doctest (Marius Gedminas)

Marius started out by outlining the reasonable uses of doctest, both inline in programs and in files that are separate from the source code, and advised us not, for example, to put complicated setup or tests for obscure corner cases in the README. DocTest can be easily integrated with unittest, by defining a test suite that runs the doctests!

He then pointed out DocTestSuite and DocFileSuite for externalising the tests, and considered how to handle complicated setup. One way is to define setup and teardown functions, and you can also put code in test.globs, but these methods should be limited for ease of comprehension.

The advantage of doctest is in encouraging simlicity and interspersed narrative to explain the tests. Another option is to define a test module. The advice is to avoid mixing long tests and regular code in the same file.

In real life, of course, some features of doctest are difficult to use when data is not reliably repeatable (such as the addresses of objects in repr()-style representations). Also problematic are ellipsis-elided outputs, which are to be ignored, and ordering issues. Web testing is even more difficult, as responses are frequently state-dependent. marcus showed some examples of testbrowser, a simple system allowing web testing.

Doctest disadvantages include the disability to step through tests. Documentation (docstrings, README files) should be separated from unit and functional tests.

Questioning revealed that Zope uses a slightly different version of doctest from the Python 2.4 distribution, although "in theory" they offer the same functionality. Jim Fulton asked whether the speaker had looked at Fit (and apprently PyFit is available) but although Jim would like to see Fit and doctest integrated there was no experience with the packages in the room.

The only slightly problematic aspect of this presentation was a couple of times when the speaker made inaudible responses to inaudible questions from the chair. This would have been less troublesome if I had not been at the back of the room, or if the PA system had allowed audibility. Otherwise I got a lot of information in a very short time, well presented.

Working Together on the Web (Kevin Dangoor)

My whole EuroPython experience was a little disjointed, starting with arrival. I and one other delegate ended up on a bus that went, not to CERN but to a local railway station three or four kilometers away. The result was a late arrival, so by the time I took my place in this session Kevin was just starting to present his first example, how to adapt a simple application to the WSGI interface.

Using simple_app was the easiest, but Kevin also showed us how to do it by defining our own class. He then considered the problem of maintaining session state (though Kevin says he prefers not to use state).

WSGI is also helpful in such situations: its concept of middleware (which talks to the web server on one side and a web application on the other) can help. The example here was of a "Latinator" that translated into pig Latin. Kevin showed us how to define a class with a method that can be passed to the WSGI framework as the start_response argument. WSGI is good because there have been many contributions to the PyPI index.

Kevin next introduced us the the Paste system designed by Ian Bicking. This contains a confusing number of packages and modules, which makes it hard to describe and hard to climb the learning curve and start using it. He suggested this demonstrated that even in the Open Source world we are faced with the "build vs. buy" choice, even though "buy" is really an investment in learning someone else's package and build, as usual, is learning and understanding the base technologies you want to implement, and writing your own implementation.

In TurboGears Kevin made the choice to use as much existing technology as was practicable.

The fictitious web package author might decide to package up other dependencies, but this can cause problems for people with other versions of the components she chooses to bundle. This is the classic version dependency. The setuptools system has defined a cross-platform format for installation of Python packages, the .egg format. This can also be used to install packages with binary components, and is proving to be very usable.

The .egg format allows other systems to read the metadata about the package, which is useful in many environments where plugins are used, even in non-Python environments such as Eclipse. The discussion about how entry points were defined to plugins didn't seem terribly easy to understand to me - maybe I was just distracted by the extraneous materials, but I didn't really see the point.

Kevin closed by talking about Beaker, starting by showing us theat it depended on the MyghtyUtils package and defined two paste entry points. The examples at this stage started to leave me wishing I had attended the AJAX session instead, but this could just be me: others might have found the material relevant to their interests.

This presentation relied for contrast on rather too much use of "humourous" slides: a picture of an egg when talking about eggs, a picture of an RJ45 plug in a socket when talking about plugins. I have no objection to such devices, but they should either enhance understanding or be shown for a very short time to give everyone a quick smile before moving on.

Kevin knows his stuff and is always an engaging speaker. A tour de force, though, should really be allocated an hour, and I suspect this session should have covered more ground in less detail or vice versa.

Front Page News

Did you ever hear the one about the man who was too busy to save time? I think we all at some stages in our lives lose sight of what we are struggling to do, when perhaps a few quiet moments of contemplation would allow us to move forward.

For a while now I've wanted to finish the www.holdenweb.com front page news feature so I could get on to newer AJAX-based work. The solution, of course, was staring me in the face when I looked for it: instead of holding up AJAX to finish off a pre-AJAX project I could actually use AJAX to speed up its completion. This strategy has worked well.

The original news feed was based on a programmed search of O'Reilly's Meerkat stream aggregator, but Meerkat went away and anyway the mechanism was cumbersome and error prone. I was looking for something lightweight that the mythical "average web user" would be able to use in a fairly foolproof way. I am happy to say that the first prototype is working exceptionally well, and the holdenweb.com front page can now be as up to date as I am!

At the moment the holdenweb.org site is still being generated the old-fashioned way, but I predict the retrofit to this parallel site will be easy.

Sitting with Nellie

When I started work "sitting with Nellie" (which meant watching an experienced member of staff do a job, and eventually graduating through helping to doing it oneself) was a time-honoured way of learning to perform a task. Back in mediaeval times it could take five years or more to learn a trade by apprecticeship, but you were a fully trained master when you were finished. Apprenticeships fell out of fashion when companies wanted their staff to be more versatile (and hence more interchangeable), and their death was hastened by the rise of "the training industry" in the 1960s, (with an interest in the training revenues that formal training and its inevitable out-sourcing brought with it).

Apprenticeships may no longer be appropriate, but it's also undeniable that sitting with Nellie had certain advantages over formal training. From a personal standpoint the major advantage was the ability to soak up what I might grandly call the prevailing culture and ethos - to learn how the place worked by talking to people who had been around much longer. I well remember having some of my more naiive assumptions and suppositions being questioned hard by cynics whose cynicism was borne of experience (and sometimes occasioning much good-humoured amusement).

Interestingly, Jon Udell has recently observed that the open source world could use similar techniques to educate people about what was involved in developing software, and specifically the techniques and tools used by the open source world. This approach to an extent generalises (because the open source world is very diverse and so the tools of one project may not be appropriate for another), but the idea has a lot of merit. Ted Leung has further observed that children really don't have any effective way of finding out what they'd like to do, and that a "sitting with Nellie" approach could be helpful in this respect too. This rings true with my own experience. Google's Summer of Code project allows open source projects to offer some experience to students for a liimted period, but there's a lot more we could do.

Le plus ça change, le plus c'est la même chose, as they say in France. We need to realise that change isn't always a good thing and, even if a change once were good, it needn't necessarily be permanent. Perhaps when apprenticeships were discarded an important baby was thrown out with the bathwater of restrictive work practices. Let's see if we can't arrange to have would-be programmers spending some time sitting with Nellie.

Need for Speed Wrap-Up

Wow! This week has flown by so quickly there's been no chance to blog about what's been going on in Reykjavik. For those of you who aren't aware of the sprint I have already outlined the basics in Do We Need Speed? but there's more to say. Much more.

Thanks for the Memories
This has been a very busy week, and it's not yet been possible to thank everyone for the roles they have played in putting it together. First of all I must thank all the participants for their efforts. Everyone has worked amazingly hard this week. The declared hours of the sprint (along the lines of extreme programming) were 9:00 am to 6:00 pm, but it wasn't unusual to see people busily working away after 11:00 pm.

I have so enjoyed working with this awesome group, and the teamwork has been great. See the participants list to find out who they are, and see pictures on flickr and on jafo's journal (permanent links on the summary page). Favourite picture of the week has to be from the Blue Lagoon (that's me, with my hand out of the water at the back).

Other web references include these blog entries from Richard Jones and occasional remarks from the effbot, as well as Sean Reifchneider's blog. Jack Diederich has also mentioned the sprint in both his general and his geek blogs.

It's been a pleasure, and I am sure that as the result of working together this week we will all be more open to future collaborations. I sincerely hope that we can find other sponsors to support this work as generously as EWT have.

Thanks for the Support
The Python community as a whole has been awesomely behind us in this effort. Particular mention should go to Neal Norwitz, Andrew Kuchling, Marc-Andre Lemburg and Brett Cannon. They have all helped to flesh out ideas, picked up code nits before they became troublesome and cleared up typos in both code and documentation. Not to mention holding us up by pointing out unfortunate misconceptions which would have led to errors if not squashed. Many others have also pitched in, showing the true Python community spirit by reviewing patches and offering advice, and the sprint's success is theirs as well.

We should not forget the magnificent hospitality of CCP Games, a recently-elected sponsor member of the Python Software Foundation. Not only have they entertained us magnificently all week, they have also had three of their staff at the sprint and have provided local knowledge that has made everyone's week more enjoyable. It's never easy being thousands of miles away from home, and this all helped enormously. Special thanks to Kristján V. Jónsson for giving up so much of his time in a week that included a public holiday.

Thanks for the Sponsorship
We should thank EWT, LLC for the most practical support of all. They have funded the air fares and hotel expenses for fourteen sprint members as well as providing the trip to the Blue Lagoon, coffee throughout the week (sprints run on caffiene) and a close-out dinner at the hotel. They have also sent three of their staff from their Beverly Hills head offices to take part in the sprint, even though that meant they missed their Memorial Day weekend. The fourth-floor sprint room is an extremely refreshing contrast with most places I have sprinted before: access to daylight is quite a novelty (though 24-hour daylight took a little getting used to).

EWT's CEO, David Salomon, addressed the sprint in a teleconference on Wednesday, and explained something about his company's ethos and general approach. EWT's support of the sprint, and more generally of Python, is motivated by business considerations, but David's talk made it evident that he sees EWT's role in a larger context. He also announced EWT's intention to make a donation to the Python Software Foundation and to institute a scholar-in-residence program targeted at supporting individuals making open source contributions.

We also each received a Nokia 770 as a gift from EWT. This amazing little device runs Debian Linux, and Richard Jones had his device running pygame by Wednesday morning.

This is also a good place to record my personal thanks to David for hiring Holden Web to undertake the organization of this event. When I made my first hesitant posting to the conferences-discuss list and ended up as chairman of PyCon DC 2003 I little realised where it would lead.

So, What Have We Achieved?
The formal record of our successes is on this wiki page (and a link is best simply because even as I write this entry there are hackers all around me striving to get even more speed into Python, so we are by no means finished yet). Sean Reifschneider reviewed the outstanding patches and came up with a list of potential speedups, which we have been reviewing and chipping away at as a part of our fairly comprehensive task list. The next release of Python isn't due out until August, so there is plenty of chance to build on the work that's been done.

Although not directly speed-related, we have also put quite a lot of work in on the pybench module, as a result of which its accuracy and repeatability should be improved. Not only that but this work may also lead to the introduction of another benchmark with even better repeatability. Lots of tests have been written this week ...

Until all the new tests are incorporated into formal benchmarks it might not be easy to detect the speedup; it's evident to everyone here, though, that many of the speedups are in areas that the average user will perceive. The improvements to string and unicode handling will probably be welcome to most users. I hope readers will feel free to post comments to let the other sprinters know how much their efforts are appreciated.

And Next?
It is the earnest of hope all the sprinters that this event will demonstrate to the software and related industries that they can engage sections of the open source community in a way that assists both sides. Sprinters have been quite willing to address specific performance issues raised by the sponsors, although there was no compulsion on them to do so, and there has been a general appreciation that both sides will benefit from the sprint.

We live in strange times, and the computing industry is having to come to terms with an infrastructure that is increasingly developed and maintained by people not under their direct control. This is scary to the average commercial manager: David Salomon deserves credit for his perception that those who engage the open source community most whole-heartedly will benefit most.

I fly out of Keflavik at 7:20 tomorrow morning. By coincidence I shall be on the same aeroplane as four American friends who change flights in Iceland. So, I have a week's holiday to look forward to, including visits to Loch Ness and a Guiness brewery. Rest and relaxation will be the order of the day. After that it's back to work, and being a director of the Python Software Foundation.

This has been one terrific week, though tiring, and has built a cohesion among the team which will remain even after the sprint. I hope it leads other organizations to consider supporting the open source community instead of just being passive consumers of the output without putting much back. With luck, much may hinge on this little project.

Let the Games Begin

I was looking for an erudite Latin tag, but couldn't find one. The Need for Speed sprint starts tonight with a welcome get-together at CCP, the Icelandic company reponsible for the Eve Online game.

I've been monitoring the flight arrivals from my hotel room in Reykjavik, and as I write the flights of the first five arrivals have all been reported as having landed on time. So if all has gone well, and thanks to the assistance of our Icelandic hosts, they should find a driver waiting for them when they clear customs and immigration, getting them to the hotel in around a half an hour.

This is exciting!