Front Page News

Did you ever hear the one about the man who was too busy to save time? I think we all at some stages in our lives lose sight of what we are struggling to do, when perhaps a few quiet moments of contemplation would allow us to move forward.

For a while now I've wanted to finish the www.holdenweb.com front page news feature so I could get on to newer AJAX-based work. The solution, of course, was staring me in the face when I looked for it: instead of holding up AJAX to finish off a pre-AJAX project I could actually use AJAX to speed up its completion. This strategy has worked well.

The original news feed was based on a programmed search of O'Reilly's Meerkat stream aggregator, but Meerkat went away and anyway the mechanism was cumbersome and error prone. I was looking for something lightweight that the mythical "average web user" would be able to use in a fairly foolproof way. I am happy to say that the first prototype is working exceptionally well, and the holdenweb.com front page can now be as up to date as I am!

At the moment the holdenweb.org site is still being generated the old-fashioned way, but I predict the retrofit to this parallel site will be easy.

Sitting with Nellie

When I started work "sitting with Nellie" (which meant watching an experienced member of staff do a job, and eventually graduating through helping to doing it oneself) was a time-honoured way of learning to perform a task. Back in mediaeval times it could take five years or more to learn a trade by apprecticeship, but you were a fully trained master when you were finished. Apprenticeships fell out of fashion when companies wanted their staff to be more versatile (and hence more interchangeable), and their death was hastened by the rise of "the training industry" in the 1960s, (with an interest in the training revenues that formal training and its inevitable out-sourcing brought with it).

Apprenticeships may no longer be appropriate, but it's also undeniable that sitting with Nellie had certain advantages over formal training. From a personal standpoint the major advantage was the ability to soak up what I might grandly call the prevailing culture and ethos - to learn how the place worked by talking to people who had been around much longer. I well remember having some of my more naiive assumptions and suppositions being questioned hard by cynics whose cynicism was borne of experience (and sometimes occasioning much good-humoured amusement).

Interestingly, Jon Udell has recently observed that the open source world could use similar techniques to educate people about what was involved in developing software, and specifically the techniques and tools used by the open source world. This approach to an extent generalises (because the open source world is very diverse and so the tools of one project may not be appropriate for another), but the idea has a lot of merit. Ted Leung has further observed that children really don't have any effective way of finding out what they'd like to do, and that a "sitting with Nellie" approach could be helpful in this respect too. This rings true with my own experience. Google's Summer of Code project allows open source projects to offer some experience to students for a liimted period, but there's a lot more we could do.

Le plus ça change, le plus c'est la même chose, as they say in France. We need to realise that change isn't always a good thing and, even if a change once were good, it needn't necessarily be permanent. Perhaps when apprenticeships were discarded an important baby was thrown out with the bathwater of restrictive work practices. Let's see if we can't arrange to have would-be programmers spending some time sitting with Nellie.

Need for Speed Wrap-Up

Wow! This week has flown by so quickly there's been no chance to blog about what's been going on in Reykjavik. For those of you who aren't aware of the sprint I have already outlined the basics in Do We Need Speed? but there's more to say. Much more.

Thanks for the Memories
This has been a very busy week, and it's not yet been possible to thank everyone for the roles they have played in putting it together. First of all I must thank all the participants for their efforts. Everyone has worked amazingly hard this week. The declared hours of the sprint (along the lines of extreme programming) were 9:00 am to 6:00 pm, but it wasn't unusual to see people busily working away after 11:00 pm.

I have so enjoyed working with this awesome group, and the teamwork has been great. See the participants list to find out who they are, and see pictures on flickr and on jafo's journal (permanent links on the summary page). Favourite picture of the week has to be from the Blue Lagoon (that's me, with my hand out of the water at the back).

Other web references include these blog entries from Richard Jones and occasional remarks from the effbot, as well as Sean Reifchneider's blog. Jack Diederich has also mentioned the sprint in both his general and his geek blogs.

It's been a pleasure, and I am sure that as the result of working together this week we will all be more open to future collaborations. I sincerely hope that we can find other sponsors to support this work as generously as EWT have.

Thanks for the Support
The Python community as a whole has been awesomely behind us in this effort. Particular mention should go to Neal Norwitz, Andrew Kuchling, Marc-Andre Lemburg and Brett Cannon. They have all helped to flesh out ideas, picked up code nits before they became troublesome and cleared up typos in both code and documentation. Not to mention holding us up by pointing out unfortunate misconceptions which would have led to errors if not squashed. Many others have also pitched in, showing the true Python community spirit by reviewing patches and offering advice, and the sprint's success is theirs as well.

We should not forget the magnificent hospitality of CCP Games, a recently-elected sponsor member of the Python Software Foundation. Not only have they entertained us magnificently all week, they have also had three of their staff at the sprint and have provided local knowledge that has made everyone's week more enjoyable. It's never easy being thousands of miles away from home, and this all helped enormously. Special thanks to Kristján V. Jónsson for giving up so much of his time in a week that included a public holiday.

Thanks for the Sponsorship
We should thank EWT, LLC for the most practical support of all. They have funded the air fares and hotel expenses for fourteen sprint members as well as providing the trip to the Blue Lagoon, coffee throughout the week (sprints run on caffiene) and a close-out dinner at the hotel. They have also sent three of their staff from their Beverly Hills head offices to take part in the sprint, even though that meant they missed their Memorial Day weekend. The fourth-floor sprint room is an extremely refreshing contrast with most places I have sprinted before: access to daylight is quite a novelty (though 24-hour daylight took a little getting used to).

EWT's CEO, David Salomon, addressed the sprint in a teleconference on Wednesday, and explained something about his company's ethos and general approach. EWT's support of the sprint, and more generally of Python, is motivated by business considerations, but David's talk made it evident that he sees EWT's role in a larger context. He also announced EWT's intention to make a donation to the Python Software Foundation and to institute a scholar-in-residence program targeted at supporting individuals making open source contributions.

We also each received a Nokia 770 as a gift from EWT. This amazing little device runs Debian Linux, and Richard Jones had his device running pygame by Wednesday morning.

This is also a good place to record my personal thanks to David for hiring Holden Web to undertake the organization of this event. When I made my first hesitant posting to the conferences-discuss list and ended up as chairman of PyCon DC 2003 I little realised where it would lead.

So, What Have We Achieved?
The formal record of our successes is on this wiki page (and a link is best simply because even as I write this entry there are hackers all around me striving to get even more speed into Python, so we are by no means finished yet). Sean Reifschneider reviewed the outstanding patches and came up with a list of potential speedups, which we have been reviewing and chipping away at as a part of our fairly comprehensive task list. The next release of Python isn't due out until August, so there is plenty of chance to build on the work that's been done.

Although not directly speed-related, we have also put quite a lot of work in on the pybench module, as a result of which its accuracy and repeatability should be improved. Not only that but this work may also lead to the introduction of another benchmark with even better repeatability. Lots of tests have been written this week ...

Until all the new tests are incorporated into formal benchmarks it might not be easy to detect the speedup; it's evident to everyone here, though, that many of the speedups are in areas that the average user will perceive. The improvements to string and unicode handling will probably be welcome to most users. I hope readers will feel free to post comments to let the other sprinters know how much their efforts are appreciated.

And Next?
It is the earnest of hope all the sprinters that this event will demonstrate to the software and related industries that they can engage sections of the open source community in a way that assists both sides. Sprinters have been quite willing to address specific performance issues raised by the sponsors, although there was no compulsion on them to do so, and there has been a general appreciation that both sides will benefit from the sprint.

We live in strange times, and the computing industry is having to come to terms with an infrastructure that is increasingly developed and maintained by people not under their direct control. This is scary to the average commercial manager: David Salomon deserves credit for his perception that those who engage the open source community most whole-heartedly will benefit most.

I fly out of Keflavik at 7:20 tomorrow morning. By coincidence I shall be on the same aeroplane as four American friends who change flights in Iceland. So, I have a week's holiday to look forward to, including visits to Loch Ness and a Guiness brewery. Rest and relaxation will be the order of the day. After that it's back to work, and being a director of the Python Software Foundation.

This has been one terrific week, though tiring, and has built a cohesion among the team which will remain even after the sprint. I hope it leads other organizations to consider supporting the open source community instead of just being passive consumers of the output without putting much back. With luck, much may hinge on this little project.

Let the Games Begin

I was looking for an erudite Latin tag, but couldn't find one. The Need for Speed sprint starts tonight with a welcome get-together at CCP, the Icelandic company reponsible for the Eve Online game.

I've been monitoring the flight arrivals from my hotel room in Reykjavik, and as I write the flights of the first five arrivals have all been reported as having landed on time. So if all has gone well, and thanks to the assistance of our Icelandic hosts, they should find a driver waiting for them when they clear customs and immigration, getting them to the hotel in around a half an hour.

This is exciting!

Tutorials from PyCon

When I finally made time to visit comp.lang.python for the first time in two or three weeks I found a post that said

I would like to know if anybody can point me to the site, where it is possible to find the tutorial "Using Databases in Python" which is mentioned by Steve Holden here: http://tinyurl.com/ectj8

So of course I had to make sure that the material was available on the web. As a result you con now download my tutorials in either PDF or Open Office Impress format (yes, for once I eschewed using the obvious Microsoft products, and found that the Open Office component was a more-than-acceptable clone).

Using Databases in Python: Impress PDF

An Introduction to wxPython: Impress PDF

These materials are available under a Creative Commons license. Thank Guido van Rossum for inspiring me with the liberal terms of the Python license.

Squidoo: Just Another Web Phenomenon?

Seth Godin, a marketing whizz if ever there was one, recently started promoting a new web service called Squidoo that's aimed at making web authorship easier. The idea is to buld "lenses", which are promoted as offering ways to look at the world.I just noticed there are graphics you can use to link to your lens.
So take a look at Python is Amazing. And if you find something amazing I haven't included please feel free to let me know!

Shameless Self-Promotion

An Amazon review of Python Web Pogramming reminds me that the book wasn't written just to explain Python ... thanks, Sheila!

20 of 21 people found the following review helpful:

Excellent example snippets; Clear explanations, February 24, 2002

Reviewer:

Sheila King "desk-worker mom who needs to exercise" (L.A. County, California) - See all my reviews

If you are going to be using Internet protocols, doing network programming, or web programming with Python, and these are new topics for you, I would highly recommend this book.

The book starts with a brief overview of the Python language. The author's intention is that someone with a fairly extensive programming background in other languages would be able to pick up enough Python from this overview to be able to do the rest of the programming in the book. Perhaps so. I already know Python, but did find the summary in the front informative.

I really like the fact that nearly every page has a code snippet on it. Examples are brief and to the point. The author explains each line of code and has a very direct and clear way of explaining things. I found the explanations easy to read and understand.

After the brief Python Language overview, comes an overview of sockets and socket programming. I've been trying to learn a bit about the whole topic of sockets by searching the web and nothing I found on the web explained it as clearly as this book. I now appreciate the difference between TCP and UDP protocols and have an idea of the situations in which I would want to use each. If you want to learn low-level sockets, or how to write your own socket protocols, this is not the book you are looking for. This book basically assumes you will go with either TCP or UDP (and ignores the other types of sockets available in the Python socket library). However, these will probably suit most people's needs.

The author then walks you through each of the Internet data-handling libraries in Python, such as the telnetlib, ftplib, poplib, smtplib and so on. He gives examples of working code for each library, showing first how to implement clients, and later on how to implement servers. If you want to work with these libraries, these explanations should be very helpful.

Later in the book, Holden addresses using databases in Internet programming, using XML and writing your own web-application framework. I haven't yet had a chance to go through these chapters in detail (I've skimmed them only). But there is a LOT of stuff there. One thing the author does at the beginning of each new section, is give an overview of the topic (such as an overview of why you might want to use a database, how databases work, or why you might want to work with a web framework). For me, I really appreciate this type of overview. It helps give me a context for the new information, and helps me to make better sense of it. I read through some of the database chapters where he explains how the SQL query language works, and again, I have to say it is one of the best explanations I've read. (Most explanations I've read about SQL have just convinced me I wanted to steer clear of it.)

Another nice thing, is how he sort of "works you up to" SQL. He starts out with regular Python code, and shows how parts of it are similar to working with an SQL database, and then eventually transitions into the full SQL language. He also addresses database design and efficiency.

Overall, I'd say if you want a good overview of the topics mentioned here, want to understand the reasoning behind their use, and want to be able to understand good design and efficiency, then this book should really help you out.

End of Internet Prematurely Flagged

In Tech Blogosphere has peaked [the blogosphere deserves a capital?] Phil Sim suggests (I paraphrase) that

"anybody who's going to have a blog has one by now, after two years as a journalist you get stale, lots of bloggers are going back to real life"

and graphs the "daily reach" of memeorandum.com, thereby confirming that for him it's all about the eyeballs. When I look back at my own blogging history I see that there are frequently months when I have written nothing at all in my blog, and hey, here I am still blogging.

If blogging is just "look at me, Ma!" then the sooner it stops the better. Blogging works best not when it's a publicity channel but when it's a reasonably consistent, selective window onto the world of the blogger. Like journalism, much blog content is ephemeral, and the blogging world needs to remember that. If it's not on page one of my del.icio.us then it's history. If you want the history it's sometimes there, but remember it could be a revisionist history as blog posts can be changed at any time.

Test-Driven Development

Some code, for a change. I recently taught an introductory Python class to some fairly experienced programmers, and we had an hour or so left at the end of the class to try a problem. We'd been discussing test-driven development, so we arrived at the idea of creating a problem that was fairly simple in scope and then writing tests and a solution to the problem.

The idea was to write the tests first, though it will be no surprise to those who've done this before that the nature of the tests changed as errors came to light during development. The problem was as follows:

Given a directory structure of arbitrary shape, locate all JPEG images and copy them into a named destination directory. [Not specified but implied: the files should continue to exist in their original positions]

It turned out that the level was fairly well chosen. None of the students managed to complete the task, but they all had a fairly clear sense of where they were going by the end of the exercise, giving them something to work on independently after I'd gone. Of course I had to provide them with a "model solution", which I'm happy to say I just managed to create in the time allotted.

Here is the test harness for the jpegcopy requirement:

import jpegcopy
import unittest
import os

BASEDIR = '/c/Steve/Projects/BrightonHove'
BASEDIR = 'c:/Steve/Projects/BrightonHove'
INDIR = os.path.join(BASEDIR, "input")
OUTDIR1 = os.path.join(BASEDIR, "output1")
OUTDIR2 = os.path.join(BASEDIR, "output2")
EXPECTED = ['%s.jpg' % s for s in "f1 f2 f3 f4 f5 f6".split()]

class TestJpegCopy(unittest.TestCase):

   def setUp(self):
       """Ensure both output directories are empty."""
       for d in OUTDIR1, OUTDIR2:
           fl = os.listdir(d)
           if fl:
               try:
                  for f in fl:
                     os.unlink(os.path.join(d, f))
               except:
                   raise ValueError, "Cannot empty directory %s" % d

   def testEmpty(self):
       n0 = jpegcopy.main(OUTDIR1, OUTDIR1)
       self.assertEquals(n0, 0)

   def testDir1(self):
       n1 = jpegcopy.main(INDIR, OUTDIR1)
       self.assertEquals(n1, 6)
       self.assertEquals(os.listdir(OUTDIR1), EXPECTED)

   def testDir2(self):
       n2 = jpegcopy.main(INDIR, OUTDIR2)
       self.assertEquals(n2, 6)
       self.assertEquals(os.listdir(OUTDIR2), EXPECTED)


   def tearDown(self):
       for d in OUTDIR1, OUTDIR2:
           for f in os.listdir(d):
               os.unlink(os.path.join(d, f))


if __name__ == "__main__":
   unittest.main()

Nothing too fancy here. The tests are parameterised. We set them up by clearing both the output directories. Then we test that they are indeed empty. Then we test to make sure that we can put the JPEGs into two different directories and verifying that each time we see six files copied. Finally we check that both output directories contain the same thing. We tear down the test by deleting the contents of both directories.

This will probably show my ignorance, highlighting the fact that test-driven methods don't yet come naturally to me. I'll be happy to integrate suggestions for improving test coverage. My solution follows.

"""Copy jpegs from a recursive to a flat directory structure."""

import os
import shutil

def main(indir, outdir, debug=0):
   count = 0
   for f in os.listdir(indir):
       if os.path.isdir(os.path.join(indir, f)):
           count += main(os.path.join(indir, f), outdir, debug=debug)
       else:
           if f.endswith(".jpg"):
               count += 1
               shutil.copyfile(os.path.join(indir, f),
                               os.path.join(outdir, f))
   if debug:
       print "Returning %d for %s" % (count, indir)
   return count

if __name__ == "__main__":
   main("input", "output1", debug=1)

As you can see I have put a simple test inline; this script is not intended to be run as a main program, but the debug output was useful sometimes when tests failed for obscure reasons.

Again, if readers can suggest improvements I'll incorporate them as I have time. You should be able to copy the code from your browser window and paste it into an editor. Thanks to the MoinMoin developers for colorize.py.

Do We Need Speed?

Well, finally the news is out. The reason things have been so quiet on this blog (and why I've been missing from comp.lang.python pretty much since PyCon) is that all available spare time has been going into trying to get the Need for Speed sprint off the ground. To my knowledge this event is unique, though an assertion like that invites correction from the better-informed.

Commercial organisations are free to use the output from open source projects without putting anything back (vide Industrial Light and Miserliness), and by and large this is expected -- open source licensing terms are fairly explicit, and few are even as "draconian" as the GPL (which currently says in essence that if you distribute GPL-derived products you have to make the source available). By the same token, of course, there's nothing to stop people from supporting open source projects if they want to.

Hopefully this little gap in the curtain will help to trigger a realisation among commercial software developers that they can make a huge difference to open source projects from which they might benefit (or from which they already have benefited) by the application of what are, in strictly commercial terms, relatively modest funds. The various Foundations controlling some of the better-known open source projects do make their own efforts, but without significant external funding (such as that achieved by the PyPy project) and management expertise (extremely variable) it can be difficult to make progress. Heck, even management training might be a useful contribution from the world of pointy hair (I'm sorry, I'll wash my mouth out with soap later).

Of course we'll have to wait and see. It could be that I am completely wrong, that hardly any of the invited developers will bother to come, and that the sprint will really show that open source people don't want to play in the commercial playground at all. In which case I guess I'll have egg on my face.

If I'm right, though, it will show that there are benefits to be had on both sides of the equation and that, although open source developers don't do it primarily for the money, they don't necessarily object to working with commercial developers when there is a sufficient alignment of goals. I feel incredibly fortunate to have been given the chance to put this belief to the test, and I can't wait to see how this effort goes.

Settle Up with Sony

The Electronic freedom Foundation is trying to ensure that Sony Corporation provides reparations to anyone who inadvertently installed the rootkit software on their computer.

EFF is a body well worth knowing about and supporting. They prosecuted a class action suit against Sony BMG, and if you want to get in on the settlement now is the time.

Find out how to get your share.

Sun on the Run?

Talking, as I have been recently, about marketing, it was interesting to see a report of Sun's James Gosling apparently going negative at a recent education and research conference. In politics, and by extension in marketing, typically the frontrunner starts going negative when they perceive their lead is threatened. It's also interesting to see him breaking a cardinal rule and mentioning "the opposition" by name -- he's a techie at heart. Even though Python is mentioned only once in the article I linked to, we should be grateful for the free publicity.

Another significant feature of the report is that although Python is mentioned with a number of other languages, it's the other languages that are then dissed, leaving Python to suffer by association. Putting all this in context, though, Gosling's remarks were actually made in response to a question from an audience member.

The link I pointed you to makes it looks like Gosling's whole purpose was to defend his products against competing languages, when in fact that was far from reality. So another lesson is "don't believe everything you read in the papers (or on the web, for that matter)". The reporters have their own agenda, which is to spice things up and make them look like they're worth reading. They want your eyeball-seconds!

Just the same, I think it's revealing how Gosling chose to respond to the question. He's obviously feeing some sort of pressure from Python and its kind. I've given my own opinions on his language before, so I'll content myself with reiterating that this defensive response, coupled with a complete absence of objective criticism, can only be good news for Python.

A Little Marketing Effort

To follow up my last effort, I've created an evangelistic Squidoo lens. Comments and suggestions for improvement are welcome.

Marketing? Why Do You Use Python?

Well, the cat's among the pigeons and no mistake. Guido van Rossum recently received an email from John Sirbu that was basically a plea for more Python evangelism, to counteract the percieved successes of Ruby on Rails (RoR). Guido chose to republish this on his Artima blog under the title Marketing Python - An Idea Whose Time Has Come, although the word "marketing" doesn't appear in the original email.

Now, I happen to believe in this case that Guido is right, and that Python would indeed benefit from some serious marketing activity. Unfortunately it appears that the storm of responses (the count was almost at 70 last time I looked) has been generated mostly by people who have little or no idea what marketing is, and if they did would probably hold up their hands in horror and run away screaming "shameless commerce".

Some respondents appeared to find the whole idea of marketing offensive, and in so far as that's what led to Java's current popularity I can agree, because I happen to feel that Java has cost the industry a lot of money with its unnecessary straightjacket (see Java is Object-Oriented COBOL). But most people who took up the cudgels by commenting on Guido's blog (I deliberately avoided doing so myself) simply roll out the many reasons why Python is technically superior to Ruby/Perl/Java/my favourite language. These people are even more clueless than me about what drives the adoption of a programming language.

The key insights into promotion by marketing come from the realisation that you don't sell people things by focusing on the features of your product -- you have to explain the benefits instead. Sell the sizzle, as they say, and not the sausage. Anyone who is seriously interested in promoting Python should view Seth Godin's video presentation in the "Google Author Series". Seth is a marketing professional with a long history in the high-tech world, and his blog is worth keeping an eye on for occasional flashes relevant to the technical world. A valuable insight from the presentation is that word-of-mouth is the most effective way to promote anything: what we need is to tell everyone who asks (and even people who don't) how effectively Python can meet their needs. Clients aren't interested in whether I'm using Python, as long as I can help them solve their problems cost-effectively and in acceptable time.

This puts me in mind of a recent thread on comp.lang.python which started out as a discussion on whether it was a good idea to talk about Python as an interpreted language. Before it devolved into the inevitable discussion about what exactly constituted an interpreter and how all languages were interpreted because the CPU interprets instructions (c.l.py is sometimes an object lesson on topic diversion), there was a huge discussion about whether using the word interpreted to describe the language would engender negative perceptions.

From a marketing point of view, it really doesn't matter whether Python is an interpreted language or not. People want to know whether it's an effective way to solve their problems, so while interpreted is insignificant to meaningless from the point of view of many adopters, widely portable might not be. Of course the feature (interpreted) and the benefit (widely portable) are opposite sides of the same coin, but the difference in emphasis is crucial from a promotional point of view. Ultimately a potential adopter wants to know "What's in it for me?"

From a marketing point of view it seems to me that some of the most important aspects of the Python language are as follows:

• Easy to learn
• Easy to apply to a wide variety of problems
• Installed on every significant operating system
  
• Great networking support
  
• Large applications base to draw from
• Vibrant, helpful community
• Integrates easily with other languages
  
• Excellent literature from a range of publishers
  

Note that here I haven't mentioned any language features at all, but the points should be interesting to anyone considering adopting Python because they speak to the user's needs. There are doubtless other benefits, that I hope readers can help me add to the list. Clearly all these things have to be true to be effective in marketing the language, and also there musn't be equally important disbenefits that haven't been mentioned - if Python enthusiasts tried to encourage adoption by lying about the language, even by omission, that would quickly become counter-productive.

One current problem seems to be that a lot of Python enthusiasts in the web world are concerned that another language (another rule of thumb: don't mention the competition by name) is getting more than its fair share of the buzz. They are trying to counter that buzz by focusing on Python's technical superiorities, without realising that the adopters of the "competition" aren't bothered about technical issues. They have found a solution for a range of problems, and they are adopting it to solve those problems.

It may be that as they try to extend those solutions they will come to realise that their adopted technology doesn't have the depth to extend in all the ways they want it. At that point the Python needs to be ready to reiterate the benefits of Python, and to show how it can be used to extend their existing solutions rather than forcing them to reqrite from scratch. Since the web world is already well-used to mixed-language and mixed-technology solutions this should be a breeze.

Ultimately the point of this blog is to try and help Python users to become more effectve advocates for their favourite language. We have a vested interest in seeing Python more widely used, so let's forget the features when we're discussing "Why Python" and focus on the benefits.

Snakes on the Web [Django How-to]

The third speaker for my session as chair was Jacob Kaplan-Moss, one of the two primary developers of Django. He took as his example a sudoku solver, explaining that his company had been paying $180 per week for a third-party service to provide sudoku puzzles, so there was a potential saving here of $8,000 per annum.

The Django application stack is Model/URL/View/Template. Jacob explained he was going to walk us through the whole stack in developing the application. Many people are surprised to see the URL in there, but hey, that's how the app interfaces with the real world.

All models inherit from django.core.meta.model, and you can easily give your application an online production-ready admin interface by creating a META class with an admin class attribute. The model describes much more than just the database structure. A class can have a validator list, in which case all validators must return true.

The applications used Eppstein's well-known PADS algorithm to generate sudoku puzzles. This creates a puzzle and saves it in a database. The database layer is explicitly called, nothing ever happens without a specific API call, as opposed to other object-relational mappers that perform database I/O autonomically.

URL patterns are established to determine which URL calls what piece of view functionality. When a request is submitted get_validation_errors() will return a dict of errors. Callable returns render_to_response(template, context).

The Django templating language is relatively simple, and this was a deliberate design choice. It was primarly so that technically unsophisticated users in the newspaper production environment would find it straightforward and intuitive.

Jacob continued the development of his example by adding a further model for solving puzzles, walking through the solution provided by the PADS algorithm.

Django applications area pretty much pluggable, so if you have Django running all you need to do is download the code, add it to sys.path, add sudoku to INSTALLED_APPS, include the URLs for sudoku somewhere in the mapping (an inclusion mechanism allows the inclusion of a bunch of URLs with a single pattern match, and ths feature extends to multiple levels). Finally use django-admin to install sudoku.

Tadaa! Once again an effective demonstration of a Python-based web technology that people should be adopting in droves.

Effective AJAX with TurboGears

Kevin Dangoor is the leader of the TurboGears project. This was the second in the session that I chaired. I was lucky in drawing excellent speakers for my session, and I was very glad I got the chance to hear this talk.

Do I need Ajax? Kevin felt the answer to this questions is "probably yes"! The web is becoming hugely interactive, and people are getting used to sites being much less passive than they have been. Gmail is two years old, and it has forced the pace. Nowadays 95% of the world uses one of three browser platforms, and toolkits are available to smooth over the differences (the two specifically mentioned were Dojo and mochikit).

Rather than present the dry theory behind AJAX Kevin then focused on the question of interest to many web developers: "How can I use it"? He gave several interesting examples based on TurboGears, but it would be easy to adapt them to other environments.

Data entry: this hasn't traditionally been a friendly task on the web. To choose between thousands of items a pick list is not appropriate: real-time searching is helpful here (the example selected from thousands of items by restricting those displayed in a pulldown to partial matches). People are used to auto-complete fields, so they quickly adapt to this behavior in web applications.

Realtime updates: AJAX can continually poll the server for updated information. This can capture simultaneous edits to the same data (the example informed an editor in real-time that someone else had changed the data they were currently working on).

Ordered lists: AJAX can be used to detect drops in drag-and-drop operations and reorder the data directly in the browser window (this time a very nice drag-and-drop example was presented).

There are also things we can do that confuse the users: people like the way their browsers work, and expect other applications to work the same way. Ajax reduces the value of the browser "back" button, and Dojo has provided a solution, but another solution is to not use AJAX! (I'm always encouraged when a speaker doesn't try to present their topic as a universal panacea). Users want to be able to bookmark the current state of their web application, and again Dojo can help to solve this problem by changing the location bar along the way. It's also possible to make controls behave in ways people don't expect: using radio buttons to submit information is generally a bad idea (examples of each faulty behavior were presented, as warnings of what not to do).

Summary: know your users - with a small user base you can train people if it helps improve their productivity, but this won't work with a large user base. Don't break what people expect to work, as they have invested a lot of time in learning the expected and customary interface behaviors.

With Guido's recent plea for more effective Python marketing I think this talk was a poster-child for what might be done. It focused on practical needs and gave clearly comprehensible examples, with the underlying code to diminish fears of complexity. If we can show people Python being used in this way they will flock to use it.

Everything About Web Programming (except Programming)

This session was the first of three where I did my PyCon duty and chaired a session. I was glad Ian Bicking's talk was included as I'd been unable to record his earlier session. Ian suggested that the focus of his talk was on "accepting my inner sysdadmin".

Imaginary Landscape is a small company with, like many small companies, vague division of responsibilities. Ian wears two hats: the programmer (who likes to write new things) and the sysadmin (who would rather redeploy well-understood software). Conway's Law applies: organizations that design systems are constrained to produce designs which are copies of the communication structures of these organizations.

The company got where they are via Zope 2 development - which "felt a lot like PHP", with no overall process. So they moved into Webware and Subversion - but again without an overall plan. Things moved into production without a definite transition.

Deployment was stressful, so they decided to stop deploying! Applications were "multi-customer", and rather than developing good tools they would build complex applications. Ian now realises this was dumb, though it seemed smarter at the time. Unfortunately it meant that since everyone got the same software customization was infeasible, and configuration data had to go into the database, with no SVN control. Deployment should be easier than that, and simple deployment obviates the need for multi-customer applications: each customer can receive their own code.

Paste is a toolset that takes advantage of standard Subversion layout, providing a skeletal setup.py file and database model and a basic framework of templates and internally-used metadata. Testing also starts with a basic model created by the tool. Functional tests are really important: even just knowing that you can access the root page of an application is a worthwhile test.

Sometimes it can be problematic developing web applications in a test-driven environment. Developing code without opening it in a browser sounds cool, but it turns out that it's too easy to overlook gaping holes in an application (like there's no link to a page that the test framework has been accessing directly). This level of purity turns out to be too extreme in practice.

Configuration data is essential to a project, and it's important to differentiate between program configuration as usedby programmers and application configuration as used by system administrators. The tool creates a template for the deployment configuratiom file. Client data is kept in separate repositories, not int he application, and controlled separately from the application code.

Deployment uses the buildutils, and installation is moving towards a two-stage process. The first step installs the configuration file and the second step sets the application up in line with that configuration. Setuptools has the ability to install multiple versions of the same product, but this turned out not to be useful, as the separation between the versions wasn't sufficiently clean.

Ian now feels that "the computer" is a very poor context for installation of anything at all. ("Site-packages considered harmful"). So each site gets its own Python environment in development, and they use sitecustomize code to configure aaccording to the ACTIVE_SITE environment variable.

A nicely-focused talk with some valuable lessons for us all.

RIP Meerkat

Well, I just discovered that the end of an era is upon us.

For a long time now the holdenweb.com site has had news links extracted automatically from O'Reilly's meerkat system. Because the site generation code is still under test I only periodically click a button in its interface that says "Update News", and today for the first time I got a traceback from xmlrpclib complaining about 302 server status.

The redirect (which should really be permanent) is to a page that says

Update: Meerkat was shut down March 2, 2006

No reasons, just that bald statement. Since meerkat was an early exemplar in the web services field I thought it would be appropriate to mark its passing.

Now I have to get serious about a replacement. I have for some time been tweaking a feed extractor using Mark Pilgrim's excellent FeedParser module, so that's now moving up the priority list.

The underlying database is still Access, and so this morning I used DBManager Professional to convert it to PostgreSQL. Still needs changes, but I'm happy to say that at least I managed to generate a local copy of the site with no errors.

Kudos to Kuchling

While I am on the subject of conference organization, I don't believe I have recorded what a terrific job Andrew Kuchling has done in putting PyCon 2006 together. I know from personal experience that the organization of this conference places huge stresses on the chairman, but Andrew has performed magnificently.

This PyCon has been better in so many respects than the three that preceded it. I also know from the chatter I've seen in the background during the preceding year that things are becoming rather better codified to assist future organizers. PyCon will continue to improve.

Well done, Andrew!

Wirelessless

Further to my original remarks about the network at PyCon, this has probably been the biggest complaint I've heard people make, and the post-mortem continues as wireless associations continue to fade in and out during the sprints. Considering that

a) The networking company is at least part-owned by Marriott, and
b) A five-figure sum was negotiated for the wireless coverage and Internet connectivity

the results were very disappointing. For the record STSN didn't have anyone on-site, and there was no evidence that they had sent anyone over beforehand to do any testing and verify that the network coverage would be robust when descended upon by a crowd of geeks, most of whom were expecting the same kind of coverage we got at GWU.

I've no wish to prejudice any negotiations here, so I'll content myself with saying that during the sprint startups, when we shared a room with the organizers' debriefing, I was reminded how difficult it is to get everything right when you depend on third parties for so much of it. This issue isn't over yet, and meetings will follow to pick over the carcass.