Marketing? Why Do You Use Python?

Well, the cat's among the pigeons and no mistake. Guido van Rossum recently received an email from John Sirbu that was basically a plea for more Python evangelism, to counteract the percieved successes of Ruby on Rails (RoR). Guido chose to republish this on his Artima blog under the title Marketing Python - An Idea Whose Time Has Come, although the word "marketing" doesn't appear in the original email.

Now, I happen to believe in this case that Guido is right, and that Python would indeed benefit from some serious marketing activity. Unfortunately it appears that the storm of responses (the count was almost at 70 last time I looked) has been generated mostly by people who have little or no idea what marketing is, and if they did would probably hold up their hands in horror and run away screaming "shameless commerce".

Some respondents appeared to find the whole idea of marketing offensive, and in so far as that's what led to Java's current popularity I can agree, because I happen to feel that Java has cost the industry a lot of money with its unnecessary straightjacket (see Java is Object-Oriented COBOL). But most people who took up the cudgels by commenting on Guido's blog (I deliberately avoided doing so myself) simply roll out the many reasons why Python is technically superior to Ruby/Perl/Java/my favourite language. These people are even more clueless than me about what drives the adoption of a programming language.

The key insights into promotion by marketing come from the realisation that you don't sell people things by focusing on the features of your product -- you have to explain the benefits instead. Sell the sizzle, as they say, and not the sausage. Anyone who is seriously interested in promoting Python should view Seth Godin's video presentation in the "Google Author Series". Seth is a marketing professional with a long history in the high-tech world, and his blog is worth keeping an eye on for occasional flashes relevant to the technical world. A valuable insight from the presentation is that word-of-mouth is the most effective way to promote anything: what we need is to tell everyone who asks (and even people who don't) how effectively Python can meet their needs. Clients aren't interested in whether I'm using Python, as long as I can help them solve their problems cost-effectively and in acceptable time.

This puts me in mind of a recent thread on comp.lang.python which started out as a discussion on whether it was a good idea to talk about Python as an interpreted language. Before it devolved into the inevitable discussion about what exactly constituted an interpreter and how all languages were interpreted because the CPU interprets instructions (c.l.py is sometimes an object lesson on topic diversion), there was a huge discussion about whether using the word interpreted to describe the language would engender negative perceptions.

From a marketing point of view, it really doesn't matter whether Python is an interpreted language or not. People want to know whether it's an effective way to solve their problems, so while interpreted is insignificant to meaningless from the point of view of many adopters, widely portable might not be. Of course the feature (interpreted) and the benefit (widely portable) are opposite sides of the same coin, but the difference in emphasis is crucial from a promotional point of view. Ultimately a potential adopter wants to know "What's in it for me?"

From a marketing point of view it seems to me that some of the most important aspects of the Python language are as follows:

• Easy to learn
• Easy to apply to a wide variety of problems
• Installed on every significant operating system
  
• Great networking support
  
• Large applications base to draw from
• Vibrant, helpful community
• Integrates easily with other languages
  
• Excellent literature from a range of publishers
  

Note that here I haven't mentioned any language features at all, but the points should be interesting to anyone considering adopting Python because they speak to the user's needs. There are doubtless other benefits, that I hope readers can help me add to the list. Clearly all these things have to be true to be effective in marketing the language, and also there musn't be equally important disbenefits that haven't been mentioned - if Python enthusiasts tried to encourage adoption by lying about the language, even by omission, that would quickly become counter-productive.

One current problem seems to be that a lot of Python enthusiasts in the web world are concerned that another language (another rule of thumb: don't mention the competition by name) is getting more than its fair share of the buzz. They are trying to counter that buzz by focusing on Python's technical superiorities, without realising that the adopters of the "competition" aren't bothered about technical issues. They have found a solution for a range of problems, and they are adopting it to solve those problems.

It may be that as they try to extend those solutions they will come to realise that their adopted technology doesn't have the depth to extend in all the ways they want it. At that point the Python needs to be ready to reiterate the benefits of Python, and to show how it can be used to extend their existing solutions rather than forcing them to reqrite from scratch. Since the web world is already well-used to mixed-language and mixed-technology solutions this should be a breeze.

Ultimately the point of this blog is to try and help Python users to become more effectve advocates for their favourite language. We have a vested interest in seeing Python more widely used, so let's forget the features when we're discussing "Why Python" and focus on the benefits.

Snakes on the Web [Django How-to]

The third speaker for my session as chair was Jacob Kaplan-Moss, one of the two primary developers of Django. He took as his example a sudoku solver, explaining that his company had been paying $180 per week for a third-party service to provide sudoku puzzles, so there was a potential saving here of $8,000 per annum.

The Django application stack is Model/URL/View/Template. Jacob explained he was going to walk us through the whole stack in developing the application. Many people are surprised to see the URL in there, but hey, that's how the app interfaces with the real world.

All models inherit from django.core.meta.model, and you can easily give your application an online production-ready admin interface by creating a META class with an admin class attribute. The model describes much more than just the database structure. A class can have a validator list, in which case all validators must return true.

The applications used Eppstein's well-known PADS algorithm to generate sudoku puzzles. This creates a puzzle and saves it in a database. The database layer is explicitly called, nothing ever happens without a specific API call, as opposed to other object-relational mappers that perform database I/O autonomically.

URL patterns are established to determine which URL calls what piece of view functionality. When a request is submitted get_validation_errors() will return a dict of errors. Callable returns render_to_response(template, context).

The Django templating language is relatively simple, and this was a deliberate design choice. It was primarly so that technically unsophisticated users in the newspaper production environment would find it straightforward and intuitive.

Jacob continued the development of his example by adding a further model for solving puzzles, walking through the solution provided by the PADS algorithm.

Django applications area pretty much pluggable, so if you have Django running all you need to do is download the code, add it to sys.path, add sudoku to INSTALLED_APPS, include the URLs for sudoku somewhere in the mapping (an inclusion mechanism allows the inclusion of a bunch of URLs with a single pattern match, and ths feature extends to multiple levels). Finally use django-admin to install sudoku.

Tadaa! Once again an effective demonstration of a Python-based web technology that people should be adopting in droves.

Effective AJAX with TurboGears

Kevin Dangoor is the leader of the TurboGears project. This was the second in the session that I chaired. I was lucky in drawing excellent speakers for my session, and I was very glad I got the chance to hear this talk.

Do I need Ajax? Kevin felt the answer to this questions is "probably yes"! The web is becoming hugely interactive, and people are getting used to sites being much less passive than they have been. Gmail is two years old, and it has forced the pace. Nowadays 95% of the world uses one of three browser platforms, and toolkits are available to smooth over the differences (the two specifically mentioned were Dojo and mochikit).

Rather than present the dry theory behind AJAX Kevin then focused on the question of interest to many web developers: "How can I use it"? He gave several interesting examples based on TurboGears, but it would be easy to adapt them to other environments.

Data entry: this hasn't traditionally been a friendly task on the web. To choose between thousands of items a pick list is not appropriate: real-time searching is helpful here (the example selected from thousands of items by restricting those displayed in a pulldown to partial matches). People are used to auto-complete fields, so they quickly adapt to this behavior in web applications.

Realtime updates: AJAX can continually poll the server for updated information. This can capture simultaneous edits to the same data (the example informed an editor in real-time that someone else had changed the data they were currently working on).

Ordered lists: AJAX can be used to detect drops in drag-and-drop operations and reorder the data directly in the browser window (this time a very nice drag-and-drop example was presented).

There are also things we can do that confuse the users: people like the way their browsers work, and expect other applications to work the same way. Ajax reduces the value of the browser "back" button, and Dojo has provided a solution, but another solution is to not use AJAX! (I'm always encouraged when a speaker doesn't try to present their topic as a universal panacea). Users want to be able to bookmark the current state of their web application, and again Dojo can help to solve this problem by changing the location bar along the way. It's also possible to make controls behave in ways people don't expect: using radio buttons to submit information is generally a bad idea (examples of each faulty behavior were presented, as warnings of what not to do).

Summary: know your users - with a small user base you can train people if it helps improve their productivity, but this won't work with a large user base. Don't break what people expect to work, as they have invested a lot of time in learning the expected and customary interface behaviors.

With Guido's recent plea for more effective Python marketing I think this talk was a poster-child for what might be done. It focused on practical needs and gave clearly comprehensible examples, with the underlying code to diminish fears of complexity. If we can show people Python being used in this way they will flock to use it.

Everything About Web Programming (except Programming)

This session was the first of three where I did my PyCon duty and chaired a session. I was glad Ian Bicking's talk was included as I'd been unable to record his earlier session. Ian suggested that the focus of his talk was on "accepting my inner sysdadmin".

Imaginary Landscape is a small company with, like many small companies, vague division of responsibilities. Ian wears two hats: the programmer (who likes to write new things) and the sysadmin (who would rather redeploy well-understood software). Conway's Law applies: organizations that design systems are constrained to produce designs which are copies of the communication structures of these organizations.

The company got where they are via Zope 2 development - which "felt a lot like PHP", with no overall process. So they moved into Webware and Subversion - but again without an overall plan. Things moved into production without a definite transition.

Deployment was stressful, so they decided to stop deploying! Applications were "multi-customer", and rather than developing good tools they would build complex applications. Ian now realises this was dumb, though it seemed smarter at the time. Unfortunately it meant that since everyone got the same software customization was infeasible, and configuration data had to go into the database, with no SVN control. Deployment should be easier than that, and simple deployment obviates the need for multi-customer applications: each customer can receive their own code.

Paste is a toolset that takes advantage of standard Subversion layout, providing a skeletal setup.py file and database model and a basic framework of templates and internally-used metadata. Testing also starts with a basic model created by the tool. Functional tests are really important: even just knowing that you can access the root page of an application is a worthwhile test.

Sometimes it can be problematic developing web applications in a test-driven environment. Developing code without opening it in a browser sounds cool, but it turns out that it's too easy to overlook gaping holes in an application (like there's no link to a page that the test framework has been accessing directly). This level of purity turns out to be too extreme in practice.

Configuration data is essential to a project, and it's important to differentiate between program configuration as usedby programmers and application configuration as used by system administrators. The tool creates a template for the deployment configuratiom file. Client data is kept in separate repositories, not int he application, and controlled separately from the application code.

Deployment uses the buildutils, and installation is moving towards a two-stage process. The first step installs the configuration file and the second step sets the application up in line with that configuration. Setuptools has the ability to install multiple versions of the same product, but this turned out not to be useful, as the separation between the versions wasn't sufficiently clean.

Ian now feels that "the computer" is a very poor context for installation of anything at all. ("Site-packages considered harmful"). So each site gets its own Python environment in development, and they use sitecustomize code to configure aaccording to the ACTIVE_SITE environment variable.

A nicely-focused talk with some valuable lessons for us all.

RIP Meerkat

Well, I just discovered that the end of an era is upon us.

For a long time now the holdenweb.com site has had news links extracted automatically from O'Reilly's meerkat system. Because the site generation code is still under test I only periodically click a button in its interface that says "Update News", and today for the first time I got a traceback from xmlrpclib complaining about 302 server status.

The redirect (which should really be permanent) is to a page that says

Update: Meerkat was shut down March 2, 2006

No reasons, just that bald statement. Since meerkat was an early exemplar in the web services field I thought it would be appropriate to mark its passing.

Now I have to get serious about a replacement. I have for some time been tweaking a feed extractor using Mark Pilgrim's excellent FeedParser module, so that's now moving up the priority list.

The underlying database is still Access, and so this morning I used DBManager Professional to convert it to PostgreSQL. Still needs changes, but I'm happy to say that at least I managed to generate a local copy of the site with no errors.

Kudos to Kuchling

While I am on the subject of conference organization, I don't believe I have recorded what a terrific job Andrew Kuchling has done in putting PyCon 2006 together. I know from personal experience that the organization of this conference places huge stresses on the chairman, but Andrew has performed magnificently.

This PyCon has been better in so many respects than the three that preceded it. I also know from the chatter I've seen in the background during the preceding year that things are becoming rather better codified to assist future organizers. PyCon will continue to improve.

Well done, Andrew!

Wirelessless

Further to my original remarks about the network at PyCon, this has probably been the biggest complaint I've heard people make, and the post-mortem continues as wireless associations continue to fade in and out during the sprints. Considering that

a) The networking company is at least part-owned by Marriott, and
b) A five-figure sum was negotiated for the wireless coverage and Internet connectivity

the results were very disappointing. For the record STSN didn't have anyone on-site, and there was no evidence that they had sent anyone over beforehand to do any testing and verify that the network coverage would be robust when descended upon by a crowd of geeks, most of whom were expecting the same kind of coverage we got at GWU.

I've no wish to prejudice any negotiations here, so I'll content myself with saying that during the sprint startups, when we shared a room with the organizers' debriefing, I was reminded how difficult it is to get everything right when you depend on third parties for so much of it. This issue isn't over yet, and meetings will follow to pick over the carcass.

On Planet Python?

Someone yesterday posted that this was "the most underground blog of 2006". I foolishly hadn't thought much about how people might get to it, since like most blogs it's not intended as much more than a place to record random thoughts and impressions. Ephemeral is the word I was looking for (and no jokes about "any port in a storm").

Well, according to Andrew Kuchling this blog should now appear on Planet Python, so I'm making this posting just to see. That might get it into Google somehow.

Lightning Talks (1)

High-Speed Stuff, No Corrections

This is a stream-of-consciousness dump of the lightning talk sessions.

Big and Bold Slides with S5 David Goodger: With Python and docutils you can do large-slide presentations. Q: Can you print your slides? A: Yes, but the layout needs work. CSS anyone? Q: Can you include images? A: Yes, of all kinds? Q: Does it work with IE? A: Yes.


Testing Web Apps with Twill and ??? Titus ??? Twill lets you browse to web pages and fill out forms. Very engaging interactive demonstration. Nice humor: "...and of course the great thing about Python is there are so many different web frameworks you can test". I think I'm going to try twill. Q: Will that work with mod_python? A: It works client side, not server side.


Building MerchantCircle.com David Creemer: They are building a CheeryPy-based online marketing tool for small local merchants. Lots of Python in the application, so thanks. Found it bery good to develop in the production environment. Found FormEncode, Routes, PostgreSQL and Tsearch2 very useful. Avoid magic, be(come) a DBA, hire the best and never stop cleaning. Q: Did you use embedded Python stored procedures? A: We've played with it, but no. Q: Would you like something similar to routes in CherryPy? A: I'd love to see that.

Python Argetina Facundo Batista: We are a user group, and have a portal web site, recently migrated away from Plone to MoinMoin. We have developed a stable workshop in a national university, and given courses. We have T-shirts for sale. We have meetings, talk a lot and drink a lot! We are planning a national meeting to allow people from the different provinces to meet each other, and want to organize a job board, talks in various cities, and keep reaching out to Argentinian users. Q: How small are your smallest T-shirts? A: Medium, but htat's smaller that your medium! (laughter). We don't have XX (more laughter). Q: What other languages are the universities teaching? A: C++ and Java.

Chandler ??? Chandler handles calendar events now, shortll appointments and to-do lists, eventually email. It also lets you share information via a WebDAV repository. 0.6.1 is the latest release, and the speaker is using it live although there arae some rough edges. Built on wxPython and twisted. The GUI allows multiple calendars to be overlaid, which is neat. An event can occur in multiple calendars. Eventually they want Chandler to incorporate many extensions from the Python community. Q: License? A: It's an open source project, with an open license.

How to Replace Yourself with a Small Bot Ben Collins-Sussman After five years of daily IRC use the speaker started to use ircbot.py and generate bots of various kinds, starting with pinkybot and wolfbot. He moved to work with Google, and realised that a bot could take his place in the various communities, and sussbot was born. It does nothing except respond with useless platitudes such as "have you checked the FAQ?" little happened, but some users did undertake "serious" interactions (hilarity). Q: URL?A: http:// svn.redbean.com/repos/ircbots/ Q: IM version? A: That'd be fun.

py.execnet: simple ad hoc networking Holger Kriekel Standard models for remote execution are cumbersome, require servers to be started ahead of time, and complicates programming. py.execnet requires no installation on the server side. You define your protocol on the client side and deploy it from there using a gateway via an SSH login or similar channel to a host with Python installed. Interactive demonstration showed /etc/passwd from a remote host. Great for propagating viruses ;-). In a controlled environment has been extremely useful for administrative tasks, and multiple scripts have been developed for tasks like remote sync. http://codespeak.net/py

Help with Jython Generator Expressions Chio??? I am trying to expect generator expressions in Jython. Sometimes it works, but sometimes it fails. If there's someone who knows Jython internals or AST would be able to help mne. Please!


Using the Web as Your Application Platform ??? We are taking REST to extremes. tasty, our tagging service, is a layer that provides tag search results as Python dictionaries. We have focused on small components, and have found this is a fruitful design approach. http://tasty.python-hosting.com.

Spec David Byng spec is a module we have been using to describe sets - think of it as a type. Allows checking for membership in sets, describing and checking attributes. Various set-theoretic primitives are implemented as functions. the proper() function can be used to verify that an instance agrees with a set of property specifications. http://www.mems-exchange.org/software/qp/. Q: Did you look at spyce(??)? A: It's kind of similar, but different enough to be necessary.

A Configuration Database Moshe Zadka The database has to be updated by all kinds of system components, and be available to many processes. A configuration wholuld be a list of actions, saying what the user did to get to the current state. We have a schema description language. This requires very little code. People are doing similar stuff with prevailer.

Another session tomorrow!!!!!

Implementing IronPython

Again another full talk, this time from Jim Hugunin, author of both Jython and IronPython. Jim summarised the state of the CLR, pointing out that it's now a standard. Jim enjoys seeing the IronPython download link on a web page with the Visual Studio icon on it.

The last year has been spent on ensuring that IronPython can pass as many as possible of the standard Python regression tests. Certain tests have been modified, because they test implementation optimizations from CPython. There are some flat differences (like IronPython doesn't use refcounting). Some are arguable, such as the use of Unicode attribute names, which people do expect to be able to do in the CLR world.

The structure of the IronPython compiler is now (since CPython's AST branch was implemented) closely aligned with the CPython compiler, but of course it produces IL operations for the CLR rather than the Python bytecodes of CPython. Python bytecodes are simpler than IL bytecodes, byt there is a significant speed gan from the CLR's use of JIT compilation into machine code. Not every operation can run 100 times as fast on IronPython, but some operations become a single machine-code instruction. Jim wishes he could get that speedup for all of IronPython.

He demonstrated some of the tools you can use to examine the generated code, unfortunately without identifying them. It seems relatively straightforward, which will surely be helpful to other implementers and more advanced users. Rather than keep a complicated line number table, no-ops are inserted into the code. Jim contrasted his code generator with that of the C implementation, and I was impressed with the simplicity of the code generation code.

Jim wanted to be able to claim that IronPython used a "simpler" object layout than CPython - you might expect that because no reference count is required in a true grabage collection environment. Unfortunately they have had to give each object a 32-bit "synch block" that holds arbitrary data to assist various implementation features, so integers are 12 bytes (presumably for 2-bit machines) in both systems. List representations are remarkably similar, but IronPython does seem to save a little space there.

The type pointer is different for IronPython, in an attempt to maintain performance. The first sensible way to proceed is to wrap every external object with a subclass of some sort of Python PyObject (this approach was taken by Jython). This requires a lot of wrappping and unwrapping during method calls. The other way, used in IronPython, is to used a "pure object model", allowing the underlying objects' methods to be called directly and the returned values to be returned directly.

Subtyping a builtin class permits changing the type of an instance by changing the instance's __class__ slot, but the type as perceived by the CLR cannot change.

Fields and proerties of CLR objects are closely parallel to descriptors. Overloaded methods analyse their arguments to disambiguate their signatures. Constructors are converted to __new__(). And so on (Jim was goung fast here). The Python object model turns out to be surprisingly convenient for implementation on CLR, which Jim felt was a tribute to Guido's design skills.

Jim demonstrated an interactive session in which he created a window in the presentation framework. He then defined a CallMe class and bound an instance of the class to a button's click event. A panel subclass was defined, and an instance rendered. Jim's bravery in performing live reali-time coding is much to be admired, and says a lto about his confidence and knowledge. Overall a very convincing demonstration that IronPython brings true Pythonicity to the CLR environment.

Jim closed with an interesting discussion about compatibility. Should IronPython expose CLR native object methods as methods of Python objects? Both sides, when Jim talked to them, felt that their conflicting opinions were the only obvious answer to the question! His attempt to please the CLR folks has led to the development of a clr module. import clr changes method call semantics change to expose the CLR methods. Jim had a lot of pushback from the CLR world not to use the from __future__ notation for this feature. The fortunate difference in naming conventions between the two environments means that there are very few conflicts.

Exception handling is another are where the two environments conflict. There are two class hierarchies to be resolved. If you try to catch a .NET exception then the corresponding Pythnon exception will be converted.

Wow! Now I have to get lunch ...

Implementation of the Python ByteCode Compiler

My primary interest in this talk was hearing Jeremy Hylton speak. Jeremy is a formidable intellect, and has made many fine contributions to the Python core. It turns out he is alsi an excellent (if somewhat rapid-fire) speaker.

He introduced the abstract syntax tree notation, showing how it contrasted with the previously-used concrete tree, then went on to look at compilation strategy.

The first pass builds a symbol table, and collects certain information (for example, if a function contains a yield then it's a generator). A second pass determines the scope of the names, with some unexpected differences between implcit and explicit global variables (whose treatment differs, for example, in the class namespace).

Free variables must be treated differently from locals, because their lifetime can extend beyond the activation record of a function call. There are many fine points to be considered in the handling of names, and some of Jeremy's examples were chosen to stress the syntax. Sadly they zipped by so fast you'll have to read the slides for the details, but you can consider his first example

class Spam:
id = id(1)

as typical.

Jeremy's treatment of code generation was concise (he clearly wanted to cover a lot of ground, so he wasn't hanging about), but I did at least understand that the output of an AST tree transformation was a set of basic blocks, pointers to which would be jump targets. Then the code generator iterates over the contents of each block, emitting the bytecodes for each AST type encountered, essentially as a visiting tree walk.

An assembler then linearizes the resulting code blocks, computing the required stack space and the required entries in a line number table (whose construction is horrendously complicated). Jump offsets can be deduced at this point, and then PyCode_New() is called to create the object. Finally the peephole optimizer cherry-picks the resulting code, performing constant folding and simplifying certain jump sequences down to a single jump.

The AST branch has simplified several aspects of the compiler implementation, but it also has potential value in other areas. If the AST code could be exposed to Python applications, for example, tools like pychecker could use the tree rather than the source. The application interface has not yet been determined, and since the core currently contains code to handle the concrete syntax tree there will be library ramifications, especially in the compiler package.

Q: If an application uses the AST, do the nodes make the original program text available?
A: For names, yes, but for example floating point objects have already been converted. Statements and expressions refer to their line numbers.

Q: Can I reproduce the source of a program from the AST?
A: That's something that we need to work on to support applications like a refactoring tool, for which the current implementation would be insufficient.

Q: LOAD_NAME only appears to be used in exec and class. Is it used anywhere else?
A: I don't know. [Compiles code and disassembles it]. Ah, yes!

Q: Has anyone considered [inaudible mumble].
A: I don't think so, but that would be a more complicated but more traditional way to do it.

Q: Can you talk about performance a little?
A: Yes. Thanks to Neal Norwitz, Neal Schemenauer, Brett Cannon and Tim Peters the new compiler is marginally faster than the old. Our primary goal was to write comprehensible code, however, so there may well be further gains. For example the peephole optimizer should really be run before the assembler.

Not really sure whether this is any use to someone who wasn't there!

State of the Python Universe

Since many people will be commenting on this presentation, I'll see if I can't just try to summarise for those who want the headlines without having to read the whole story.

The Move to Subversion - the main reason for the move was the slowness of SourceForge. [Apparently Guido got a message yesterday to say that SF do now have Subversion available]. I wonder how many projects that didn't have the resources to move to their own SVN repository are still struggling with delays before their commits appear in the anonymously-available versions. Maybe SVN on SF has helped.

Buildbot - with the introduction of this system we expect to see big improvements in the detection of single-platform errors and checkins performed without passing tests.

The Cheese Shop - has been a great step forward in Python non-core code availability. More people should be using it to release their code.

www.python.org - would you like to be a python.orb webmaster? We are looking for helpers, and the new design was specifically intended to make content management easier. Email to pydotorg@python.org if you'd like to help.

Python 2.5 release schedule - expect to see 2.5 around September 30. There will be a 2.4.3 bugfix release before the first alpha of the new release, and a final 2.4.4 (barring real security problems) after the 2.5 dust has cleared.

The import mechanism will change, introducing a mechanism to specify import from a position relative to the current package.

The new conditional expressions have a slightly quirky syntax, and it sounds as though Guido has made this choice to discourage its over-use. The essence of his remarks was that it's not Pythonic (though he didn't use that word) to write too many lengthy expressions spanning multiple lines.

It will become possible to write exception handling constructs with try ... except ... finally rather than having two nested constructs. One of the advantages of Guido's time spent in Java-jail is that he realised after reading the Java documentation that this does not introduce any ambiguity.

Generator semantics will change to allow the yield keyword to return a value rather than introducing a statement. This value is returned by calling the generator's .send() method. It's actually possible to cause the yield to raise an exception by calling the generator's .throw() method. Generators also have a .close() method that raises a GeneratorExit exception. This new method will be closed when the generator is garbage-collected.

The with statement is intended to ease the coding of critical sections which require both initialization and clean-up. with is followed by an expression whose value's __context__() method returns a value that has .__enter__() and .__exit__() methods. The expression is known as the context manager and in the majority of cases __context__() can return self, allowing all three methods to be defined in a single class. Some context managers do need to be independent, however, which is why the semantics were defined this way. Certain standard objects will be redefined as context managers to allow their use in with statements, the major ones being mutexes and lock, decimal objects and files.

Exceptions are finally going to become new-style classes, with the inheritance tree rooted at the new BaseException. The end of string exceptions is now in sight. This may eventually lead to a bare except clause becoming illegal (sensible, because this forces you to use an interface in the sys module to access the exception). User-defined exceptions will continue to use Exception as their base class, but KeyboardInterrupt and SystemExit will no longer be subclasses of Exception, allowing cleaner handling of the mainstream exceptions.

Types will be able to define an __index__() method, guaranteed to return an integer or long value. This has been introduced principally to meet the requirements for slicing of complex structures, where the __int__() method isn't appropriate.

The compiler has been revised to create an abstract syntax tree rather than a concrete one. This was a development that almost didn't make it, after much work. Only after Guido said "let's forget it" did the developers realise that they didn't want to lose results of the hard work and finally integrate it into the trunk. This has led to improvements in the maintainability of the compiler, allowing developers to make experimental syntax changes more easily.

Other highlights - Pretty much invisible to users, but invaluable for C developers on 64-bit machines, is the new ssize_t type. This will allow strings (and even lists!) to be longer than 2G elements. The -m command-line option will now allow you to run package submodules as well as top-level modules. This has already proved its value in running regression tests.

Python eggs support is at least on the horizon, making simpler installation a practical possibility for the average Python user.

A new dictionary object, collections.defaultdict, will allow much more flexibility and efficiency for certain dictionary applications.

lambda will survive beyond Python 2.x because "it's perfect" (loud applause). Because it's perfect, no changes will be made.

Q: Will Python access to the AST allow, for example, the implementation of LISP-style macros?
A: I suppose it could be abused for that. (AMK, after galloping to the microphone: The AST BoF session will be discussing this topic at 3:15)

Q: Was "exception filtering" considered?
A: No

Q: Can you add a __missing__ method to a dict?
A: No, builtin types can't be modified in that way.

Q: What happens when two generators' finally clauses call each other?
A: I have no idea. There's a fair chance that it will work, but pathological cases can doubtless be constructed. Don't try and play games with this: you won't even understand what it means yourself!

Q: Can a defaultdict's factory function access the key value?
A: No. To pass the key to the factory function would slow things down. You'll have to override __missing__ in a subclass to do this, as the standard implementation is focused on the common use cases.

Q: How do you like working at Google?
A: Ask me during the break. I love it!

Industrial Light and Miserliness?

Does LucasFilm Division Have Hidden Financial Troubles?

I guess you don't get rich by giving money away. That, anyway, is my conclusion after a little research into the personal and corporate finances of George Lucas, scion of the film industry. Forbes magazine regularly assesses his personal wealth at $1 billion plus, but one of his companies appears to be suffering unacknowledged financial hardship.

Last May George's company LucasFilm moved to a new $350 million headquarters building at the Presidio, allowing them to "be more efficient and leverage assets, technology people and innovations that come from every corner of the company", as he apparently said at the time.

According to industry analysts the difficulty until then had been that the separate divisions of the Lucas empire were located at disparate locations in Marin County. According to LucasFilm's President, Micheline Chau, "Marin has been really tough on us, we ran our businesses as silos". Since July the effects and animation division, Industrial Light and Magic, has joined the other LucasFilm divisions at the new headquarters.

The advantage of this move is that LucasFilm can now use the same assets they created for a film to produce the spinoff game and other ancillary products. This can be big business - it's reckoned that even running as separate silos the 2003 turnover at LucasFilm ran to $1.2 billion, which certainly lends a new meaning to the word "tough". Estimates are all we have, of course, because private companies like LucasFilm don't have to publish accounts or have their books scrutinized.

It's a good job that the company appears to be doing well overall. Yesterday's Python Software Foundation members' meeting reminded me that two years ago we were asked to vote to admit Industrial Light and Magic as a sponsor member of the Foundation (sponsor members pay annual dues, currently $2,000, to help fund the Foundation's growth). Their membership was approved by 42 votes to 1, mine being the sole dissenting voice. This was not a serious attempt to block ILM's membership but primarily because I felt that a company like ILM could have been doing much more to support the Foundation.

I have heard (from staffers I have met at conferences) that the company employs in excess of seven hundred Python programmers. Here's what Tommy Burnette, ILM's Senior Technical Director, is on record as saying "Python plays a key role in our production pipeline. Without it a project the size of Star Wars: Episode II would have been very difficult to pull off. From our crowd rendering to batch processing to compositing, Python binds all things together." Philip Peterson, a Principal Engineer for R&D adds "Python is everywhere at ILM. It's used to extend the capabilities of our applications, as well as providing the glue between them. Every CG image we create has involved Python somewhere in the process".

This is clearly an organization that makes extensive use of Python, and one imagines it would profit by it (as the PSF hopes all users profit by their use of Python). Sadly this appears not to be the case. When you look at the PSF members roster, you may wonder why you see ILM listed at the bottom of the page as an emeritus member rather than a sponsor member.

This is where my incisive intelligence tells me the company must have been through bad financial troubles. Shortly after the membership voted to accept ILM as a member a PSF director received a 'phone call from our contact there explaining that they could not find $2,000 in their budget to pay their membership dues. Since by that time they had been elected to membership our only choice under the existing constitution was to convert their membership to the non-paying non-voting emeritus status normally reserved for retired and inactive individual members.

I'm not one to bear a grudge, and I hope that ILM's financial position has improved since then to the point where they can not only afford the fees to become a sponsor member, but also provide Platinum level sponsorship for PyCon like Google, ITA Software and EWT have done this year. I have a suspicion that if they looked really hard they might find that they had good reason to help the Foundation that Guido van Rossum created to administer the intellectual property that he and the other Python developers have created. We need more supporters with the experience and clout of Industrial Light and Magic, and I think we should welcome them into the fold.

These ponderings also raise a larger question: since FLOSS (Free/Libre Open Source Software) is produced and distributed essentially at no cost, what (if any) obligation do its users have to support the development process? Clearly there's nothing that can be done directly to force any return on development efforts, and most open source contributors I know would rather be wrting code (or even documentation!) than spending time policing payback. Many open source users contribute effectively to open source projects by having staff work as developers, so their work is incorporated into the intellectual property of the project. Others prefer to donate to supporting foundations and similar bodies.

If you think about the larger context of open source, though, there's a sea change afoot. In today's world the proprietary model dominates most of the software market, but open source is gaining ground. Large companies have responded to the change in various different ways. There's almost a paradox about open source development in a capitalist economy: if everyone uses open source software without taking steps to ensure that a project keeps going, it will falter and die. You might also argue that this is also quite Darwinian: if a project is sufficiently important to a sufficiently wealthy section of the user universe, it will prosper and grow.

I don't think anyone has a clear idea of where this is all going. It will be interesting to see what eventually transpires.

[Steve Holden is a member of the Python Software Foundation Board]

Understanding Unicode

David Goodger set himself the possibly ambitious goal of dispersing unicodaphobia, which he defined as fear of Unicode and all the words surrounding it. Unicode is intended to be a cross-platform standard applicable to any program and any language (possibly even including Dwarvish and Klingon in the future). It is currently defined by ISO 10646. What follows is my attempt to record my developing understanding during the presentation.

The basic element in Unicode is the character: a character set takes a subset of the characters and maps each one to a code point, a numeric value associated (in that particular character set) with that character. There are a number of 8-bit character sets that all associate the ASCII characters with the code points from 0-127, but mapping completely different characters onto the upper half of the code points.

Unlike Western alphabets, many other languages have large characters sets that therefore cannot be mapped onto only 256 code points.

An encoding is a specification of the code points mapped to each of the characters in a character set. An encoding will be performed by a codec. Encoding a Unicode text maps each character in a Unicode text into one or more bytes in a representation. Decoding is the converse process. in a Python program the internal representation might use 16 or 32 bits per character, depending on the build configuration.

Whatever the internal representation of the Unicode text, text that is read in must be decioded from its external representation to the internal representation, and when a Unicode string is written out it must be encoded into a suitable encoding (but not necessarily the same one that was used to read it in). ASCII is an encoding, as is Latin-1 throughLatin-N and Windows-1252; all thses encodings are ASCII-compatible because each code point is a single byte containing the ASCII set on code points 0 through 127. UCS-2, UCS-4 and UTF-16 are all encodings that are capable of representing all Unicode characters. Because they use multi-byte code points they aren't ASCII-compatible.

The new default encoding is UTF-8, which is ASCII-compatible because the characters of the ASCII set still map onto code points represented as single bytes with values of 1 through 127. It is capable of representing all Unicode characters, because it uses multi-byte representations (from 1 to 6 bytes in length) for the non-ASCII characters, and none of the bytes in those representations are in the range 1 through 127. The first byte of a non-ASCII character is always in the range 0xc0 to 0xFD, and the second byte will be in the range 0x80 to 0xBF.

For this student the main failing of the presentation was the complete absence of any graphical materials. Some of us understand pictures better than just words. I was left with the feeling that my understanding of Unicode had increased, but I couldn't necessarily say I can proceed confidently from here on in. I wish the presentation hadn't overrun its time, as the noise of delegates departing early distracted from the later slides.

Plone IS Easy to Install

It turns out they weren't lying. I downloaded the Plone installer, and now have an instance running on port 1080. Way to go, guys!

Using Django to Supercharge Web Development

A nicely judged talk with pleasantly humorous diversions.

Django is intended to enable the creation of in-depth web sites such as washingtonpost.com, and Adrian Holovaty gave an example (lawrence.com) which certainly proved the point. He explained that another project had developed a web site to process reader feedback on the presidential debates ... in four hours! Since the roots of Django are in the newspaper world, they remind themselves that "the ink is never dry" on the web, and sites are subject to continual development.

After driving themselves stupid with PHP they discovered Python, fell in love, and set themselves the goals of:

1. Making web development stupidly fast
2. Automating the repetitive stuff
3. Practising loose coupling
4. Following best practices
5. Being efficient

An early (three-day!) project involved taking the local little league and building a web site featuring much of the same information available about major league teams. The kusports.com sites allows subscribers to sign up for baseball stats as short messages tot heir cellphones. Many other examples provided convincing evidence that Django is indeed versatile.

Last year (at PyCon 2005, when Djanfo was about 18 months old) Adrian gave a presentation on what was then "the CMS", and received a lot of feedback that suggested an open source distribution would be popular. Django is built around:

A URL dispatcher
A database wrapper
A template system
An admin framework, and
"Other pain relievers".

Django URLs are designed to be pleasant and readable. The scheme allows the programmer to write Python code that directly maps the URLs to program callables. This decouples the URL from the business logic. The callables, known as "views" in Django terminology, take a request object as their first argument (possibly with values extracted from the URL as additional arguments) and returns and HttpResponse object.

The usual development model is to design the database. Django helps you to create and populate the tables (though it doesn't stop you from using SQL if that turns out to be easier). The design process also results in the automatic generation of admin functionality that can be used imeediately by editorial staff to manage content. The Django team is fortunate to have a designer to build great interface pages.

The initial design is now production-ready, and the team can write the views and templates to produce the required pages. The templating language was deliberately kept simple (avoiding XML and programming languages) to maximize its utility to the designers.

The framework is fully ready for internationalization (the first add-on contributed to the project after it was open-sourced), and provides three different levels of caching: per-site, per-view and low-level (this latter being performed in raw Python).


Repeatedly-required view functionalities have been abstracted into "generic views" that are automatically available to developers with no further formality. Sites using Django include chicagocrime.org (an award-winning site that integrates with Google maps), washingtonpost.com (many feeds are provided by screen-scraping Python scripts), the Ellington system (an online publishing system for news and entertainment sites) and traincheck, a site you can SMS to find out when and where to catch your metro.

The 1.0 release is approaching, with much work upcoming in a post-Python sprint, and a book is expected later in the year from APress.

A nice final touch was the announcement of the availability of t-shirts featuring the slogan

I wish I were
powered by
Django

It was great to see enthusiastic and capable developers using Python for pragmatic purposes. Django does not attempt to be a fully-integrated content-management framework, but a loosely-coupled collection of components that work extremely effectively together to provide extremely efficient mechanisms for buildding database-driven web sites. Go to djangoproject.com and take a look!

The State of Dabo

Ed Leafe is an engaging speaker, and the principal developer of Dabo, a desktop application development tool (which, as he says, is a novelty nowadays).

Design goals for Dabo included database independence, UI independence (they chose wxPython as their base for its clean look, maturity and portability) and a simplified API. The GUI development aspect of Dabo has led to rapid takeup by wxPython users without database needs, but the primary target is the world of developers with Visual Studio or Visual Basic experience. These people need to see attractive development platforms with similar capability levels before they will feel comfortable migrating away from the proprietary world.

Since last year Dabo has added a report designer that enables visual design; the same principles are being applied to class design, and there's a new editor. Ed gave a quick demonstration of application building, explaining that Dabo assumes you already have a data repository, and that you want to run applications against it.

The report writer creates PDF files from an XML report description (and some data, of course). The report designer isn't (yet) hugely sophisticated, but it's better than much of the competition in the open source space. Like the rest of Dabo it''s cross-platform. The GUI editor has come on immensely in a year, and is capable of producing versatile layouts without the heavy lifting required by wxPython when used directly.

Dabo doesn't use any object-relational mapping layer, as the developers are "traditionalists".

A nice presentation of a solid project that's obviously continued to receive a lot of attention from its developers in the last year.

Impressions from "Python for S60" by Matt Croydon

Jurgen Schieble has wriiten an awesome tutorial.

PyS60 provides an application user interface layer, which applications can leverage to reduce code size and easily interact with users, as the appuifw module. The development environment makes it really easy to test code on a real device rather than in an emulator.

While the whole standard library isn't present a useful set is provided, and many pure Python standard library modules can be added and will run wihtout change.

Specific libraries offer telephone-based functionality such as

image = camera.take.photo()

telephone.dial('411')

You can manipulate your contacts and calendar easily from Python.

The PDIS team have added many libraries, and Nick Burch has built a webbrowser.py.

Earlier applications tended to be quick hacks, but the more recent batch of applications are really polished. Traffic cam viewing, stock quote grabbing dictionary lookup and weather display were among those mentioned.

Installation sounds really easy.

The S60 world is excited about Nokia's recent release of the Python implementation as open source, and applications are rapidly getting more mature, with an active community pushing the technology forward. Lots of sample code is available online.

Slides for the talk are available at http://postneo.com/talks/pycon2006/

Some amusing and interesting ideas from the keynote "Plone: It Ain't About the Software" by Alexander Limi and Alan Runyan.

The $4 million company that operates by emailing Excel spreadsheets around. Not intranet, just a single shared drive where one user can delete another user's files to make space for their own.

The idea of the "J2EE Liberation Front". Those guys need a serious dose of freedom. [I gave up Java a long time ago because it seemed too like a straight jacket].

The Plone world has taken translation issues seriously, with the base system now available in 50 different languages. This issue isn't yet taken seriously in America, but the more dependent the US economy becomes on foreign markets the more attention it will get.

They claim that Plone was easy to install, so I shall be testing that assertion. It's a long time since I played with Plone (back in the days when the Python Business Forum decided to use it as their main web presence), and it sounds like it's a completely different deal now.

I really enjoyed the story about the company that had bought an in-house training class, and allocated the two spare seats to members of their janitorial staff. At the end of the course the cleaners were happily writing Python!

I think the speakers started out nervous, but they hit their stride halfway through (PyCon audinces are pretty friendly) and made some interesting points about how frameworks by themselves have little value. What people pay for is a tool that will help them communicate their ideas more successfully. They were realistic about Plone, admitting that the insides were ugly in places, but that from the outside Plone is a usable solutions for many content management needs.

Q: "What's new in the upcoming 2.5 release, and are users demanding AJAX functionality?"

A: Introduction of more Zope3 components as part of a gradual migration. AJAX will be a 3.0 feature. The developers aren't yet decided on the kind of AJAX framework they will use. Sites with usability concerns, such as government departments, are unlikely to use AJAX anyway.

End of the keynote. There are signs that the wireless network is under stress (hardly surprising with the PyCon audience). STSN, the wireless provider, have never had to cope with this density of demand at the venue, so they have sensibly put a couple of staff on-site to learn how their infrastructure reacts to this load and provide help as necessary. Still, I can get through to Blogger to post, and despite occasionaly dropouts I am receiving email, so I'm pretty sure that connectivity will be OK.

Next for me (after coffee) is Python in Your Pocket: Python for the Series 60. Last year's talk by Erik Smartt was one of the many I had to miss. Since the BDFL now claims to have a Python program answering his cellphone maybe I'll have to look at a change of technology myself.

PyCon Has Started!

Well, the tutorials are over, for better or worse. If anyone who attended either (or both) of "An Introduction to wxPython" or "Using Databases with Python" happens to read this I'd be very interested in your feedback. There was no formal process for capturing attendee opinions, so I'd really like to know what went well and what could be improved.

During the runup to the conference there was some debate about whether tutorials would be worth organizing, but I think the attendance figures have left little doubt that the community's opinion is in favor.

As I sit here I'm listening to the PyCon 2006 keynote by Alexander Limi and Alan Runyon, and enjoying the rather strange experience of not having to do anything to help with the organization. After three years it's a huge change to be able to attend sessions and just chew the fat with people I meet every year. Signs are that Andrew Kuchling and the support team local Texas user group have done an excellent job.

More later, when the keynote's over.

PyCon is Here!

After (yet another) horrendous journey (I'm starting to think the airlines have a personal grudge against me) I have managed to get to the Addison Marriott Quorum in time to get dinner and have a good night's sleep before I present my tutorials on databases in Python and wxPython.

Since this is the first year I won't have been chairman it'll be interesting to attend as a regular delegate, and get to go to more of the presentations.

A Very Poor Map Indeed

In preparation for PyCon I thought I'd take a look at a map. I lazily typed "Addison, TX map" into the Google toolbar, and was mildly surprised to see a Yahoo map as the first link. After I looked at the map I suspected Google were doing it so people would stop using Yahoo maps!

Talk about not being able to see the wood for the trees ... I still don't know where Addison is in relation to Dalas and Fort Worth. So I'm off to (where else) Google Maps to find out!

Polite Incompetence

I occasionally keep in touch with various non-technical blogs (if only to pretend to myself that I have a life outside of technology). One such blog is that of Seth Godin, who appears to concern himself professionally with marketing. Seth's current latest post is to be found at

http://sethgodin.typepad.com/seths_blog/2006/02/hearing_listeni.html

and said piece ends with a heartfelt question:

The folks at Eata Pita don't get the Hearing, Listening, Action scenario.

You get it, it's obvious. So why don't they?

Well, there's a good reason why they don't: it's a lot cheaper to keep the payroll down and provice a crappy service than it is to take the effort to supply your customers with what they actually want. Most consumers are so focused on achieving the lowest possible cost of acquisition that they don't critically evaluate whether what their money buys them is what they really want or need. The remaining five percent are vocal about what's wrong with what they have received, but only after they have received (and therefore paid for) it and are thus stranded without leverage.

This puts me in mind of my erstwhile neighbor Simon Hacker's theory of "polite incompetence". The US (and specifically the Virginia) employment market is currently so under-supplied that employers find it profitable to employ someone who clearly does not possess the skills to provide the goods or services that the employer contracts to provide. As long as the incompetent employees are polite enough, and apologetic enough about not being able to give the necessary service, that they avoid giving offense to the customer their employers, incredibly, manage to stay ahead of the game.

Where once we would have voted with our feet (not to mention our wallets) and gone to some other supplier, we now apparently choose to suffer the slings and arrows of outrageous (but inoffensive) incompetence rather than tell the people we deal with that they clearly do not know what they are talking about and take our business elsewhere.

Adam Smith must be turning in his grave.

I am happy to say that there are still bargains to be had if one bothers to look, and among these I count the Complete Care contract for my Dell Inspiron 9200. The system is now just over a year old and until a month or so ago had been operating without missing a beat. Lately it has developed an intermittent fault which, when present, appears to replicate the same scan line across the whole of the screen. The intermittent nature of the fault meant that I have been unable to collaborate with Dell's fault-finding process but they have noentheless already replaced the video card. Since the fault is still present I expect to see another engineer (but this time in Boston: the last fault call was handled in Philadelphia).

Several people have enquired what I paid for this laptop and, when I told them, have expressed the opinion that they could have bought it for less. Which indeed they might for all I know. As far as I'm concerned the Complete Care contract is non-negotiable, and terrific value for money. Without the contract I might have been compelled to part with my system (possibly several times) to allow the Dell support group to repair it. The on-site repair alone is worth the cost of the maintenance contract. Had I not been supported in this way you would not now be able to read this blog (no exaggeration).

So, do you think I'm going to buy some Brand X laptop next time around? I seriously doubt it. Thank you, Dell! See you when I need a new laptop (and hopefully not before :-)

Blender User Interface Tutorial

Wanting to add a little spice to a talk I have to give I decided I'd investigate Blender, the Python 3D rendering application. The idea was to see how easy it would be to create a modest 3D animation. Blender has a reputation for quirky interface so I was intrigued to see how quickly I could get started and expected something in the way of frustration.

I'll ignore download and installation as I'd done it some time ago, and I suspect it must have been easy because I don't remember any problems. I'd played with the program a couple of times and given up because its behaviour seemed so counter-intuitive, so this time I decided to work through Bart Veldhuien's Blender User Interface tutorial, two clicks away from the web page that came up from the Help menu ("can't be that counter-intuitive if I can find the tutorial," I thought).

I noticed that at the bottom of the page it said "This page does not have a maintainer. If you would like to help, please read 'Getting Involved'" and if you've already followed that link and it hasn't changed you will know that a charitable instinct is rewarded with the news that "We have received many offers for help lately and we need some time first to get everyone going." followed by an invitation to join up for the website editors list. Maybe later, I thought. For now I'm just going to critique each section of the tutorial as I work through it.

Introduction explains the motivation behind the creation of the document - apparently Bart was himself intimidated by Blender at first and took several weeks to to discover the basic princiles behind Blender's interface. Well, I'm certainly all in favour of saving several weeks, so I ploughed on.

What am I looking at? is a walk throught he empty blender scene, explaining that the stripe across the top of the window is called the info window. This has the 3D window underneath it with objects representing the standard plane (whatever that may be), the camera and the lamp with a 3D cursor. At the bottom is the buttons window, apparently used for changing scene characteristics. I read that each window has a header, and that the header of the 3D window is at the bottom. I'm not sure I need to know how to reposition these headers at this stage, but I learn that. I am told that the Icon slider button can be used to select a window type, and wonder why it isn't called the window type selector button. There's a lot of that kind of stuff going on.

Configuring the screen your way tells me how to change the size of the windows, and to split windows (thereby creating new ones) using the middle button (which, since I only have two buttons, I fortunately know how to emulate by chording the right and left buttons). I accidentally activate the "No Header" feature of one of my new windows, and in trying to put the header back I remove the header on the window next to it. This makes it difficult to use the information that the window having the focus is displayed with a lighter-coloured header. I manage to get the headers back, and use CTRL/up-arrow to "make the current window full screen" (which actually makes the current window occupy the whole of the Blender window - couldn't they have chosen another name for their "windows"?) and return the view to multi-window. So now I have four 3D windows described as "the traditional four-view window" (so are windows really views?), though unlike the illustration my 3D views are all showing pretty much the same thing.

The Toolbox - Adding a sphere explains that Blender functions can be accessed by either keyboard or mouse, and that the toolbox appears if I hit space or click on "the toolbar icon at the top right of the Blender screen" (so am I dealing with views inside windows or windows inside screens - I wish the terminology were more consistent here). Unfortunately there's no picture of the icon, and a three-minute hunt where I click on almost everything doesn't reveal it, so in the end I just hit the space key. I'll find that darned icon later. I discover that the space key only works when a 3D view or window has the focus. I am told to select Add, then Mesh from the menu that appears, then UVSphere from the final menu. I do this and then apparently move the mouse too far, because the resulting popup, which is supposed to allow me to select the number of segments in the sphere, disappears before I can use it. Now for some reason my toolbox "Add" item no longer shows "Mesh" as an option, the Mesh item appears in the toolbar but the shapes I can create are hanging off the "Add" item. Boldly ignoring this menu transmogrification I finally manage to insert a sphere into my 3D window, and it gratifyingly appears in all four views.

About Edit Mode explains that you can operate on an object as a whole, or on the individual vertices of an object. My newly-created shpere appears to have about four hundred closely-spaced vertices, and I wonder how to select just one or two of them. It mentions in passing that the G, R and S keys translate, rotate and scale respectively. I guess there must be some really important other operation that begins with T. Undaunted, I attempt to G, R and S my new sphere into something I can work with. I learn by experimentation that rotation is a two-phase operation where you first rotate in one plane and then in another. I realise that am wrong, and that rotation actually appears to be much more complicated than I had thought. My sphere is now at least large enough that there is space between the vertices. I fail to learn how to select individual vertices, but this may not be a bad thing.

Loading and saving your work optimistically begins "By now you have probablky created a Blender scene", which I suppose I have even though it only has a single sphere in it. It suggests I might like to save my work, or take a look at some of the files that come on the CD that comes with the book (which is annoying, because of course web pages don;t have attached CDs).

Loading a Blender File tells me how I can load up a new file. There isn't anything that I cna see to load, but it's obvious from the dialog that Blender's approach to opening files is just as quirky as everything else appears to be. I discover by accidentally selecting Quit that Blender isn't one of those applications that will warn you about unsaved work before quitting, and lose everything I've done so far. I restart Blender to see what I can remember. I see that Blender has forgotten about that nice four-pane view it took me so much trouble to create, and wonder whether that would have been saved with the file, had I realised that the next section is ...

Saving a Blender File which tells me, only a few minutes too late, that saving works pretty much like loading. I try to create the scene I had before, with rather limited success, but I do manage to create a scene and save it. Then I modify it and, in opening up the previous scene I learn that the cavalier attitude to current work is repeated when loading files. I curse Blender, and wonder when I will start to learn how to use it properly. I curse again because this is the end of the user interface tutorial.

Well, that was interesting. Fortunately there's a whole host of other material available, as indicated in the sidebar of the web tutorial I've just been working from, and some of it at least looks quite helpful. Having had to work with CAD programs in the past I'm well used to complex interfaces, and resigned to the fact that a certain complexity is inevitable. I have to question the wisdom of providing application-specific file load and save dialogs, however, when most GUI toolkits give the programmer ready access to the platform's standard dialogs.

It's definitely not helpful that there doesn't appear to be any standard terminology surrounding Blender (e.g. the tutorial describes the Info Window, but this is actually selected in the program's menus as User Preferences - how is this supposed to help?) and overall the beginner gets the impression of a professional-grade program with amateur documentation. The usual qualifications apply, of course. Since this is open source, nobody is forced to write manuals for the software, and anyone who's spent the time to find out how to use Blender would probably rather play with Blender than write documentation.

I am also surprised that Blender doesn't by default open up with a four-pane view of the scenes, since later experience shows that this makes manipulation of the objects in the scene much more precise and much less counter-intuitive. It's particularly galling that the four-pane (top/fron/side/camera) view of a scene isn't stored with the scene, so it had to be re-established each time.

Now of course it may be that these complaints are in fact invalid, and that it would be possible to make Blender behave in what I, at least, would find a more natural way. If this is the case, though, there is even more of a crying need for adequate documentation. So far I haven't even learned how to select objects in the scene, let alone use the many buttons in the buttons window. Overall I'd advise against picking up Blender if you're in a hurry to produce some impressive graphics for that important presentation. It will take time and patience to master this software, possibly more of each than I currently have.

Firedrop2 Revisited

My initial experiences with Firedrop2 were recorded as Open Source Frustrations. I'm happy to say that the initial problems in running the software were eventually traced to an error in the installation of the wax GUI package. At the time it looked like everything was copacetic, (making my remarks at the time about how trouble-free the installation was quite ironic) but alas needed sub-packages weren't installed by wax's distutils setup, so it was only after I did a manual install that Firedrop2 finally started to run.

As a result of all this Hans Nowak, to whom I reported the problem like a good open source citizen, has decided to stop using distutils. My own experiences with distutils have also been less than entirely happy, and I wonder whether in the long term we might not end up using something more like setuptools. While its developers acknowledge that it still isn't perfect I am sure they will be valuable enhancements to the distutils before their developers are finished.

The Firedrop2 tests are still completely broken, and they should be removed until they are worth running. Tests that fail when software is correctly installed and functioning are worse than useless, and give an incorrect impression about the software. In this case I suspect that they were vestigial when the software was handed from one developer to another, and that they haven't received any attention since.

It's also clear that the software hasn't yet been engineered for general use: simple problems like a missing configuration file result in a Python traceback window that would be excessively confusing to a non-Python user. After a few such errors, however, I realised that I needed to create a blog configuration before anything else would work. Once I'd done that, and edited the configuration to taste, things did start to work.

The function of some of the buttons remains far from clear, but the basic entry creation, editing and publication functions do seem to work.

President Spy

From the point of view of public morale, the US response to the 11 September airliner attacks was probably the worst choice of the many that President Bush could have made. Why not instead try to persuade the American people that the best response was to demonstrate that those atrocities would not change the American way of life?

It appears that it suited the foreign policy agenda, though, to undertake an unwinnable war on an abstract enemy without any definition of "success" that could be used as a decision point to cease hostilities. Broad powers were accepted in haste by a panicked Senate and Congress, and those powers can effectively be used against anyone who is perceived to be dangerous.

We now learn that under Bush's "control" the NSA has felt free to ride a coach and horses through what few protections were left to American citizens by the PATRIOT act. The Electronic Freedom Foundation asks the question: "what good is legislative reform if the Administration considers itself above the law?"

Of course the Guantanamo Bay imprisonments have already made it clear that Bush is prepared to redefine the parameters of acceptability to gain his own ends. Along with his repeatedly revealed contempt for domestic law, he feels that international law is insufficient to cope with terrorists. This reveals a disdain for the rule of law that casts doubt on Bush's suitability as the leader of the free world, should further doubt be necessary.

The tally of American dead "supporting operation Iraqi Freedom" continues to rise, and is now higher than the number killed by the 9/11 attacks. If America were invaded and occupied in the same way that Iraq has been the resistance to the invaders would be immense. Yet the power of the media and its collusion with the administration is such that the average American still believes that freedom is something that can be imposed by invasion and coercion. How much better it would be if America looked to its laurels and put its own house in order, thereby lucidly demonstrating the advantages of freedom and democracy in a practical way.

If democracy means that the security services can breach citizens' constitutional freedoms with impunity, how is it better than an arbitrary dictatorship?

Open Source Frustrations

This is a tale of how a simple attempt to use a piece of open source software descended into a frustrating and unpleasnt experience. Although specific sites are mentioned herein the lessons, if any, are intended to be generic. I have come across other sites pointing to the same absent download.

A recent search for Python blogging clients took me to the Voidspace web site and thence to the Voidspace Firedrop2 page. About two screens down the enthusiastic puffery I found what purported to be a link to Firedrop's home page, but was actually Hansje Novak's open source projects page.

This did contain some material on Firedrop including a number of links to pages showing a download link, which sadly contained not a trace of firedrop2. A search for "Firedrop2 download" revealed that the package was now maintained by the author of the first link, but the only download available was a link to the Subversion repository with the rather off-putting comment "There is a public repository of the bleeding edge version, which I try not to break. ... The version in SVN is slightly more up to date than the released version. New release to follow soon !" No links to the mythical "released version", though.

OK, I've now been poking around for this tool for five minutes and so I guess I should try Firedrop. So I fire up Tortoise (the easy way to interact with Subversion repositories) and find that there's a firedrop2 branch, but nothing I can spot in the trunk (which is where stable versions normally live).

So, I figure I've now invested ten minutes in locating this frigging software, why not take a look at it. I create an empty directory and download the firedrop2 branch contents into it. Oops, no README or anything else that purports to be documentation. So, back to the web page, where I find a section on "Installing" that begins with the useful warning "Currently Firedrop2 doesn't work out of the box. You must follow these instructions. I will address this issue soon." Better than no warning at all, I suppose, and it appears that all that's required is to drop the software into a directory on the Python path, but a little less than satisfactory.

Before I go any further I scan back up the web page I've been using and spot this tip: "Firedrop2 is written in Python and uses Wax for the GUI ...". So I follow the Wax link, end up on Sourceforge and download Wax. This, it appears, uses a standard Python distutils installation, which I can handle quite easily. So, fifteen minutes after I start, I think I'm in a position to install Firedrop2. At least I can import wax from Python.

So now I have to drop Firedrop2's installation into a suitable directory and ... erm ... yes, some documentation would be useful here. Just to make sure everything's kosher it would seem sensible to try and run the tests in Firedrop's test directory (at least there are tests). The first test fails due to the absence of some file. The second fails because I've mistakenly installed the software as Firedrop2 instead of firedrop2. Ho, hum. After renaming the directory the second test succeeds, but the third fails with ValueError: too many values to unpack. This looks like pretty fundamental stuff to go wrong, and I've now spent over twenty minutes poking around in this stuff for zero net gain.

A further ten minutes poking around and I've managed to create a data file that passes the tests. Still ten further minutes and I've got the whole test rig working. However, I have to ask myself: if the tests are so crappy, how good is the software itself, and how do I get it to work without documentation and a properly working test suite. So. let me remind myself how I got here: a recommendation from someone who said:

• Firedrop2 - The Python Blog Client.

  This is a nice blog tool that runs client side. This means you can maintain your blog offline and migrate your blog to a new server with the minimum of fuss. It has some useful plugins like an emailer and spell checker.

Really? Finally, just to confirm my suspicion that the whole deal is hopelessly broken I go into the main directory and run wxfiredrop.py, which results in the traceback

   C:\Python24\Lib\site-packages\firedrop2>wxfiredrop.py
  Traceback (most recent call last):
  File "C:\Python24\Lib\site-packages\firedrop2\wxfiredrop.py", line 32, in ?
    from firedrop2.mainframe import MainFrame
  File "C:\Python24\lib\site-packages\firedrop2\mainframe.py", line 22, in ?
    from wax.tools.errordialog import ErrorDialog
  ImportError: No module named tools.errordialog

I presume someone, somewhere, is using Firedrop2, but after this experience it isn't going to be me without a solid indication that its current problems have been solved. Unfortunately this kind of thing is all too common in the open source world. I can't estimate the number of times something like this has happened to me, and I can't believe it doesn't happen to other people as well. Sometimes I roll up my sleeves and start to fix the problems, but other times I just mentally shrug my shoulders and write the wasted time off to poor release engineering. In this one case I decided I'd invest a little more time in documenting my frustration, to see whether this is a common occurrence.

I can quite understand how this situation comes about, and as long as individuals are maintaining software on their own dime and in their own time we probably won't be able to do much to address the problems. If we could find some way of avoiding this situation, though, I'd be really happy. Even just regular link checking might help. Perhaps that can be one of my missions for the New Year. I'm going to try a few Python project installs and document my experiences. Hopefully this will do at least a little to improve overall quality control - or at least let some others avoid similar time-wasting experiences.

It goes without saying that if you have a working copy of Firedrop2 I'd be interested in hearing from you!

Sloppy Thinking

Seth Godin published a blog entry bemoaning his neighborhood's clean firetrucks. He attempts to make the point that organisations spend too much time on busy work, concluding with this thought:

What a great way to describe a stuck but busy organization. "They sure have clean firetrucks."

But he misses the point. Firemen clean the truck when there's nothing else to do. Quite apart from the beneficial effects on morale and esprit de corps that such cleanliness engenders, you really don't want a fire service that's busy all the time.

Such a fire service can't afford a single idle moment: queueing theory tells us that a 100% busy resource inevitably falls behind, as it can never recover any slack time that it experiences. Given the urgency with which a fire service is required when it is required, I'd much rather see clean fire trucks (which, by the way, in the UK are called appliances) than have to wait when I need a fire put out.

I've enough respect for Godin that I'm prepared to overlook his stretching an analogy to make a point, but I'm sure a lot of people have given him hell for "badmouthing the fire service".

Blogging by email

Apparently now I can illuminate the world with my iridescent thoughts from any email client in the connected universe. I somehow doubt this is going to help the sum total of human knowledge, but it will be deucedly convenient for making sure those odd little observations don't get
overlooked.

Turns out that the emails appear as drafts in the blog, giving me a chance to polish the thoughts before they appear before an amazed public. Which is how this second paragraph got added.

Having finally bothered to discover how to put sensible titles on blog posts, I now find that Thunderbird displays the updated posts with the date they were edited rather than the original publication date. Ooops...

Happy New Year!

I decided that it would be good to get in touch with the larger Python environment, away from my usual newsgroup sources. I'm happy to say that there seems to be a very active community, with many Pythonistas commenting on what's going on.

The best overall site for this kind of information appears to be Unofficial Planet Python, which has a great selection of feeds. Maybe one day they'll add one of mine.

Last year's runaway success in the web world appeared to be Ruby on Rails, and the Python community's initial reaction was something like "it's all over, Ruby has won the web battle". I always thought this was a bit feeble, so it's been good to see Ian Bicking coming back with a reasoned response triggered by Peter Hunt's How Python Wins on the Web and a reminder that Python has many web successes behind it as well as not a few still to come.

I'm not really one for language battles (despite having been labelled on the python.org Wiki as a "language bigot". Programming languages are tools, and one should choose an appropriate tool for each task. Python's chief virtue from my point of view is the way it seamlessly welds together great facilities for both procedural and object-oriented programming. If someone finds PHP or Ruby a better language for a particular purpose I don't really see that detracting from Python.

If the energy that goes into hand-wringing went instead into producing better and more usable Python applications, this alone would help to redress the balance (if indeed it needs redressing).

Versioned Reviews Implemented

The post-PyCON goal of incorporating reStructuredText processing into the Holden Web production system as a practical way of investigating how the initial architecture can be extended has now been initially achieved. A sample page is linked from this article's title.

The whole site is static HTML generated from a database. The links under "Python Reviews" in the left-hand navigation feature of the linked page are the first example of a "non-front-page" section - all previous sections in the web have navigation links to this section on both the homepage and internal pages. Eventually each section will have its own potentially unique navigation structure, though at present all internal pages use the same link structure.

The new section is generated from a new VerReview table, keyed on product name and version number. Several versions of the same project can be reviewed, and all reviews of a given project are interlinked. At the time of writing this navigation feature is demonstrated by the reviews of "wxPython"; the other reviews are single-review projects. Only one review of any project appears in the navigation bar (currently the most recent).

The main new feature used to produce the reviews is this ability to generate web page images in two steps -- each row in the VerReview table claims its own row in the Pages table, and a separate module handles the generation of the Page rows (each of which is a page in the finished site) from the VerReview rows. So some of the content is now generated from a separate table, potentially with its own relationships. This architecture could relatively easily define a plug-in API for new tables and publishing components, although it does not yet do so.

Once the section's content has updated the site's main Pages table the normal content interlinking processes handle the integration of this new content in just the same way as always. [The existing design will not gracefully handle sections with a large number of items, but the potential is clear and aspects of the data like recency could be used to reduce section size where necessary].

A further first is the use of reStructuredText, the format developed by the docutils project, to generate web content. Note that reStructuredText-for-the-web does not exploit many of the web's more versatile (and sometimes therefore more dangerous) possibilities, in the interests of maintaining a uniform concept of "document" across as many media as possible.

It would be imprudent to treat this as anything but a database publishing demonstration. The limited hypertext that the docutils software does handle proved very useful in generating the inter-version links between the different reviews of a project, but this capability should no tbe pushed too far. Further iterations can, if deemed desirable, generate reStructuredText in tables similar to VerReview from yet further content in as-yet-unknown tables.

Holden Web is grateful to David Goodger for guidance in navigating the distutils architecture to arrive at a first attempt production system that uses no external processes. Thanks also to Dave Beaudoin of Samara Design and Tim Parkin of Pollenation Internet for assistance with look-and-feel abd graphical design. The production system originates with Holden Web.

docutils Install Review

This is the entire review for a docutils install I recently performed:

Very simple installation under Cygwin: literally unpack the tar file, change to the source directory and run

python setup.py install

No errors, docutils is now a site-package!

Now that PyCon is over I've had a little time to start tweaking a feature I've wanted to add to the Holden Web site for quite a while. I end up looking at, and frequently installing, a lot of open source software. The installation routines are of variable quality and sometimes I have to stumble my way through, sometimes I never get to the end.

So the new "Python Reviews" section (which may end up being renamed) is there to report on installation activity. It is also a test bed for my first significant use of reStructuredText (of which more later), so I suppose it's appropriate that this blog's title links to the install review for the docutils software that converts reStructuredText into HTML.

As with other non-production features the section isn't listed on the home page, but it should appear in the left-hand navigation bar of all body pages. The article format is still experimental, and decisions about whether it's worth keeping the source in a database rather than in files remain to be addressed.

The news feature is currently driven by analysing the results of a search for "Python" in O'Reilly's meerkat feed, but it isn't yet updating automagically on the new host, and other sources should ideally be included.