Twisted Stakes a Claim

Disclosure: this post discusses remarks made by a Holden Web business partner.
In his Electric Duncan blog, Duncan McGreggor argues that Twisted now provides a viable LAMP alternative, and asks some very pertinent "so what?"-type questions:

As such, this is a direct competitor for LAMP. Here are some questions about that: What is the value of a full stack? Why is an alternative to LAMP good or needed? What is a good alternative? [...] We need to be asking ourselves what our applications are, what the network is, what services are, and what our dev teams and engineers need.

This led me to ask some questions of my own. Is this true? Is Twisted actually able to provide a full-service alternative web application solution? How adaptable is the architecture? Suppose I want to retain my Django templating instead of transitioning to Nevow: how easy would that be?

The case against LAMP starts with out-of-the-box scalability, as Sean Reifschneider (among other luminaries) has pointed out. I believe that Twisted has been proven at reasonably high levels. Whether it's processing Google-sized firehoses I don't know, but it's certainly supporting a significant part of the airline reservations structure.

As to "what is a good alternative", I suspect a lot of developers would go with "something that's orthogonal to my current code, so I don't have to rewrite anything". Perhaps that's asking a bit much, though. I know from various transitions I have made that some things just "fit my brain", and one of those was the Django templating scheme. Django users are pretty much database-agnostic, though a surprising number of them seem to use PostgreSQL. So perhaps Axiom support of Django templating?

The thing is, you can't just go around claiming to be an effective competitor to LAMP. You have to demonstrate the truth of that claim. I am sure the Twisted guys can do that if they are forced. They just don't realize the marketing necessity yet.

Nicholas Negroponte: An Ego Unbound?

I just finished reading Ivan Krstic's excellent Sic Transit Gloria Laptopi blog post. It appears that Negroponte would now like the OLPC XO to become yet another laptop we have to throw Windows off before we install some open source operating system supporting the GNU toolset.

I don't believe Nicholas Negroponte could manage to find his way out of a paper bag, let alone manage a hardware supplier trusted to build the infrastructure of developing countries. His Windows bait-and-switch makes him look like a shill for Microsoft. I wonder how much Microsoft stock he holds?

He just doesn't get that he has blown a huge chance to make a difference here. The thing that pisses me is the way he has played fast and loose with the work of literally hundreds of open source programmers who have attempted to support what they saw as a worthwhile educational project. But now Negroponte says that OLPC isn't an educational project after all. It's not about you. Nick, it's about the kids in the developing economies.

Thanks for the Money!

Well, apparently seven thousand other people also wanted to support brain tumor research yesterday at the eleventh annual "DC Race for Hope" in aid of the Brain Tumor Society and Accelerate Brain Cancer Cure. So many, in fact, that I didn't even pass the starting line until ten minutes after the first contestants. But it wasn't all about speed, which is just as well, since the lead runner past me on his return journey before I had even covered the first mile of the course.

The most important thing was the huge amount of money the event generated (a donation of a million dollars was announced just before the race commenced). Thanks once again to everyone who helped me to smash my sponsorship goal. With your help I raised $1,215, handsomely exceeding my target of $1,000, and my wildest expectations.

For those of you kind enough to inquire about my friend Sharon, her tumor turned out to be treatable with a combination of surgery and other therapies, and with luck she will be recovering from the surgery as I write this. I'm hoping that next year she can come to the race and join the other survivors. Thanks again for all your help!

The Media We Deserve?

Many thanks to Patrick Logan for bringing this New York Times article to my attention. It highlights the fact that the "military analysts" used (and paid!) by the major networks often appear to have business dealings and Pentagon relationships that cause blatant conflicts of interest with their independence as analysts.

My favorite quote shows that the networks feel they have something to hide, and in particular just exactly how "fair and balanced" Fox News wants to be:

CBS News declined to comment on what it knew about its military analysts’ business affiliations or what steps it took to guard against potential conflicts.

NBC News also declined to discuss its procedures for hiring and monitoring military analysts. The network issued a short statement: “We have clear policies in place to assure that the people who appear on our air have been appropriately vetted and that nothing in their profile would lead to even a perception of a conflict of interest.”

Jeffrey W. Schneider, a spokesman for ABC, said that while the network’s military consultants were not held to the same ethical rules as its full-time journalists, they were expected to keep the network informed about any outside business entanglements. “We make it clear to them we expect them to keep us closely apprised,” he said.

A spokeswoman for Fox News said executives “refused to participate” in this article.

CNN requires its military analysts to disclose in writing all outside sources of income. But like the other networks, it does not provide its military analysts with the kind of written, specific ethical guidelines it gives its full-time employees for avoiding real or apparent conflicts of interest.

Nothing like shutting up when you have something to hide. You can almost hear the ranks closing.

Microsoft Can't Support DRM

So it seems that "PlaysForSure" actually meant "Plays until the first time you change your computer or operating system after August 31, 2008". About ten days ago Microsoft's general manager for MSN Entertainment and Video Services sent out an email to customers advising them that license keys can no longer be retrieved and new computers cannot be activated after that date.

This is a very clear indication that digital rights management (DRM) is a loss for the consumer. If I wanted to keep songs from my eight-track stereo around then there was nothing to stop me from transferring them to audio cassette or CD (despite the RIAA's rather specious arguments that this was a breach of copyright). If I'd been silly enough to trust Microsoft's assertion that DRM offered me protections then I'd have to get ready to shell out for new copies after any change to my playing environment.

The Electronic Freedom Foundation has written to Microsoft stating in no uncertain terms that (by suggesting that users copy the music to a CD and transfer it to new computers that way) the company "is asking its customers to invest more time, labor and money in order to continue to enjoy the music for which they have already paid".

If Microsoft, with all their resources, can't keep a DRM scheme working then there is little chance for anyone else. It also doesn't encourage customers to trust the company if it is prepared to abandon them in this way, though they might have anticipated this desertion when Microsoft closed the MSN store when it started selling the Zune.

I suspect this is the beginning of the end of DRM. Let it rest in peace.

Twisty Little Passages ...

I have been rather disappointed that the "Teach Me Twisted" session I ran at PyCon didn't get more heavily blogged. Not only did I have a lot of fun, but I also learned something. The Twisted guys learned something too, I'm happy to say.

No announcements yet, but with luck it will eventually lead to bigger things. That session may have been a butterfly flapping its wings.

Django Nudges Closer to 1.0

Malcom Tredinnick just announced that he has merged the long-awaited queryset-refactor branch into the Django trunk. This puts one of the major blocks in place for the 1.0 release.

James Bennett recently posted his list of requirements for a 1.0 release (with a warning that suggestions from twits would not be tolerated: do not annoy this man if you don't know what you are talking about), and this announcement reduces the length of the list to eleven items.

So it's got significantly closer, as queryset-refactor was a biggie. Now I have to go test my code against the new branch ...

Impressive Silly Trick

Nice to see that the Python community still enjoys a challenge!

I Want Your Money!

Not for myself, I hasten to add, but to support the Brain Tumor Society. The Race for Hope 5km run is an annual event organized by my wife's employer, and we had agreed to participate before we discovered recently that one of our friends has a brain tumor. It's too early to know whether it's operable at this stage, so we are hoping for the best.

[UPDATE 4/22: I have discovered that my wife has already raised almost $2,500 so I have upped my target somewhat to see if we can't show the world that geeks aren't cheap!]

In the meantime, I have set what I feel is a fairly modest goal for additional sponsorship. I am hoping readers will show how generous the open source world can be. If you've ever enjoyed a PyCon then please drop by my personal race page to make a donation, and point your friends to it as well. I'll be quite happy to up the target if the existing one proves too easy, and any amount will be welcome.

Thanks to anyone generous enough to support this effort. If the team I'm on is in the top 5 fund raisers next week we'll have an extra $1,000 added to our sponsorship, so dig deep and treat the target as something to beat, not exceed.

Impeach?

I mostly keep politics (except perhaps the politic of open source) out of this blog, so if you aren't interested you won't want to follow this link. I'm not sure why I got so fired up about this today, specifically, except that the day's news has been full of politics. But when wasn't it, lately?

Blogger Annoyances #319

Is this just me, or is there some trick I just haven't learned? I was reminded, yet again, when completing the previous post in this blog, that Blogger tagging is way less user-friendly and convenient than del.icio.us and other similar sites.

I wanted to tag the item with "mono", but alas I was foolish enough to tag a previous post with "monopoly". So when I enter "mono" the widget is already suggesting the auto-completion (see image), and when I type a comma, it completes the entry as "monopoly", the existing tag. I try entering an escape character—no good, the comma still completes it as the existing tag, as does a tab.



Eventually I let it complete as "monopoly" and then go back and delete the last four characters from the tag. I really dislike applications that insist they know better than me. Since tab expands the tag to complete as an existing tag, why the heck does a comma have to do the same thing? The real annoyance is: I have this ugly feeling that somewhere out there there's a programmer feeling really smart for having committed this atrocity. Someone, please tell me there's a sensible way to do this before I go mad.

Learning IronPython

Some good advice from Curt Hagenlocher on the IronPython mailing list in response to this question:

I program in Python, but my programs are typically run from the command line or IDLE. I have never built a GUI, and would like to try my hand at creating GUIs that I can run my programs from. Is there a book or tutorial that covers IronPython, IronPython Studio and the dot Net platform that is aimed squarely at the new user?

Any book or tutorial about the .NET platform in general will be relevant to IronPython's use of .NET. In fact, taking existing C# or VB examples and translating them into Python may be a good way to leverage your existing Python skills while teaching yourself the .NET class libraries.

As far as IronPython-specific resources go, you can't beat the IronPython Cookbook at http://www.ironpython.info/index.php/Main_Page. If you want to see some sample Python code that uses the Windows Forms or Windows Presentation Foundation libraries to produce a GUI, just go to the Contents at http://www.ironpython.info/index.php/Contents and scroll down a bit.

It's fashionable to pooh-pooh .NET, but remember that this stuff runs on Mono too, so it could be an important way to build cross-platform compatibility in Python applications. Not everything that came out of Redmond is worth ignoring (but let's not talk about OOXML), so there's possibly something to be gained from a knowledge of that framework. Don't throw the baby out with the bathwater.

Galcon Cheats!

I'm not going to ask for my money back, or anything, but I really do think it's a bit naughty of Galcon to mobilize its forces before I've even pressed the start button.

If you haven't tried this game by Phil Hassey (met him at PyCon, terrific bloke) it's quite an addictive space war adventure where the objective is to overrun neutral and opposing forces to gain control of planetary territory.

You can also run it as a network server if you want to host multi-player games! Well worth a try. But it still cheats, dammit.

For the Want of a Nail

We've all done it. One step forward, two steps back. I just thought I'd try to relieve my current frustrations by writing this particular sequence up. It's amazing the diversions and byroads you can end up taking to achieve an apparently simple goal (as I learned yet again last week in trying to accommodate Outlook on my SMTP server). This one started at PyCon, now four weeks in the past. So, bear in mind the goal: submit a Django documentation change.

At the Django sprint (of which, due to a family illness, I was only able to attend a truncated first day) I was struggling with an unstable laptop, so I managed to assign myself a documentation ticket, and that turned out to be pretty much it what with PSF responsibilities and needing to go home early. I have been working on the ticket since: it isn't a difficult issue, but I needed to learn a little more about how the Django database layers worked.

Having finally decided on the required change, during entry in the Django tracker I decided it would be much more developer-friendly if I were to provide the change in the form of a patch. This isn't really necessary but I figured it couldn't hurt, and that I should really be able to rebuild the documentation locally to be sure that any future changes were stylistically correct and didn't mangle what is, after all, quite good documentation.

This is where things started to get a little rough. Descending into the documentation directory I discover that it's driven by a Makefile. "No problem", I think, "I have Cygwin, I can build under that". So I type "make html" and discover I don't have make installed. This is quite normal, I like to keep Cygwin fairly lean, and I haven't previously done any Cygwin building on the new laptop.

So, I run the usually very cooperative Cygwin installation utility, and discover that my usual mirror site is providing very slow response. So slow, in fact [minor pause here to repair cable retraction mechanism on traveling mouse: this really is not my day] that several attempts to update my installation have to be abandoned, though it notes that several of my other installed applications should also be upgraded and has even managed to pull down some of the installers.

Eventually the fifth or sixth attempt gives me the dread message from Vista (how I hate this operating system, yet despite my best efforts there are tasks I currently still need it for) that the Cygwin installer "is not working correctly".

After a reboot and several further repeats, I eventually track this down to a corrupted /etc/setup/openshh.lst.gz file, and delete it. I establish contact with a different mirror and lo! The Cygwin installer now runs to completion, and I can run make. So I confidently re-enter "make html" and see the message

make: sphinx-build: Command not found

Fair enough, it shouldn't (I foolishly think, despite all the evidence that should by now have convinced me to set the computer aside for the day) take me long to get that up. As a first stab, before doing anything else, I type "easy_install sphinx-build". Hello, another even more interesting error message:

bash: /c/Python25/Scripts/easy_install: Permission denied

What? ls -l gives me the following rather unsavory news:

----------+ 1 sholden None 6656 Apr 2 10:33 /c/Python25/Scripts/easy_install

Damn, that can't be right? How did that happen? Well by this time I am past caring how it happened. I own the file, so I'll just reassert the correct permission, right? Wrong:

$ chmod 755 /c/Python25/Scripts/easy_install chmod: changing permissions of `/c/Python25/Scripts/easy_install': Permission denied

Now this is getting really annoying. Note how my problem has subtly changed from "how do I submit a Django documentation change" to "how do I get Cygwin to allow me to change the permissions on this file", and that I am now several steps further away from my original goal than I appeared to be at that happy moment when I set out to submit the change. Note also the assumption that running easy_install is actually necessary to solve my problem. If this had turned out to be wrong I would probably have shot myself, thereby relieving you of the task of reading this post.

Fortunately I quickly realize that this is the easy_setup from my Windows Python that I am trying to run, and that the solution is to run ez_setup.py with my Cygwin Python. [There are some who doubt the wisdom of doing this to a Python installation, but I am blasé beyond belief about such things]. So I run ez_setup.py, which installs setuptools. There is the minor annoyance that Cygwin refuses to recognize the correct version of easy_install and continues to run the Windows copy, but I solve that by opening a new Cygwin terminal window. Another final false start: the package required isn't called sphinx-build, so easy_setup can't load it. So I take another reasonable guess.

Now, finally, I run easy_install sphinx followed by make html, and in less time than it take to write a blog entry I have a local copy of the documentation that I can patch to my heart's content. Wonderful things, computers. When they do what you want.

Tweaking the Bytecode

Ned Batchelder has come up with an interesting hack for increasing the granularity of his code coverage testing. It involves modifying the code object associated with the code being modified: specifically, the co_lnotab member contains a starting line number followed by a sequence of (code offset, line number delta) pairs.

Ned's technique involves rewriting the member so that each bytecode appears to be on a single line on its own. While this allows him to ensure that all bytecodes are exercised, he hasn't yet ironed out the quirks in how to use that to present coverage information (which is hardly surprising, since most Python programmers could care less about the structure of their bytecode).

It seems to me, however, that by mapping the modified co_lnotab back to the original you could at least flag lines whose code isn't 100% covered by your tests, and this is surely valuable information. Ned's already thought of this, and includes the throwaway remark "The original line number information is lost. It would have to be stored off to the side to make this useful" towards the end of his blog entry.

So I believe we can expect coverage.py to start producing even more useful information soon.

Meme du Jour

Interesting. I was quite surprised by this myself.

# history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}' |sort -rn|head
135 svn
134 ls
87 vi
76 postmap
57 cd
45 more
36 ps
35 /etc/init.d/httpd
33 /usr/sbin/postfix
32 grep

This is on my (virtual private) web server, where I've been editing fairly heavily recently.

Outlook is Driving Me Nuts!

[UPDATE: two changes were required. First I had to set broken_sasl_auth_clients = yes in my postfix main.cf, then I had to add login as an available authentication mechanism in dovecot, which postfix uses as an authentication server. Thanks for all pointers and suggestions, some of which came by e-mail. Not quite as stupid and fundamental as I feared. Steve].

OK, I am in serious need of assistance here. I suspect that there must be something really stupid and fundamental that I am overlooking. The problem is that I cannot get Outlook 2003 to authenticate to its SMTP server.

I have tried deleting the account and recreating it (shutting Outlook down in between). I have verified that the same settings in a Thunderbird client allow me to send mail without difficulty. I have even installed Outlook on an entirely separate computer and got the same dismal results.

Why Outlook? Well, although I am a Thunderbird user myself my wife uses Outlook. So my most important support client has a problem -- only with sending mail, though, receiving works fine.

Just so you can visually verify that I haven't done anything stupid, here are screen shots of the relevant dialogs. I have even set up a local printing SMTP proxy to verify what's happening! With Thunderbird it clearly shows an AUTH command being send to the server, but with good 'ole Outlook I get what you see below (slightly redacted): straight from EHLO to MAIL with no AUTH at all.

Connected
==> (0) '220 holdenwe.user.openhosting.com ESMTP Postfix\r'
<== (0) 'EHLO DoroPC\r' ==> (0) '250-holdenwe.user.openhosting.com\r'
==> (0) '250-PIPELINING\r'
==> (0) '250-SIZE 10240000\r'
==> (0) '250-VRFY\r'
==> (0) '250-ETRN\r'
==> (0) '250-AUTH PLAIN\r'
==> (0) '250-ENHANCEDSTATUSCODES\r'
==> (0) '250-8BITMIME\r'
==> (0) '250 DSN\r'
<== (0) 'MAIL FROM: <xxxx@holdenweb.com>\r'
==> (0) '250 2.1.0 Ok\r'
<== (0) 'RCPT TO: <xxxx@gmail.com>\r'
==> (0) '554 5.7.1 <xxxx@gmail.com>: Relay access denied\r'
<== (0) 'QUIT\r' 

So, if you can tell me what's wrong you may rescue my sanity! Help me, please, this is driving me nuts.

Google Busts the Web Wide Open - In Python!

Well, if ever there was a compelling reason for using Python it's probably Google App Engine. Here's the canonical "hello world" Google App:

print 'Content-Type: text/plain'
print ''
print 'Hello, world!'

Doesn't get a lot simpler than that, does it?

There's a lot behind this announcement, including a detailed description of the data store (which interestingly seems to bear the same kind of relationship to a classical relational database that Python does to statically-typed languages), access to some of the attributes of authenticated Google account users, the ability to fetch remote URLs and a startlingly straightforward way to send email.

If I were going to write another book about Python web programming I wouldn't be able to ignore this development. What a tempting idea ...

Twisted Bite the Web Bullet

The Twisted community has decided it's time to clean up its web act. The problem for a while has been that there were three "competing" packages that might have formed the basis of any web application: twisted.web, nevow and twisted.web2.

Now they have decided that twisted.web is the right namespace, and the immediate plan is to backport the stable functionality from web2 and the nevow application server into twisted.web, giving a more advanced set of web functionality in a way that offers best compatibility to existing users. Nevow will intially remain, as a pure templating solution, but there's a possibility that functionality might also be repositioned eventually.

This has the smell of a compromise, but as compromises go (and as you'd expect in the Twisted world) it seems to be a pragmatic engineering solution to the current rather confusing situation.

One-Day Class Topics

Since the first Introduction to Python class seemed to go quite well I am thinking about expanding the offerings by slotting in one-day introductory classes on Python-related topics.

From PyCon I know there is a lot of interest in both Twisted and Django. But I wonder how to contact the people who want to learn these technologies. Are they already members of the Python community (to whom this blog is primarily addressed most of the time), or do I need to generate some new marketing channel to reach them?

What other Python technologies do people need a fast introduction to?