Wrong About MySpace ... and What Else?

Now the the Murdoch empire's six-year trip to the shitter has seen MySpace thrown out of the group for chump change (and there's a stockholder lawsuit waiting to be tried, since arguably it has sold a 95% stake in a still very large chunk of Internet traffic) I thought we might look back and see what it was that Murdoch thought made MySpace a good buy in the first place.

At the time News Corporation bought Intermix, a minor media company whose principal property was MySpace, a News Corp. official was reported by the New York Times as saying MySpace generated "healthy" annual profits of "a few million dollars." Sadly the only way I can quote the News Corporation annual report from 2006 is to insert a graphic, since this is a company that still maintains the traditional stance to "copyright" and won't let you copy text lest you abuse some "corporate right" or "digital right" that has to be "managed". Whatever. The report said:

If ever a statement were designed to bring hubris down on the head of an errant corporation, "our company has begun to ... surpass the Internet elite" would be it. It's the corporate equivalent of John Lennon saying the Beatles are bigger than Jesus. Clearly the gods decided there and then that, no matter how wisely it was managed, no matter how exceptional the staff assigned, no matter how crystal clear Fox Interactive Media's Internet vision was, this company was doomed to failure.

The message of doom had not penetrated in 2007, though, when the most significant things said about MySpace in the report were:

Profits, improved infrastructure, broadened offerings. These are all things a growing business likes to be able to report. With increased revenues and profits on the horizon it would be, in Private Eye magazine's timeless phrase, "treble brandies all round" in the boardroom.

In 2008 News Corporation, tiresomely, and in a magnificent display of lack of understanding the Internet medium, turned to Flash for the annual report. The Flash tool was so opaque the only option was to download the PDF again. Only now I had to search it for myself. Arguably, News Corporation ought to know what "publishing" is. This is not it. Mention of MySpace is sprinkled throughout the report, with nary a sign of the coming storm on the horizon. A photo site had been acquired to compliment MySpace, and MySpace had been in partnership with this, that or the other enterprise to produce some Internet media coup or other. All very boring and normal, really.

In 2009 the trail again leaves PDF as the favored viewing option (no way am I going to listen to and watch the ridiculous glitz the Flash "HTML" version imposes on the numbed viewer). The first mention of anything that might be described as disquiet about strategy now appears:

No clue is given as to the identity of these "people" from "within the industry", but since these purport to be the words of Chairman Murdoch, we may safely assume he counts himself among their number. Charging bravely onwards we see MySpace now relegated in reporting to an "Other" section created almost specially for Fox Interactive Media (the only other company reported under that heading is an Eastern European hoarding space sales organization). And what do we read?

MySpace is just one of a number of domains in a tiny little part of a huge empire. These people have bigger fish to fry, right? Since I haven't yet seen a 2010 annual report I will decline to speculate about the level of enthusiasm, but I think even Bertie Wooster would, in MySpace's place, be feeling a certain coldness and considering a return to London's safer climes.

And now, today, comes the announcement that what remains of MySpace is to be unceremoniously sold like, well, like a business that's had all the value pissed out of it by people who don't understand "new media" and think that old media muscle is all that's really needed to weather this technology storm.

The interesting question for me is, if Murdoch could be so wrong about MySpace a scant six years ago, what other losing technology bets has he made? MySpace demonstrates very clearly that value can disappear from a networked enterprise in a surprisingly short time. News Corporation could be in serious trouble if it fails to adapt to the Internet age as Microsoft so nearly did.

Lightnings, 17:30 Saturday March 12

Eric Holscher says use ReadTheDocs, they are hosting scads of stuff for some great people. DjangDash gave birth to this fully open source project. Learned chef ass-basckwards. Heading for half a million hits a month, custom themes, tags & branched from Git with PDF generation thanks to Sphinx. Subdomains and project shortcuts will bebrilliant. They support CNAMEs. Chef is awesome. Multiple servers, because it's architected and shit. Think hard about your URLs before they become dumb-assed lame legacy URLs. Promote your project and find a designer. Open source is cool because they've had patches. You don't get arbotrary code execution rights until you are trusted. You now have one less reason for not writing docs.

Nate Aune says managing servers sucks, nobody likes setting up nginx or Apache, establishing backups, monitoring. Why don't we just have a "DeployNow" button, and that'd what DjangoZoom is supposed to to this. Copy your source code URL from github, paste it into DjangoZoom, and add a new project. Zoom inspects the .py file to make intelligent guesses about what's in there, setting it up to be served from nginx. In less than a minute, here's a django CMS instance running against a publicly-available Internet address. Now there is no reason to admin your servers for deployment, just use DjangoZoom.

Update on the Red Cross donation challenge. So far 45 definites, somewhere between $450 and $800 raised so far. If you haven't donated, please do so. A nuclear reactor in Japan has breached containment. Thngs are pretty dire. Please test REDCROSS to 90999 and then tweet with hashtag #pythoncares.

Oops, this speaker suddenly realizes this isn't the latest version of his stuff. Philikon talks about English, which started out giving us German words, then French, anwadays Finnish, Afrikaans, anything. We can manage to communicate nevertheless, and despite English being a mess the whole world manages to communicate. JavaScript's kinf ont he same. JavAScript, y u not pythonic? It's a mess. But we have all these really great apps, and slowly even replace desktop apps. JavaScript will be moar pythonic, ES5 deprecates all kinds of silly stuff. ES Harmony (much of it already implemented) will do more. Structural (unpacking) assignments, generators (stlen wholesale from Python), iterators, array and generator comprehensions, fewer parentheses, records, all kinds of great stuff. Use these features they are super exciting.

Enrique Bastos is talking about Community Bootstrap. He is deeply involved in building the Python community in Brazil, which is growing really fast, but fortunately community can scale. Why do this? If you don't have a good reason you could falll into traps like chasing an amount opf money rather than what you could experience with it. You need to live an experience to understand it, but don't rush. At first simply get people together to allow interpersonal bonds to form naturally. Don't try and build a group, hold events. People like to join in, they have a lot of events. Your community is the people who turn up at your events. Actively reduce entrance barriers by focusing on newcomers as the most important part of the community. Students are often valuable contributors. Empower people. Rules have side-effects, try to avoid them but discover your principles. Shirts are available with the Brazilian group's ten principles.

Jacob reminds us that daylight savings time begins tomorrow, and that iPhone alarm clocks will not work tomorrow.

Barry Warsaw confirms we all internationalize our applications, right? (of course we do). The flufl.i18n module helps you much more than gettext. Doing stuff with this is pretty difficult. Mailman 3 has some internationalization stuff, which has now been refactored out of the project for standalone us. You can find catalogs using the built-in strategies, or write your own. You can build stacks of languages and query the state of the stack.  Adaptable, it's on PyPI and Launchpad, available now. I think that FLUFL thing has gone to Barry's head.

Aron Grifiths (sp?) wants to tell us he's been through a series of Window managers pretty much like us all. They are either not scriptable or you have to write complex things like C plug-ins. Tomorrow's window manager won't use today's plug-ins. PyWI allows you to program your WM in Python, sand focus in any direction, do all kinds if wonderful stuff. Nice screencast to finish.

Josiah Carlson is talking about Over-Engineering for Dummies. This means either over-building a system or service. RUle 1: write everything yourself. Under no circumstances use convenient modules that do exactly what you want. 2. Start with something completely unsuited to the task; like the professor Josiah knew who build Prolog on top of Scheme, or the guy who added distributed counters to Cassandra. 3. Keep adding features and functionality and so on until no-one can think of anything else to add. 

Grace Law wants to tell you how to get that job you want. Lots of really qualified people don't get the jobs they want, Grace is here to help. Insight into some and interview questions and answers. Employers are looking for hidden remarks. Unintentionally disses QA people to ribald amusement. Be able to talk about learning new things - the more complex a technology you are learning the more impressive that is. Build open source projects and offer to discuss the code. You have initiative, take actions, and undertake complex projects for fun. Poor speed of answers to questions can give a bad impression. Tips on job search, see Grace.

Thanks to all speakers and, of course, Jacob Kaplan Moss.

PyCon Lightnings, 17:30 March 11, 2011

Session starts almost 20 minutes late - looked to me like logistical problems with the stage change, but I am so far out of the loop and the people in charge are so competent that the best thing I can do is start the post and wait patiently like everyone else.

I am sitting with Titus Brown and Peter Conerly, who are discussing matters of great technical moment. It's Peter's first PyCon, and the opening announcement is the every lightning slot in the conference is allcoated.

The opening talk is a request for PyCOn to raise $10,000. If 1 in 5 gave $10 each ... Text REDCROSS to 90999, then tweet that you gave with hashtag #pythoncares. End of talks on Sunday we will get a count.

Next speaker's display isn't working. Awkward pause. Silent audience. I am hoping things soon run more smoothly for Jacob (Kaplan Moss, the evergreen sprint chair who is here for the first time as a gentleman farmer).

Jeff Daily talks about Global Arrays Parallel Programming Toolkit, that added Python bindings to it toolkit in the last year. It gives access to a high performance computing array, giving access to distributed numpy-style ndarrays. Compared with other models, Global Arrays is less general than some techniques, but it scales well to at least 200,000 cores. This finally achieves highly parallel programming with true Python programming ease. Much simpler than message-passing interfaces, the technique is extensible and scalable, but requires that algorithms acknowledge the need for data consistency. If you're at PyCon, see Jeff at Poster #13.

Matt Harrison talks about his geek inferiority complex. He bought a computer at Fry's because computer class at school was too full. For years now people have been declaring "the year of Unix". Linux people run stuff like Gnome and KDE and a tiny set of other window managers. This guy inhabits a pretty specialized point in the Linux desktop fractal space, a rarified atmosphere in which indexing by 1 is awesome. And still we get more specialized. Matt's wrists forced him into this (in)sanity by screaming when he tried to pick the mouse up. Impressive demonstration of driving a window manager from the keyboard ensues, and lo! relevance: the QTile window manager is written in Python and is scriptable: now windows are flashing on and off in four different virtual screen. 2 further users will amount to exponential growth in the user population. Please join him.

Pete Fein wants us to hack for freedom. Video of protesters, under attack by flame throwers, getting pictures out through cellphone. Pete explains how he has helped freedom by getting stuff out on the street in 105 cities in a week. He worked with telecomix.org to help keep sites up and services running. They are leaderless and fanatically devoted to freedom. When the net was up they provided encryption and other services, mirrored censored sites and manually relayed tweets for egyptian users with no network access later. With Anonymous they set up a network to send information by fax, and by injecting information into logs of a few servers they found still up. It can be exhausting working for freedom, but we have the power to help people achieve it.

Next speaker (fumanchu at aminus.org(?)) not using his own laptop, another pause but we are now seeing images. No, gone again. Back again. Go. Logging. Really? Yes, because all the modules in your application can use the logging model independently, but this isn't applicable for things like performance statistics so that independent modules can log performance data and report on it. Solution: log into Python dictionaries, with package names as the top-level namespace. You can define record structures, and (for reporting only) can insert functions into the dict to be used when reports are required (writing the log should be high speed, so no function calls). This allows people to build their own reporting tools, or to user ready-made ones such as the one built into CherryPy. Open Space tomorrow.

More AV glitches as the next speaker arrives, then departs so I have no insight into the problem. Ah, they need a VGA dongle for a mini-display port. Strange, PyCon normally has this stuff buttoned down. I hear someone say "waht are the odds" and don't know whether this is a good sign or a bad one. BVideo at last appears, and Nathan Nichols wants to tell us some lessons he learned writing automated content generation systems using DSLs. Clients send data, they create quality content for it. Journalists are needed but hey they don't know Python. So instead use DSLs. Rules: 1. Keep Parsing Simple, Stupid (PyParsng was overqualified for the job, regexes and YAML serve better); 2. Keep things on rails (no, not Rails - Nathan knows his audience) and remember that while we think Python is awesome journalists don't and would be overwhelmed by its complexity; 3. Tools are super-important, and DSLs allow you to write very specific tools. Good talk.

Another pause, this time to load the next presenter's files ont he same laptop. It's Giles Thomas wants to tell us why Resolver Systems switched from IronPython back to CPython. Resolver are known for their eponymous spreadsheet, which is quite clearly a windows application. They now have a new project, Dirigible, was written for the web to provide similar services. The initial strategy of marketing to financial companies, and Windows was mandated bh the conservative nature of the client base. So Resolver 1 was a full .NET application. For Dirigible, CPython was the natural choice since none of the .NET stuff was required.

The SingPath competition is about to start somewhere close by. More lightnings in the morning. Awesome day 1!

Supporting All Versions of Python All The Time With Tox

This talk actually used the differences between Python 2 and Python 3 as the major version differences, but these are just an extreme example of inter-version differences.

Tox uses a config file that allows you to specify (among other things) which Python versions should be tested, dependencies between tests and so on. But how do we cope with syntax differences, where(for example) Python 2 code will raise syntax errors. The 2to3 utility is a fast and very comprehensive tool, and you can test for Python 3 and run 2to3 automatically in your setup.py.

It is possible to generate a single source file that compiles under both Python 2 and Python 3, but this is not easy. Often it's easier to convert - particularly for tests, where automatic conversion may not work. Tests may be distributes or non-distributed. In both cases converting the tests to Python 3 is likely to be required. Automated conversion is always subject to failure.

It's possible to use different environments (virtualenv) for the different versions, and even with Jython. Although tox can be used from the command line, it was designed for use in continuous integration. It does need to be bootstrapped a little bit: it should be installed locally (probably under a virtualenv!) You can also define a custion PyPI index.

Summary: the most important thing is to be able to support Python3. The single source approach requires most discipline. Mozilla is hiring. Had to leave the Q&A to start a meeting. Sorry.

JavaScript for Pythonistas

My first extreme talk, and Ian Bicking is always worth a listen. Because it's extreme, no intro to JavaScript. "It's a small, simple language with some weird things that are totally understandable." Ian wants to focus on what the language does rather than why it does it.

JavaScript is easily accessible to Python programmers, but the data structures can be tricky. Arrays are like Python lists, objects are like Python dicts. Objects can be fairly lightweight, and can be passed around. Keys are always strings, -- subscripted notation and attribute notation are equivalent. Exceptrions work very similarly to Python, but JavaScript programmers don't use them anywhere near as extensively.

Function declarations are fairly intuitive, but unlike in Python an undeclared variable is assumed to be located in some "containing" scope, and the var keyword causes them to be bound locally. Function arguments are always optional, and parameters for which there is no corresponding actual argument will be undefined. Undefined is "falsish", and we have to use typeof(name) == "undefined" to test for this.

Object classes are created by prototype, which is more like Self, or perhaps Ruby, than Python. The new operator is the only way to effectively create new instances. A function call implicitly binds to the variable this, but without new the binding will be to some global object (in a browser typically the document or the window). The inheritance hierarchy is handled similarly to Python except that instance creation calls the prototype function using new. It's not, therefore, as simple to use factory functions and classes interchangeably - though it can be done if you work.

The this object is almost comparable to Python's self, but the way JavaScript figures out what value it takes is, Ian submits, insane. With no equivalent to Python's bound methods, and method calls are determined syntactically. Consequently assigning an object's method to a local variable will cause this to be computed completely differently depending on whether the method is called directly or through the local variable. This is also problematic when closures are used to create private data stores. You can sometimes overcome it my saving this for later use (Ian chose the name self for the local variable he used to do this).

Functions have .call() and .apply() methods, whihc you can use to call them in specific ways.

Monkey patching should be avoided.

Types are one of the aspects of JavaScript makes least sense to people. It has numbers, strings and Booleans. The language also has the same kind of distinction between primitive and boxed values that Java does. There should really be no need to use the classes that wrap the primitive types. This gives rise to strange non-intuitive rules that are applied during comparisons. The "===" operator is similar to Python's is.

Arrays are disappointing to Python users, being a pale shadow of the Python list. There is not iteration protocol, and no test for containment of a value within a list.

Much of JavaScript's applications work on the browser Document Object Model (DOM). C++ is responsible for the DOM's complexity. The language is improving, but the new features are only slowly working their way into implementations. Many newer features have been inspired by, among other languages, Python.

If you like Python, you might want to try using CoffeeScript. It is more whitespace sensitive than Python, and handles arrays better than JavaScript.

Alas, time for only a few questions:

Q: What do you recommend as the closest equivalent in JavaScript to Python's interactive console. A: Firebug and the like.

Q: What does inheritance look like with prototypes?
A: If a prototype has a prototype that is essentially subclassing.

Promoting and Supporting Open Source

I was a little surprised today to discover that my Twitter buds mostly hadn't heard I had moved to Portland, and when I thought about it I realized that blog readers had no way to know either, since although I have been talking about the move a lot I have been too busy to write about it. This condition still holds, really, so this is a sketch for now.

In order to engage the Portland business, government and open source communities I organized a reception a couple of weeks ago, which also represented the announcement of a new business, The Open Bastion. As you can see from the web site we are still in well, not exactly stealth mode, but it's early days yet. I am happy to say that the reception appears to have started many conversations abut open source, which I will be following up on as they develop.

Why Portland? Well, I have been feeling the pull of the West coast for some time, and have visited Portland quite a but recently, including running DjangoCon US there last September. So the plan for the new business is to organize and run technical events–conferences, presentations, training sessions. There is also a very active Python user group here, which I attended shortly after my arrival in the city as one of about forty people there that evening. Portland likes to think of itself as the open source capital, but I want it to put its money where its mouth is.

Portland, being a conference city, is marvelously well endowed with venues. I am happy to say that it appears to be a city that values its heritage and it still has many fine original buildings, which makes the place a pleasure to walk around. With excellent public transport (including a fare-free zone in the center to discourage the use of private cars) it's a wonderful place to live and to visit, and I am looking forward to spending time here.

I will be keeping in touch with the East coast, too, by spending roughly a  quarter of my time in New York, where I expect technical events will also be required. We are, of course, equally happy to help you with events in the location of your choice. So if you are thinking about running a technical sales presentation, or a conference, or any other similar sort of event and you would like someone to take the organization off your hands, drop me a line. I am "sholden at theopenbastion dot com".

Community or Fanbois?

Regular readers will know that I place a lot of stock in community. I started PyCon, the US Python community conference, which has taken off worldwide. In earlier incarnations I was chairman of the Sun UK User Group and Treasurer of DECUS UK. As chairman of the Python Software Foundation I spend a lot of time thinking about how to engender community spirit and encourage people to contribute to the Python language in any way they can. This is an ongoing battle, but I do think that the Board is showing signs of understanding how to involve people, and also that the Foundation relies on the involvement of third parties in order to achieve most of what supports its mission.

In my own life, I have recently decided to become (even) more involved in open source, hopefully to the extent that I can make a better living at it - the PSF chairmanship is an honorary position and takes time away from business matters. So I have moved to Portland (whose natives frequently refer to it by its PDX airport code), and recently hosted a reception to get PDX open source people together with elected representatives and business people with the intention of starting new conversations about how each can benefit the other. I am hoping that the move will allow me to work in a more sympathetic environment, and one in which the potential of open source is more clearly perceived.

So anyone who cares to look should be able to discern that I am at least fairly serious about Portland and its open source community, and working towards improving things for the open source community (on the theory that a rising tide lifts all boats). I have found in the past that it's generally possible to share plans with open source community members and have them respect the sensitive nature of the information you have shared with them. This is one of the things I like about working in the open source community: generally speaking (and with the occasional unavoidable exception) people are willing to respect your concerns, and are generally much more concerned about producing good software than scooping each other on news of features and the like.

The people I meet in the open source world are generally responsive to new ideas and quite willing to discuss them. Generally speaking people are both willing and able to discuss the work they are doing - after all, it i going to be published, so there is little point in secrecy. Contrast this, however, with the brouhaha that arose today about the latest version of Apple's OS X operating system. It seems that Apple shared a beta version with some developers, who have naughtily (and anonymously) disclosed what purports to be real information about what Apple rather grandiosely term "the world's most advanced operating system". To which  my reply is a snort of derision, since I think that Mac OS X is actually in some ways inferior to Windows.

It turns out that they really don't like it when engineers who are given access to a pre-release (in this case OS X Lion, the forthcoming 10.7 release) copy and then promptly spill the beans to all and sundry. But frankly these look like small beans indeed. It seems as though Apple has tweaked quite a bit, but hasn't introduced any fundamentally new features into the operating system.  It's all very well for Apple to delight in being different, but the radical differences between Apple's GUI and everyone else's just don't seem to actually make using the computer any easier, and leave me wondering whether Apple is really heading in the right direction.

Whatever else they may have going for them they certainly don't seem to have engendered a lot of loyalty in their fan base. Or maybe this was just a few bad apples (so to speak).,

Less Than One Week to Become an OSCON Speaker

Remember that the OSCON call for papers closes on February 7. My attempts to add more Python skills to the assessment panel have resulted in three new Python-oriented members (including myself). There really is room for Python to expand this year, so don't be shy.  Submit your talk now!

Lanyrd Already Awesome, Likely to Improve

Interesting news today that the company behind lanyrd.com is to receive attention from Paul Graham's Y Combinator. This is excellent news for a company whose product is already awesome (and written in Python, naturally, or why would you be interested).

I really get the feeling that Python is gathering momentum, and can't wait to see how well-attended PyCon is going to be this year (remember: attendance is capped this year).

PyCon: No Dead Kittens This Year

What, me, harm an innocent kitten?

Last year I was perhaps a little over-enthusiastic in suggesting that if people didn't sign up for PyCon then kittens might be harmed. After a certain amount of retraining at the hands of the ASPCA (who may be kind to animals but surely know how to chastise a recalcitrant human) I am sadder but wiser, and will be threatening absolutely no more tiny harmless furry animals. People misunderstand one's motivations so easily.

Anyway, the purpose of this post is to bring to your attention the fact that for the first time this year places at PyCon US are limited. There is a boat, and it's sailing, so if you want to be on it make sure you proceed straight to registration. Proceed directly to registration. Do not pass "GO", do not collect $200. Why are you still reading this? Didn't I just tell you to go straight to registration?

It is true that registration will have to increase somewhat (about 30%) over last year for the cap on places to be reached, but PyCon has had a growth spurt like that before, and places are definitely selling earlier than they did last year. And why should this be a surprise? After all, Python is one of the faster-growing programming languages at the moment. I have an idea there might not be many last-minute places available this year.

If you or your employer is looking for greater visibility among the Python community PyCon is by far the best place to achieve it, and there are still sponsorship opportunities available. Go to it!

And yes, I have already booked my place. I got the early bird rate. You'll have to get there earlier next year ...

Too Busy Writing About Python to Write About Python

Though the subject line of this post might seem self-contradictory it does in fact reflect why I have blogged relatively little of late, and particularly to the Python channel. I became something of a hermit towards the end of last year to put the third of a series of four Python classes into production with O'Reilly School of Technology.

Given that each class is required to represent the approximate equivalent of forty learning hours, I think we can agree I have been writing about Python. Just not in my blog.

Anyway, this leaves the fourth class, which will complete the basic certificate series. So now I have a somewhat different problem: in this last class I want to focus on making sure that successful students have all the fundamentals in their grasp as they set off to earn their livings or indulge their hobbies hacking in Python. I need to wrap the language up--encapsulate it, if you will--in a way that gives students confidence that they understand the whole thing.

In the past, various denizens of comp.lang.python (python-list at python dot org) have offered to help by reviewing existing materials, but I haven't yet cleared that with O'Reilly. Since I suffered some data loss around Christmas time those addresses aren't currently available to me, but I am hoping if I post a link to this article those who are still interested will make themselves known by commenting here.

If you have anything to add, dear reader, kindly do not hesitate to comment yourself. What bits or bits of Python could you absolutely not bear to be without? Techniques, patterns, representations, algorithms and documentation are all grist to the mill. Favorite pieces of code, and even descriptions of warts (but remember, this is a Python 3 class). Tell your friends, invite the gang around, wire in, have a party.

It would serve students well to have as many external links in the last class's lessons as possible, I think, as well as to offer rules of thumb and handy hints. All contributions making it into classes will be appropriately acknowledged.

Once the series is complete I then have to think about what other topics I might tackle. Or should I take a rest and blog a bit more?

Disclosure: the author of this blog will benefit financially from sales of the courses mentioned above.

Serious Python Conference Information

Just in case the world needed reminding how active the Python conference world is, I did a search for Python conferences on the lanyrd.com site. At time of writing the query returned 84 conferences in the last couple of years. This is gratifying, but I am pretty sure it's only the beginning.

Answering My Own Question

If I had paused a little longer, or perhaps even taken a slightly fuller screen shot, I would have realized that my original capture was not of Twitter's @holdenweb home page (which is actually shown above, complete with tweet input widget—though it will link to your Twitter home page, as I don't know how to send you to mine; I don't know what you will see if you don't have a Twitter account, but most likely a sign-up page). In fact the previous post shows the @holdenweb profile page.

Never too old to learn, and never (apparently) too old not to need to. Thank you for reading. I now return you to your normal programming.

Wossup, Twitter?

A screen shot of today's @holdenweb Twitter home page

Maybe I am such an old fart that something is staring me in the face and I can't see it, but the "new Twitter" interface in its current manifestation is giving me plenty of reasons to consider usng a third-party application. Where do I enter my tweets?

Python at OSCON 2011

The OSCON organizers have asked me to help them recruit more Python talks there this year. It seems that maybe PyCon is the principal venue for those who are already using Python, but there is a vast audience of people who are interested in open source but not necessarily yet committed to Python, and such people are much more likely to attend OSCON, which runs July 25-29, to stay in touch with a range of open source technologies rather than ust Python alone.

Unlike PyCon OSCON is a full commercial conference. Tutorial speakers get help with their travel as well as an honorarium for talking, regular speakers get a free regstration. Both times I have attended I have managed to cover the majority of my costs. And, of course, OSCON is back in the wonderful city of Portland for the second year after a brief flirtation with California. It's a great place to visit, and by the time the conference comes around I hope to be living there.

If you are interested in going one further and would like to join the team that puts the Python syllabus together then drop me a line or simply make a comment here. That's another way to pay your way to OSCON!

The Future Has Arrived

So, after a very nice lunch I am sitting here typing this blog post on my MacBook Air (a device well beyond the limits of the imaginations of most of the science fiction writers of my youth) using a device which I invented three years ago, as I imagine did many other switched-on technologists.

Fortunately unlike most of us Novatel wireless went a bit further and actually built what Verizon are marketing as the "MiFi". It's a device about the size of a credit card and maybe just over a quarter of an inch thick incorporating a cellular broadband wireless router and an 802.11b/g wireless interface.

So to get Internet-connected all I have to do is switch it on (the only control is an on/off switch) and my laptop associates with it and is immediately connected to the 'Net. Not only that but the connection can be shared by up to five wireless devices, so my lucky pals will also be able to take advantage of the connectivity.

As far as I can see the MiFi is a sensible little device that is destined to become the preferred way of accessing the Internet. I hope that coupled with solar chargers it might also help African and other less-developed nations to build a viable Internet infrastructure (not that this will necessarily be their most urgent need).

Quite where this leaves the plans to enforce an Internet ban on copyright offenders in the various jurisdictions of the world I have no real idea. Technology sometimes makes fools of those who would control it, and it is only a matter of time before the MiFi or something similar is available as a "pay as you go" service to all comers.

Equal-opportunity (network neutrality) is an excellent ideal. The problem is with those who would abuse their power to change the laws unfairly to their own advantage, and those who stand idly by as their rights are eroded. The Electronic Frontier Foundation is celebrating its 20th anniversary in 2011. Celebrate the new year by joining the EFF or at least making a donation!

Linux Journal Readers Choose Python Again

I've just noticed the latest Linux Journal, which contains details of their 2010 readers poll. In former years they attempted to distinguish between "programming languages" and "scripting languages" but last year and this they just allowed readers to vote for any languages they liked in either category.

The interesting outcome was that readers voted Python as both the best programming language and the best scripting language. Thanks, Linux Journal readers!

DjangoCon Retrospective

Tumultuous events seem to occur in clusters in my life, and the week of DjangoCon found me in the middle of one such cluster. This being more of a professional than a personal blog I concentrate here on the conference, where the tumult manifested itself first in the shape of a chance meeting with a congressman. The next day I helped the Conference Direct team prepare for evening registration and helped people sign in.

The three days of DjangoCon were a most stimulating time. The Django community has its share of characters, and being among so many was truly a joy. I ended up just up the corridor from James Bennett[1], who is also quite a social person, and so a certain amount of partying ensued (which I regret to say I was not really in the best mood to enjoy). James was happy to entertain the people I kicked out, and they were apparently happy to be entertained, so we achieved a pleasant win-win.

Conference time is usually fairly closely scheduled, and DjangoCon is no exception. I used my "keynote" to try to get people relaxed and in the mood to enjoy themselves a little during all the serious learning that was taking place [2]. Then it was into tracks and running. Lunch seemed to go down well and everyone appeared to enjoy the breaks. Due to the generosity of our sponsors it was possible to provide a continental breakfast this year as well as lunch and breaks. After James Bennett's keynote were the lightning talks, which I had to miss because I was running a Python programming competition (won by Justin Lilley, against some worthy competition). I finished the day with a quiet room-service dinner with Kirby Urner, who was acting as Snake Wrangler and my general factotum.

This reminds me to mention that the Django pony was again in evidence as was Naga, the PSF_Snake. Everyone who came to the conference received their own "My Little Pony" toy, and I believe I may have detected early signs that the Django community is moving towards being "ponied out". The pony was the subject of a question during my keynote that rather sandbagged me - I was unaware of strong feelings in certain elements of the community about the pony as a symbol. Having said which, honesty is the best policy and I said what I believe.

Day two moved on in strong style. I was much preoccupied with how best to try and accommodate a visiting speaker. In the end we just moved the whole afternoon forward by a quarter of an hour, leaving a 5:15 to 5:30 slot for Congressman David Wu. I don't really know what the audience made of being addressed by a congressman. The decision to invite him was a no-brainer for me. I would rather hear a bit of "why ever did you bring a congressman in to speak?" that any "why on earth didn't you ask the congressman to speak?" Congressmen aren't like buses, they don't come along on a regular schedule, and the particular congressman in question happens to have impeccable science and technology credentials. He certainly seemed reasonably well-briefed on open source, which is a big thing in Portland right now.

The early part of the evening was spent at a reception my company gave, which a couple of campaign staff also attended. I entertained them a little longer after the reception finished, and then went downstairs to grab a roomful of people to help me consume wine and food I had ordered for my room. This turned into a most stimulating party, but at midnight or so I realized I needed to get some sleep. Fortunately James sent a scouting party down, and a transfer to his rooms was affected with only one use of the verbal crowbar.

Later to bed than I had anticipated, I had to rise early to attend a breakfast meeting away from the conference. As I returned it was gratifying to learn that the bacon had been delivered for breakfast at DjangoCon. The hotel had to serve the savory pastries late and in insufficient quantity on day two because their bakery supplier let them down. They asked if we would like a per-delegate refund, but I suggested instead that they add bacon to the Thursday menu, which they duly did. Unfortunately the bacon was apparently set down in the middle of the general display, and although I made light of it at the time[3] it would have been nicer for the vegetarian population if it hadn't been thrust under their noses.

The day appeared to go well. At lunchtime I gave away books from APress and O'Reilly, our two media sponsors, and awarded prizes for various pieces of arcane knowledge ("What's the animal on the front of this book?"). The only slight logisitical glitch in the proceedings occurred right at the end, when the short length of the sprint preparations caught us on the hop. I got a call and immediately went down and "turned the bar on" so people could get a drink, and a couple of minutes later Nancy from Conference Direct arrived with the drinks tickets and it was business as usual.

I like to think that most people had a good time, and I have tried to deal with criticism as it was raised in so far as I could. Sometimes, though, people insist on waiting until they have left and then making negative remarks, and I have seen a few such:

What I learned at #djangocon: 1. Not enough core committers. That's it. Total bummer of a conference #circlejerk I hope this twitterer realises that the chairman and community volunteers are the right people to take this up with, and there has already been talk of an Exreme Django track.

Oh fast internet, how I missed you! Back from #djangocon.  I don't think the Internet service was that bad (considering that PyCon spends four times as much and still gets it wrong when they don't let Sean Reifschneider do it), and there weren't extensive complaints as far as I knew (except maybe one fairly short period). Or maybe the author is, like me, a happy consumer of fiber to the customer premises. You do get spoiled ...

What I disliked most about #Djangocon: I have a toy for a 3 year old kid as a shwag. The pony isn't funny anymore. And yet, such is the power of the six dollar pony that you kept it despite the fact that you hate it. Nobody was checking at the doors to make sure you took your pony home with you. You're just a schwag collector, aren't you?

@pydanny please be careful around those dirty Django men, @pythonchelle watch out too #djangocon. I don't remember who tweeted this, but I hope the dissatisfaction was momentary.  Dirty Django men? Not dirty (mostly), simply in way too large a majority. We need to work on that. And the dirt ...

http://isalexacoredeveloper.com/ is just cruel #djangocon I have to agree, it is, but I don't think that this is in any way DjangoCon's responsibility [4].

Overall I think most delegates enjoyed their conference. While running these events can be a little stressful it is rewarding to feel that the conference has helped in some way to advance the state of the Django art and allowed people some relaxation during the conference "white space".

My thanks to all DjangoCon US's sponsors, and to everyone from the Django community who helped to make the event such a success. I am already looking forward to 2011.

NOTE: DjangoCon US 2011 will be held at the Hilton Portland and Executive Tower on September 6-8.

[1]: OHWar quote: "Beer. My suite. Now."
[2]: As is perhaps indicated by the title: Matters of Little Consequence or The Care and Feeding of Ponies.
[3]: OHWar quote: "Screw the vegetarians."
[4]: OHWar quote: "Pinax is sort of a reusable application thing." As you will have seen if you followed the link, Alex's status has changed since the site was instigated, and the question is now answered in the affirmative.

PyCon Financial Assistance

PyCon US is again offering assistance to delegates who may not be able to cover their own travel and/or accommodation costs. Applications have now opened, and the announcement page contains a link to the application form (separately linked here in case that's all you need).

While there can be no guarantees of financial assistance I can think of many delegates who have enriched the conference by their presence only because they received assistance. And those from places where few people even travel abroad can attend and set minds on fire when they return home. So financial assistance is a real investment in broadening the Python community, and I hope it will continue to be so.

There's a deadline, though, so don't spend too long thinking about it!

Templating Systems

Recently Danny Greenfeld wrote a blog post called Stupid Template Languages in which he attempted to advocate the Django approach to templating. Basically this is to keep any logic in the templates as simple as possible (but no simpler). So it includes the ability to write for loops but not, for example, to define re-usable procedures with parameter passing.

Armin Ronacher reacted to this with a blog post of his own called Not So Stupid Template Languages, in which he attempts to defend Mako and similarly complex templating schemes. I don't see the Jinja2 examples he gives as countering the main thrust of the Django wish for simplicity in templating. He does, however, explicitly point out the weaknesses of the Django include mechanism, and highlights the "macro" feature of Jinja2, which is its equivalent of Django's include feature. Jinja2 macros require that data is provided to the call by argument passing rather than simply being extracted from an implied context. I agree that requiring the use of arguments is probably a good way to make the couplings explicit.

This doesn't however, negate the value of keeping complex logic out of the templates. This desire originally comes from Django's background in the newspaper industry, where the designers were definitely not programmers. So they didn't need a hugely complex templating languages, and were prepared to ask the programmers to generate in their views the values required by the templates. I have worked with tools like Mako, which includes the ability to write very complex logic which should really be part of the controller rather than the view. It doesn't help that Django's designers perhaps didn't understand MVC that well, otherwise they would have called views "controllers" and templates "views". Nowadays I believe they call it the "model view template" model, but that still doesn't avoid the ambiguity about views and whether they should include logic or not.

Even with a pure MVC architecture Jinja2's argument-passing enhancements would be an improvement - as Armin says, making the couplings explicit would be a Pythonic thing to do. The nice thing is that argument passing could be implemented as a backwards-compatible extension, where a call without arguments could fall back to using the context as a default global namespace.

Larry Ellison Prostitutes Java?

I just saw a tweet from a character called Zed Shaw. Zed is a colorful guy, and a man of determined opinions. If he thinks something is wrong he will say so, most likely dropping f-bombs freely in the process.

His first tweet was rapidly followed by another.

Thinking about it, I cannot remember installing any other language and have it dick around with the settings on some other application.

I thought possibly Zed had caught the toolbar from somewhere else (I came within an inch of simply throwing WinAmp away when it intruded on my browser's search facilities in a similar way), but sure enough there's a (sadly undated) article at java.com entitled How do I unistall the Yahoo! toolbar from Firefox?, a clear enough indication to me that Java is likely responsible for Zed's woes.The FAQ explains about not just the Yahoo! toolbar but also the Bing and Google toolbars, but it appears that only the first two are installed by Java.

I suppose I could be wrong, but it seems to me that this behavior is new since the Sun empire was assimilated into the hive-mind by the Oracle collective. The fact that you can deselect the install options is neither here nor there. The real question is, what's the strategy behind this change? I can hardly believe that any revenues this hare-brained scheme might bring in are of the slightest significance to the world's second largest software company. But why would the company that effectively owns Java (in so far as anyone can own a language) deliberately try to discredit it in this way?

We can only speculate. Let's make it interesting ...

Lazy Programmers, Installment 315

So, during a brief engagement with ING Direct's web site I receive the following error message:

 It was complaining because a) I had used a one-digit form for the month (i.e. 9/19/2010 rather than 09/19/2010), and b) because I had just entered the telephone numbers, as is my wont, as ten-digit strings.

What I want to know is, is there any user-centered QA in effect on this site? Who exactly decreed that dates had to be entered in that inflexible format? Who reviewed the stupid decision to require that specific format for telephone numbers, when it's actually much easier to simply throw away anything that isn't a digit, store that and then format numbers when they are printed? Finally, does anyone at ING care how much trouble they put their users to?

See, I've been in the business long enough to remember the days when computers were there to make things easier for user-type people, not for the programmers. This is clearly a lost approach, one sees so many ridiculous requirements such as the above.

What Do We Tell the Government?

This coming Thursday I am speaking at GOSCON, a conference organized with the specific intentions of informing governmental IT users about open source. Most of the speakers will be from the government side if the last such event I attended is typical, so I am giving a talk called No Free Lunch. The idea is to suggest ways that government can maximize the benefits of using open source, and provide open source teams with resources that will enable us to increase production, so this isn't just a talk for the Python Software Foundation - I want to represent a wider community if I can.

Twitter friends have already provided some useful ideas, but I will be happy to weave more in there if you can provide them. So, what would you like to see government (which in this context is principally the executive branch, but I am happy to include points for the legislative and judicial branches if you have them) doing for open source?

Pony Outlook Fine, Say Wu and Holden

There's been some reportage about the presence of Congressman David Wu at DjangoCon this year (to which the most I could do yesterday was briefly allude). After my Labor Day post I think I can safely say that Chairman Wu's presence was reassuring to the many at DjangoCon who might have been seriously inconvenienced by the pony shortage, which has been discussed in panels at the conference.

Of course it would have been terribly unfair to the Congressman to have asked public questions, and as he is in a busy campaign I know you would not have wanted me to delay him in your name. After a breakfast meeting this morning I am happy to be able to tell you, though, that his staff have convinced me there really is no sign of a pony shortage (and I doubt many reading this will have bought more ponies than I of late). In fact, had I known that I could have procured the ponies locally (Oregon price: 50c less than manufacturer's) I might have saved the cost of carriage, too. Next time I'll just call the store in advance.

If further reassurance is needed, I would like to note that after a third meeting with Congressman Wu I am happy to endorse him as a two-pony family man. Those ponies will be in safe hands, and the people of the First District of Oregon would be ably represented by a Congressman of such integrity.

Open Source and Politics Mix

Kirby Urner has written this great summary of the events of late Wednesday (DjangoCon time).

DjangoCon Triggers National Pony Shortage?

Hasbro Toys today took the unprecedented step of insisting that pony supplies were "absolutely not a concern" in response to persistent rumors that a foreign national had cornered the market  in ponies. A spokesman said "while it is true that there has been an unusual demand for ponies in the Portland area our normal stocking policies were quite adequate to meet demand. Anyone who wants a pony can have one".

The House Committee on Science and Technology is rumored to have directed its Subcommittee on Technology and Innovation to investigate this potential shortage of a strategically important resource in the creation of critical national web site infrastructure. The chairman of the Subcommittee, congressman David Wu of Oregon's first district, was not available for comment. Hasbro could not explain why American pony manufacturing facilities had opened on Labor Day, normally a national holiday.

Preparedness, Privilege and Discrimination

Coming very late to the party I noticed a blog entry from June about the JavaScript community's response to Google's financial support to allow more women to attend JSConf Europe. It sounds like it was indeed the usual real can of worms, and many of the comments show the usual lack of appreciation of the problems from the privileged side of the issue (in this case men, since it is a gender issue and women appear to be about as well represented in the JavaScript community as they are in the Python world, which is to say hardly at all).

The reason I found it interesting was that this year, for the first time, PyCon also used Google's support to fund the attendance for more women, who came from as far afield as Roumania and India. I was gratified to be told by several of those who would otherwise not have been at the conference how happy they were to have had the chance to attend. (In truth I had little to do with it: Google should get the credit for the funding, and the actual hard work was done by Peter Kropf and Gloria Willedsen). In the Python community's case there was one short period of adverse comment on the #python IRC channel, which resulted in my exchanging emails with one person to explain why I thought it was a good idea to encourage more women to be at PyCon. End of story, except that in 2010 women represented 11% of PyCon attendance, up from 2% the previous year. I count that as one of the better results of my time as chairman.

Now I am not saying this to be smug, but because I believe there was a reason for the difference in the reactions. Last year the PSF, at Guido van Rossum's urging, started a diversity mailing list which discussed the questions of race, gender and other discrimination extensively and sometimes acrimoniously. Eventually this led to a proposal for a "diversity statement", which was referred to the membership where it triggered another round of extensive and sometimes acrimonious discussions, leading to a referral back to the diversity list and a further proposal which was finally accepted by the membership more or less unchanged and adopted by the Board:

The Python Software Foundation and the global Python community welcome and encourage participation by everyone. Our community is based on mutual respect, tolerance, and encouragement, and we are working to help each other live up to these principles. We want our community to be more diverse: whoever you are, and whatever your background, we welcome you.

This may not be the best statement ever, but if anyone bothers to look it does make it clear that these issues have been addressed. Thus anyone who feels discriminated against can decide that at least they would have a chance of a fair hearing should they choose to complain (which, sadly, I imagine most don't, instead choosing to vote with their feet). Similarly, anyone about to indulge in discriminatory behavior might think twice before doing so.

The hidden benefit of this long-drawn-out process was the creation, on the diversity list, of a corpus of varied individuals who had discussed these  issues and hammered out a shared approach to the problems that included a refusal to punish individuals for things done out of ignorance. It also meant that when one speaker used a slightly ill-advised graphic in a presentation the issue was dealt with then and there in a very direct manner without any recriminations, and I didn't even get to hear about it until  much later that day. The speaker was advised that the material was inappropriate and that therefore the slides and the video of the talk wouldn't be published, and hopefully left without feeling that they weren't welcome at next year's conference.

I hope that the JavaScript community manages to develop its own understanding of diversity issues and its own process for dealing with them. I know it took up a lot of my time as PSF chairman and gave me some uncomfortable moments (and does not exempt me from the results of my own stupidity in the future), but I am glad it led to a tolerant community process that nevertheless has made it clear that discrimination is not acceptable.

The Hiring Smiley Curve

I recently commented on how hi-tech companies seem to advertise for "rock star" developers. This appears to me to be a little shortsighted. Maybe Google have enough money to do it, but the average company looking for a singer wouldn't be able to go out and hire Mick Jagger or Van Morrison (yes, I know, I am showing my age - it will become obvious this isn't an accident). So why do they think they can afford the software equivalent?

As I mature (that sounds so much less pejorative than "grow older", doesn't it?) I have realized that my decision to eschew the corporate world and plough my own furrow wasn't entirely disastrous. I have been self-employed, or an employee of a company which I owned, for over twenty years now. The last large company I worked for was Sun Microsystems, and I left in 1988 after realizing that I was a misfit for the corporate environment. Since then I can honestly say I have never worked for a more charming or delightful person. Nowadays my boss never hesitates to take my interests into account.

Just occasionally, I have toyed with the idea of third-party employment, most recently with Google (twice). The first time I explained before going for interview that I was not prepared to relocate to the West coast . At the time I was moving back from the UK to the USA, and had just bought a house in Virginia, so I was a little surprised to learn three weeks after my interview that "we aren't prepared to make a remote hire, but would consider you for Mountain View". Large company mentality: ignore what the potential recruit wants, and try to hire them anyway. No sale. About a month ago they called me to ask if I would consider employment in the DC area, but it turned out they were only hiring for Java projects. I used to write Java but I'm all right now, so the recruiter and I agreed to part company amicably after ten minutes on the 'phone.

None of this amounts to a hill of beans but I was encouraged about a year ago, when I was talking to someone in New York City about experience levels and consulting opportunities, to learn about the "smiley curve" theory of work rewards. In brief, this theory says that you hire young raw recruits because they are full of energy and don't cost much. As people become more experienced they expect more money but their productivity doesn't go up commensurately, so they are less desirable but you need them because there aren't enough young turks to do all the work. Then, as you mature, your desirability goes up again because (direct quote, as far as I can remember) "you have seen everything and you know everything".

If true, this is quite encouraging. There must be lots of companies looking for someone as experienced as me. The only question now is whether they can afford me!

Apple Going Over the Top?

Yet again I rejoice that I am not an iPhone user. Indeed, given this latest news I might well just chuck  my Mac mini away in protest (no, you probably don't want it - it's an aging obsolete PPC mini from about six years ago).

In news from the Electronic Frontier Foundation I learned today that Apple has applied for patents on method of spying intrusively on the users of their devices. Here's a partial list of possible applications:

• The system can take a picture of the user's face, "without a flash, any noise, or any indication that a picture is being taken to prevent the current user from knowing he is being photographed";
• The system can record the user's voice, whether or not a phone call is even being made;
• The system can determine the user's unique individual heartbeat "signature";
• To determine if the device has been hacked, the device can watch for "a sudden increase in memory usage of the electronic device";
• The user's "Internet activity can be monitored or any communication packets that are served to the electronic device can be recorded"; and
• The device can take a photograph of the surrounding location to determine where it is being used.

So enjoy your iPhones. I won;t be doing business with a company that thinks this way. Sigh. I suppose that means the new laptop will have to be PC based.

Windows Vista Mystery Shares

For reasons best known to Microsoft, when I try to delete a folder which has been shared (through the Explorer interface) it takes forever to complete. This would not be so bad if there were just one or two shares, but sadly (for reasons best known to Microsoft) a large number of folders randomly appear to have become shares (see the screen dump of a portion of my home directory at the right). I have no idea how these folders became shared. It certainly wasn't any intentional act of mine, and heaven alone knows what this does to performance.

Now, you are probably wondering why I don't just switch off sharing before I delete the directory. The answer to that is that although Windows is displaying the folders as shared, it doesn't really seem to believe that they are shared. So there doesn't appear to be an easy way to switch this sharing off.

If I had some idea how it had been switched on in the first place that might help, but as with so many other aspects of Windows performance this remains a mystery. If someone cold offer some insight I'd be happy to find out what's going on here.

Schmidt Foot-In-Mouth Attack Continues

Wow, two consecutive Eric Schmidt posts. But this really is quite newsworthy. An interview with Wall Street Journal published on Saturday claims

He predicts, apparently seriously, that every young person one day will be entitled automatically to change his or her name on reaching adulthood in order to disown youthful hijinks stored on their friends' social media sites. 

This really is the most patent hogwash. Surely someone like Schmidt, with a brain the size of a planet, could foresee that if such name changing were to become commonplace it would inevitably lead to the creation of services that mapped between past and present-day identities? Given the ability to identify images recently demonstrated by sites like tineye.com it's only a matter of time before changing your name will no longer be a way to erase the records of your misdeeds.

This makes it all the more important that privacy and basic information security become high school subjects, but alas the corporate overlords that have the ear of government in most developed (and many less-developed) countries will be attempting to make sure that isn't a priority, because it isn't in their interests.

Eric Schmidt Looks Forward to Big Brother

"You only need to give up
just a little bit of freedom"

Well, Google's senior management are really keeping us guessing this month. I am not really sure any longer whether the company actually has any coherent point of view on individual privacy. Nowadays it seems you can't even expect consistency. In a startling repetition of his prior assertions that not only is Internet anonymity dead but that he wants to dance on its grave. According to a Read Write Web report Eric Schmidt, CEO of Google, included the following comments in his remarks to the Techonomy conference:

"The only way to manage this is true transparency and no anonymity. In a world of asynchronous threats, it is too dangerous for there not to be some way to identify you. We need a [verified] name service for people. Governments will demand it."

Any government that requires individuals to give up their rights to anonymity is a government past its sell-by date. To my mind this reveals further insight into Schmidt's "what's good for business is good for the people"  view exemplified by the much-discussed recent joint statement with Verizon on network neutrality. It's fairly obvious that Schmidt sees government's role as paving the way for corporations to increase their profits, not preserving the freedoms of the citizenry who elected it. This is so far from "government of the people, for the people and by the people" that it's apparently time "do no evil" was replaced by "make more money".

"But that would hurt Google's
stock price!"

Yet it was only in May (yes, three months ago) that Schmidt was publicly suggesting that as far as Google was concerned "privacy is paramount". Does he really know what the priorities are any longer? It seems like we can dismiss any further utterances as the self-serving flip-floppery of the 129th richest man in the world. What does he really think? Apparently it depends on which way the financial wind is blowing. The really depressing thing is it's transparently obvious that here we have a man who will do well in politics. "The Best Democracy Money Can Buy" indeed.

Tests That Test Your Tests

I just had an interesting experience with Steve Miller, the technical editor of the Python classes I am writing for O'Reilly School of Technology. We are just getting to the end of the second of four courses, and I like to think that we are moving right along: in the final lesson students write a simple GUI-based program that searches for and displays e-mail messages stored in a MySQL database.

I introduced test-driven development in the second course. Not only does this encourage good habits in the students, it also makes it somewhat easier to test some of their exercises (though I still do not have a good approach to testing Tkinter-based GUI applications). In time-honored fashion we start with tests and a program full of stubs, and then expand the stubs to pass the tests. For the email database the initial API is very simple: there is one function to store messages and two others to retrieve them, by primary key and Message-Id.

In the final chapter I start out with a very simple database table that stores the body of the message as a LONGTEXT column. The only other columns in the table (to start with—it gets more complex later) are the automatically-generated primary key and the Message-Id Header. The tests use a setUp() method that completely re-creates the database table and populates it from messages stored in a bunch of files:

 FILESPEC = "V:/Python2/Lesson12/MailData/*.eml"  
 class testRealEmail_traffic(unittest.TestCase):  
   def setUp(self):  
     """  
     Reads an arbitrary number of mail messages and  
     stores them in a brand new messages table.  
     DANGER: Any existing message table WILL be lost.  
     """  
     curs.execute("DROP TABLE IF EXISTS message")  
     conn.commit()  
     curs.execute(TBLDEF)  
     conn.commit()  
     files = glob(FILESPEC)  
     self.msgids = {} # Keyed by message_id  
     self.message_ids = {} # keyed by id  
     for f in files:  
       ff = open(f)  
       text = ff.read()  
       msg = message_from_string(text)  
       id = self.msgids[msg['message-id']] = maildb.store(msg)  
       self.message_ids[id] = msg['message-id']  

There were two relatively simple tests, one to test each of the retrieval functions in a fairly simplistic way by verifying the correspondence between the primary key values and the Message-Id headers using the msgids and message_ids dicts created during the set-up. The initial stub under test only implemented the store() function, so these tests were initially expected to fail:

   def test_message_ids(self):  
     """  
     Verify that items retrieved by id have the correct Message-ID.  
     """   
     for message_id in self.msgids.keys():  
       pk, msg = maildb.msg_by_id(self.msgids[message_id])   
       self.assertEqual(msg['message-id'], message_id)  
   def test_ids(self):  
     """  
     Verify that items retrieved by message_id have the correct Message-ID.  
     """  
     for id in self.message_ids.keys():  
       pk, msg = maildb.msg_by_message_id(self.message_ids[id])  
       self.assertEqual(msg['message-id'], self.message_ids[id])  

Steve and I had both run this code under much the same conditions as the students would, and verified that the tests did indeed fail due to the AttributeError exceptions raised by the missing msg_by_id() and msg_by_message_id() functions. Later steps have the student implement these functions, which makes the tests pass.

We were somewhat surprised to find when Steve ran his final checks that the tests were now passing, even though he had reverted to the original module with no implementations of the message retrieval functions! It took me the best part of an hour chatting with Steve to eliminate everything I could think of that might be wrong: no .pyc files left lying around, no odd path settings that allowed import from other copies of the code, and so on.

We finally tracked the issue down to something stupidly simple, as is often the case with bugs that have you scratching your head for an extended period: for production purposes the data files had been moved to an area where all students could share them (each student has their own V: directory). This meant that the setUp() method was not inserting any rows into the newly-created message table. The empty table in turn meant that neither test_message_ids() nor test_ids() was running the the body of the for loop, and consequently no AttributeError exceptions were being raised. The tests were passing even though the functions they were supposed to test had not been implemented!

My solution to this was to add a further test to verify that the table was not empty. That way, even if the other tests passed, this one would fail:

   def test_not_empty(self):  
     """  
     Verify that the setUp method actually created some messages.  
     If it finds no files there will be no messages in the table,  
     the loop bodies in the other tests will never run, and potential  
     errors will never be discovered.  
     """  
     curs.execute("SELECT COUNT(*) FROM message")  
     messagect = curs.fetchone()[0]  
     self.assertGreater(messagect, 0, "Database message table is empty")  

The check could have been added to one of the other tests but it seemed to make more sense to keep it separate, since several tests relied on the table having been populated.

In this case the error condition was that the test were passing! I am happy about this because it clearly demonstrates the value of test-driven development, even though the result I was getting was normally the desired goal of testing. It has also taught me to be more careful about tests in loops: if there is no guarantee that the loop body will execute then the tests inside it can be completely useless.

DjangoCon US is International

As registrations for DjangoCon US grow it's been interesting to see where people are coming from. I originally thought that it would be US-only. While US delegates dominate the lists as you might expect, we have people coming from all over the world. Here's a graphic showing the distribution of delegates across the globe.

I think it's a measure of Django's excellence that DjangoCon attracts people from so far away. I well remember when Jacob Kaplan-Moss and Adrian Holovaty first came to PyCon in Washington DC to describe the system they were putting together. At that stage Django wasn't open source, and the encouragement of the enthusiastic PyCon audience was a major factor in its becoming so. The software has come a huge distance in a relatively short time, and is now a major Python success story.

There are still places left at the conference if you would like to register. Portland is an intriguing city that offers a warm welcome to visitors, and the Doubletree is a green hotel with excellent accommodation. The conference room rate is only guaranteed until August 13, so make sure to book your accommodation soon.

Wave Goodbye

So Google's blog announced today that development of Google Wave "as a standalone product" will end because "Wave has not seen the adoption we would like". It's kind of a shame, because Wave was intriguing, but as an infrequent Wave user I found several issues that made it less than user-friendly. So here are a few points that developers might like to take home from the train-wreck that is Wave (100 developers for two years is a substantial investment, even for Google). The servers will continue to be available "at least until the end of the year".

1. Don't try to replace standard GUI components with inferior and non-intuitive substitutes. The Wave scrollbar was a user interface disaster, and a source of frustration to many of the users I interacted with.
2. Don't promote technologies that depend heavily on high-bandwidth connectivity, or at least not for Internet use. Many times I was left frustrated, not knowing whether the Wave had crashed or whether it was simply waiting for a server response.
3. Realize that even the best technologies need marketing and publicity. 80%+ of desktop computer users don't use Windows because it's the best system, they use it because it's the best alternative they know about. If people don't know about your technology they won't use it, and techies alone probably aren't the right user base to make a product viral.

In some ways I am sorry that the Wave didn't succeed as so many techies apparently thought it would. In other ways I am disappointed that the technology didn't really deliver on its promise. It's one of those things that needs to be ubiquitous to succeed.

Google's misstep with Buzz earlier this year probably didn't help either - it led to distrust about Google's intentions with regard to (or, worse, competence at securing) users' personal data.

So whatever the next big thing on the Web is going to be, it isn't going to be Google Wave. RIP.

Paypal Pending Balances

According to the PayPal user agreement:

10.7 Reserves. If you receive Purchase Payments, PayPal, in its sole discretion, may place a Reserve on funds held in your Premier or Business Account when PayPal believes there may be a high level of risk associated with your Account. If PayPal places a Reserve on funds in your Account, they will be shown as “pending” in your PayPal Balance. If your Account is subject to a Reserve, PayPal will provide you with notice specifying the terms of the Reserve. The terms may require that a certain percentage of the amounts received into your Account are held for a certain period of time, or that a certain amount of money is held in reserve, or anything else that PayPal determines is necessary to protect against the risk associated with your Account. PayPal may change the terms of the Reserve at any time by providing you with notice of the new terms. 

 What this doesn't say is that PayPal will also deny you the ability to earn interest on those funds while they are held in reserve.  Because Holden Web is organizing DjangoCon we currently have a substantial balance (which will soon disappear once the outgoing starts). So PayPal have decided that they can place $20,000 on reserve, thereby refusing me the right to earn interest on that money.

Naturally the "notice" I received just said that the funds were being held on reserve until further notice, and when I inquired about this the "explanation" I received didn't explain anything at all: it simply pointed me to the user agreement and suggested that I could find out what information they had used in the decision-making process by taking out a subpoena. Definitely not cool. In terms of customer service PayPal are scoring about 1 out of 10 here.

Death by Oracle

This is probably not an original idea. I am not one of those who complain about Oracle's acquisition, as a part of Sun Microsystems, of the MySQL mark and the associated software. Larry Ellison is plenty shrewd enough to make money out of open source - OK, maybe not as much as from proprietary, but when you are Larry Ellison you might figure you are getting close to enough anyway. Or maybe not.

But suppose someone evil in the Ellison empire were to decide to bury MySQL for ever, it strikes me that all they would have to do would be to bundle it under the Oracle installer. This gargantuan framework with basic Java look-and-feel and often totally inadequate error handling (disclaimer: I have little recent experience of Oracle, but I have used their products since 1986 or thereabouts and would be surprised if things had changed radically) would be quite enough to deter anyone who hadn't already forked over huge amounts of money for the software behind it. A free software release would simply not be worth the pain.

Progress, Of a Kind

Remember all the stuff that visionary management guru Peter Drucker was enthusing about in the 1980s - about how markets will appear and disappear, and collaborations will form to meet them and dissolve as the market disintegrates? Only after 30 years has the information technology industry been able to come  up with systems that stand some chance of meeting the requirements of such markets. Boy, are they enablers. Who wants to know what the vendor wants us to believe, when we can ask actual customers among our social networks?

Metaclass Madness (Python 3 Version)

Last week I was at PyCon Asia Pacific to deliver the opening keynote. Liew Beng Keat, the conference chair, was kind enough to invite me to give a technical talk as well, so I brushed up a talk that I had given previously to the Icelandic Python User Group entitled Metaclass Madness. The material is fairly straightforward, but metaclasses have the reputation for making people's heads explode, so the title was something of a warning for the unwary.

The evening before the presentation I decided to update the code, so the current download includes not only the PowerPoint slides but also usable source code for both Python 2.x and Python 3.x.

I had thought of adding class decorators to the talk, but interestingly I realized that the example I was using didn't translate. The issue was that the metaclass is called with three arguments, the third of which is a dict containing the namespace that has been constructed during the compilation of the class body. So in the metaclass's __new__() method it is easy to decorate each method by iterating over the namespace dict and replacing each callable (whose name does not begin with a double underscore) with the result of applying a decorator to it.

A class decorator cannot work this way, though (at least with a new-style class, which is all you have in Python 3). The reason is that new-style classes use a dict_proxy object as their __dict__, and the dict_proxy does not all you to set items. Consequently, by the time the decorator gets called the class __dict__ is already pretty much set in concrete.

Since the particular example I chose deliberately omitted the methods whose names began with a double underscore someone asked me whether name mangling would affect the process. [For those who don't know about name mangling it is an attempt to protect "private" variables, those whose names begin with a double underscore and end with at most one underscore. See this documentation page for further details]. I was able to demonstrate interactively, after a couple of false starts, that mangling apparently took place *after* the call to __new__() (presumably in type.__new__(), which the metaclass __new__() method must call to ensure completion of the class creation).