Showing posts with label web. Show all posts
Showing posts with label web. Show all posts

January 7, 2010

Absent DB API Test Files Fixed

I saw a somewhat annoying comment from a web site reader recently:
The following message was received via the web site.

From:       
Email:      
Subject:    /PyConTX2007/dbapi.tgz  Not Available
Telephone: 

I am grateful to the anonymous commenter for letting me know that I'd failed to redirect certain essential links in the recent web site migration.

It would have been even better if I could have replied to the comment to let my reader know that the problem is fixed (which it now appears to be). Alas he or she left absolutely no contact information whatsoever.So all I can do is record the fix and hope that this file wasn't being urgently sought (which I doubt it was).

December 3, 2009

Google Thinks I'm Naughty

I've been doing quite a lot of web stuff lately, and this has involved doing link checking on my site. Since it now has AdWords in it, this means my (totally unsophisticated) web crawler has been following the advertisers' links. Oops.

This afternoon I get an email from WellKnownSearchEngine saying:
It has come to our attention that invalid clicks have been generated on your Google ads, posing a financial risk to our AdWords advertisers. Please note that any activity that may artificially inflate an advertiser's costs or a publisher's earnings is strictly prohibited by our program policies.
You can imagine the rest. Dire Warnings, Serious Threats, and so on, coupled with a refusal to offer specifics. So I just thought I'd ask: what tools do you use use for link-checking, and how do you avoid this issue?

February 24, 2009

How Marketing Sucks (pt. 117)

I came across this interesting (but sadly totally predictable) little snippet in a marketing blog (which, to be fair, did point out that this technique is appalling) today. Apparently Jenny lives in Brooklyn, NY.

But wait, there's more, she also lives in Pensacola, FL:


Make user geolocation possible and marketers will use it to add false appeal to advertising claims by pretending to have local consumers using their products. No wonder that marketers are held in such high esteem.

The web used to be a place of wonder. Now, sadly, much of it is about as interesting as commercial television. I wonder if a lobotomy would help (me, that is)?

January 28, 2009

Google Groups Plus

OK, I have tired of having ideas that I never get around to implementing, so I am going to take a page out of Seth Godin's book and just write this thing up so somebody with more time and inclination can get rich from doing it. Or at least provide a service to people who could use it.

It's time that Google Groups (or at least some of them) were available via NNTP. Sure, I can use the somewhat quirky and always less than satisfactory web interface that Google provide. Or I could set the groups I am interested in to deliver the posts via email. But an NNTP client would just be so much more convenient and usable. So what we need is a system that scrapes Google Groups and offers NNTP access to the resulting content.

There is, of course, the minor issue of Google's terms of service. The most significant part says:
Google authorizes you to view and download a single copy of the Materials solely for your personal, non-commercial use. You may not sell or modify the Materials or reproduce, display, publicly perform, distribute, or otherwise use the Materials in any way for any public or commercial purpose without the written permission of Google. Special rules may apply to the use of certain software and other items provided via the Services, and are noted where appropriate.

If you would like information about obtaining Google's permission to use any of the Materials on your Web site, please send an e-mail to groups-support@google.com.
So we will have to persuade Google not to be evil - I imagine they will be upset not to be able to insert advertising in the stream, but if they did that wouldn't be Usenet, now would it? I would hope Google would be mindful of how they came by the archival content that started this venture in determining how to license the materials.

I know there are people in the Python world who could put this together in under a week.

While I am dealing in quirky ideas, here's another one, in this case to monitor (and hopefully control) spending on government credit cards. Could this work?

November 11, 2008

United.com Broken

So I have this trip coming up to San Diego, and I fly tomorrow. I get this friendly little Web-2.0 e-mail from United. It comes from an address in a subdomain of p0.com, telling me that United are using Yesmail, and it includes a link for me to follow to "view my complete itinerary". They even have it link to a server in the p0.com domain so my browser doesn't think this is a phishing attack. The only problem comes when I click on the link, and repeatedly see 
Connection Interrupted
The connection to the server was reset while the page was loading.
The network link was interrupted while negotiating a connection. Please try again.
"Not to worry", I think to myself, "I'm a United frequent flyer (God help me), so I can just log in to my Mileage Plus account and check my itinerary there". Of course this wouldn't have been much of a post if that had worked, but good old United didn't let me down. Every attempt to log in gives me
Redirect Loop
Firefox has detected that the server is redirecting the request for this address in a way that will never complete.
The browser has stopped trying to retrieve the requested item. The site is redirecting the request in a way that will never complete.
Attempts to retry just bring up the same message again. Wonderful. Let's hope the people who run the web site have nothing to do with keeping the 'planes flying.

[PS: Fortunately this appeared to be a transient condition. The site now responds normally.]

April 8, 2008

Twisted Bite the Web Bullet

The Twisted community has decided it's time to clean up its web act. The problem for a while has been that there were three "competing" packages that might have formed the basis of any web application: twisted.web, nevow and twisted.web2.

Now they have decided that twisted.web is the right namespace, and the immediate plan is to backport the stable functionality from web2 and the nevow application server into twisted.web, giving a more advanced set of web functionality in a way that offers best compatibility to existing users. Nevow will intially remain, as a pure templating solution, but there's a possibility that functionality might also be repositioned eventually.

This has the smell of a compromise, but as compromises go (and as you'd expect in the Twisted world) it seems to be a pragmatic engineering solution to the current rather confusing situation.

March 14, 2008

Practical Applications of Agile (Web) Testing Tools

This was a practical tutorial by Titus Brown and Grig Georghiu, two accomplished presenters who have given popular presentations at the last two PyCons. Titus opened the presentation, and told us he wanted us to be "test infected" - to make testing a necessary part of our development process. He confessed he feels a sense of physical discomfort when he writes untested code.

He gave us a number of practical guidelines, the most interesting of which were: start small; focus on actual problems; use continuous integration. The goal of testing should be to fail as quickly and as hard as possible: failure is good, because it reveals a need for revision. You should plan for testability, and as your experience level increases you will find that you naturally tend to do this.

Use a hierarchy of tests. The initial tests should run in under 20 seconds; only if those tests pass should you run the full test, which will take longer; then run regression tests, and then (if you have defined them) the acceptance tests. The point is that much of this work isn't done on your time: your continuous integration may be an overnight run, but you don't have to stay while they run. Titus feels that any test that takes over a couple of hours has limited (though definitely not zero) value, due to the lack of immediacy in the feedback it provides.

Taxonomy of testing: unit tests, functional tests, regression tests, UI tests, acceptance tests, integration tests, continuous integration, performance tests. Do you even know what all these are? Which to you need to define and use?

The more complicated it is to run the right tests, the higher the probability they won't get run. Titus talks about different strategies: TDD (Test-Driven Development) and SDT (Stupidity Driven Testing) , which he equated with TED (Test Enhanced Development). The general myth is that you are a bad person if you don't use TDD, but SDT is very important because you can ensure that bugs get properly squashed by adding specific tests after discovering one of the gruesome faults we all put in ur software from time to time.

Tests can be used to constrain your code; they force you to document your expectations, both internally and externally. internally, use more assert statements. Externally, write unit, functional and regression tests as "black box" tests to verify external behavior.

A "test umbrella" removes the need for memory. Everything should report back to a single source. This gives you a way to tell a new developer "run this; if it doesn't work, you have something to fix". You just have to remember to run the tests, not go through some complex sequence of commands that need separate documentation. This automation investment will reap big dividends in both time and quality.

Testing really helps when you want to refactor.

While code coverage may not be a good metric, it's definitely the case that if you don't run every line of code at least once in your tests then the possibility of failure becomes much higher. Line coverage is necessary, but not sufficient. Continuous integration helps you discover less obvious errors, like changes in process you forgot to document.

WSGI intercept is a useful test mechanism because it monkey-patches the Python web libraries.

[Interesting demos; I went for water for the speakers just before they started; my tests failed, and I never discovered whether they were supposed to or whether I had missed some crucial step while I was out].

The scotch tool is a recording proxy that will record everything that passes between the browser and the server: exceptionally useful for capturing AJAX interactions. It can also replay recorded sessions, and ensure that the responses that come back are "the same" (with minor exceptions such as ignoring date changes).

Figleaf is a code coverage recording tool. It saves it results in a pickle-based format, and can perform intelligent comparisons between tests, taking the union and intersection so you can easily discern what still needs testing.

Nose is a system that allows testing with a much lighter footprint than unittest. It will also all other doctests and unittests it can find if properly encouraged to do so, and can be used to add module-level setup and teardown.

Grig then took over and first described and then demonstrated Selenium, a testing tool written in Javascript that runs inside the browser (and probably one of the best-known web testing tools). It's one of the few tools that can adequately test AJAX functionality, and it's the default testing tool at Google). It has several components.

The core is augmented by an RC (remote control) module, and IDE (a Firefox add-on), and a Grid module that seems to relate to parallelized testing.

The biggest issue in defining Selenium tests is locating the required elements in the page. It would be much easier if web authors gave all elements IDs, but in practice you often end up using xpath, which has downsides. Many times there is no other way.

The IDE component allows you to record interactions with web servers, but again you have to be careful to use repeatable event sequences (e.g. pasting is not repeatable); the IDE is probably not as easy to use as Twill, but some AJAX applications are almost pure client-side, with very few server round-trips, and Twill is useless with these as it can't intercept the traffic.

Firebug is an essential component in developing the tests, as is the XPath Checker add-on, which helps greatly in generating the element locators.

The basis of Seleniun tests is a long string of locators and assertions. Selenium RC runs as a reverse HTTP proxy, bypassing the "same origin" cross-site scripting protections that normally insist that scripted content has to come from a single domain. By fooling the browser this way remote control allows you to write test of a remote server inside the local proxy. It also allows you to build simple looping constructs. Of interest to readers will be its ability to export test cases as (rather ugly) Python code. Google has a whole team working on Selenium, and others are also working to improve it, and extend it to broader browser support.

Grig then went on to describe buildbot, a continuous integration testing tool that allows a master system to run build (or indeed any other) commands on remote slaves and aggregate reports from the clients in a simple, readily available web format.

Although some people perform continuous integration on a single platform, buildbot was designed from the start to allow the master to drive the same tests across multiple clients and thereby discover cross-platform bugs that might otherwise go undetected.

Greg ran us through a configuration file. The learning curve can be steep, but once your buildbot is configured it usually needs little maintenance. He also highlighted the fact that someone has to keep on top of buildbots and avoid bitrot by displaying the currrently parlous state of the Python buildbot farm. Many clients are down, and those that are running appear to be failing their tests.

The evening closed with a brief discussion of Fitnesse, which is a higher-level tool that has you define tests declaratively in a language that allows you to refer to the concepts of the application domain; this makes them easier for the customer to comprehend, for example. The tests are maintained in a wiki.

I was beginning to flag a bit by now, so I can't really give you an intelligent summary better than Grig's "it's a Wiki that runs things". Ideally the client will at least understand the test descriptions.

All in all an excellent and enjoyable presentation that not only gave me some great ideas for testing but also left me more determined to introduce it into my web projects.

January 8, 2008

Windmill Web Testing Framework

If what this page says is true then Windmill will be very well worth some investigation. Of course the major problem with the Python web world is that so much is going on it's simply impossible to keep up with everything! Of course WSGI lays the foundation for much else, and it appears that Zope 2 now plays well with others.

I think I might start trying to make shorter blog entries just to note the most interesting developments as PyPI (Cheese Shop) contributions fly past at a rate of knots.

In the past I have looked at TurboGears, and was impressed how easy it was to implement the functionality of another web site. Recently I have been doing a similar thing with Django. We are very fortunate to have two such capable frameworks available for our web developments, and I am going to try and make sure that they both get some coverage in Python Magazine.

September 27, 2007

Blogger is Buggered?

Well, three posts in a row about Blogger is a little unusual, but I think this is merited.

From time to time I amuse myself by following the "Next Blog" links to get an idea of what was going on in the blog stream. If today's experience is typical then about one blog in three is complete spam, obscured by a pop-up ad window and full of links to pornographic sites. It seems clear that nobody is exercising any kind of critical control over Blogger content, and frankly it's got to the point where I am no longer sure I want to be associated with it.

I've been a web use for a long time, and it saddens me to see a potentially fantastic medium ruined by the swarms of lowest-common-denominator advertiser scum. If this is "what the people want" then they are welcome to it. I really don't think Blogger are doing themselves any favors by letting this situation continue.

September 15, 2007

ReStructured Text to Anything

I stopped using the "How'd I Get Here" Firefox add-on when I had to reload Windows last time, so I don't remember where I saw this, but there's an interesting new service that lets you convert your ReST documents to HTML or PDF with a choice of styles. Worth a look.

The authors give credit to many open source products including Docutils, Python, Django, MochiKit, LaTeX, ImageMagick, MySQL and Ubuntu.

It's good to see great services coming out of the Python community, and this is a nicely put together example of what you can do with open source.

January 18, 2007

Things You Don't Expect

It's strange. I use the web an awful lot (the company is called Holden Web, after all) and so I am used to seeing server error messages appear in my browser. So I had to stop and wonder just why I found this one so fascinating:

The answer finally came to me after a minute's puzzling: it's the very first Google server error I have ever seen. Given the amount I use Google and the scale of the service they now provide I think that says a huge amount about the quality of their engineering.