This talk actually used the differences between Python 2 and Python 3 as the major version differences, but these are just an extreme example of inter-version differences.
Tox uses a config file that allows you to specify (among other things) which Python versions should be tested, dependencies between tests and so on. But how do we cope with syntax differences, where(for example) Python 2 code will raise syntax errors. The 2to3 utility is a fast and very comprehensive tool, and you can test for Python 3 and run 2to3 automatically in your setup.py.
It is possible to generate a single source file that compiles under both Python 2 and Python 3, but this is not easy. Often it's easier to convert - particularly for tests, where automatic conversion may not work. Tests may be distributes or non-distributed. In both cases converting the tests to Python 3 is likely to be required. Automated conversion is always subject to failure.
It's possible to use different environments (virtualenv) for the different versions, and even with Jython. Although tox can be used from the command line, it was designed for use in continuous integration. It does need to be bootstrapped a little bit: it should be installed locally (probably under a virtualenv!) You can also define a custion PyPI index.
Summary: the most important thing is to be able to support Python3. The single source approach requires most discipline. Mozilla is hiring. Had to leave the Q&A to start a meeting. Sorry.
"Any sufficiently advanced technology is indistinguishable from magic" -- Arthur C Clarke
Musings from the technological frontier
Showing posts with label testing. Show all posts
Showing posts with label testing. Show all posts
March 11, 2011
August 6, 2010
Tests That Test Your Tests
I just had an interesting experience with Steve Miller, the technical editor of the Python classes I am writing for O'Reilly School of Technology. We are just getting to the end of the second of four courses, and I like to think that we are moving right along: in the final lesson students write a simple GUI-based program that searches for and displays e-mail messages stored in a MySQL database.
I introduced test-driven development in the second course. Not only does this encourage good habits in the students, it also makes it somewhat easier to test some of their exercises (though I still do not have a good approach to testing Tkinter-based GUI applications). In time-honored fashion we start with tests and a program full of stubs, and then expand the stubs to pass the tests. For the email database the initial API is very simple: there is one function to store messages and two others to retrieve them, by primary key and Message-Id.
In the final chapter I start out with a very simple database table that stores the body of the message as a LONGTEXT column. The only other columns in the table (to start with—it gets more complex later) are the automatically-generated primary key and the Message-Id Header. The tests use a setUp() method that completely re-creates the database table and populates it from messages stored in a bunch of files:
There were two relatively simple tests, one to test each of the retrieval functions in a fairly simplistic way by verifying the correspondence between the primary key values and the Message-Id headers using the msgids and message_ids dicts created during the set-up. The initial stub under test only implemented the store() function, so these tests were initially expected to fail:
Steve and I had both run this code under much the same conditions as the students would, and verified that the tests did indeed fail due to the AttributeError exceptions raised by the missing msg_by_id() and msg_by_message_id() functions. Later steps have the student implement these functions, which makes the tests pass.
We were somewhat surprised to find when Steve ran his final checks that the tests were now passing, even though he had reverted to the original module with no implementations of the message retrieval functions! It took me the best part of an hour chatting with Steve to eliminate everything I could think of that might be wrong: no .pyc files left lying around, no odd path settings that allowed import from other copies of the code, and so on.
We finally tracked the issue down to something stupidly simple, as is often the case with bugs that have you scratching your head for an extended period: for production purposes the data files had been moved to an area where all students could share them (each student has their own V: directory). This meant that the setUp() method was not inserting any rows into the newly-created message table. The empty table in turn meant that neither test_message_ids() nor test_ids() was running the the body of the for loop, and consequently no AttributeError exceptions were being raised. The tests were passing even though the functions they were supposed to test had not been implemented!
My solution to this was to add a further test to verify that the table was not empty. That way, even if the other tests passed, this one would fail:
The check could have been added to one of the other tests but it seemed to make more sense to keep it separate, since several tests relied on the table having been populated.
In this case the error condition was that the test were passing! I am happy about this because it clearly demonstrates the value of test-driven development, even though the result I was getting was normally the desired goal of testing. It has also taught me to be more careful about tests in loops: if there is no guarantee that the loop body will execute then the tests inside it can be completely useless.
I introduced test-driven development in the second course. Not only does this encourage good habits in the students, it also makes it somewhat easier to test some of their exercises (though I still do not have a good approach to testing Tkinter-based GUI applications). In time-honored fashion we start with tests and a program full of stubs, and then expand the stubs to pass the tests. For the email database the initial API is very simple: there is one function to store messages and two others to retrieve them, by primary key and Message-Id.
In the final chapter I start out with a very simple database table that stores the body of the message as a LONGTEXT column. The only other columns in the table (to start with—it gets more complex later) are the automatically-generated primary key and the Message-Id Header. The tests use a setUp() method that completely re-creates the database table and populates it from messages stored in a bunch of files:
FILESPEC = "V:/Python2/Lesson12/MailData/*.eml"
class testRealEmail_traffic(unittest.TestCase):
def setUp(self):
"""
Reads an arbitrary number of mail messages and
stores them in a brand new messages table.
DANGER: Any existing message table WILL be lost.
"""
curs.execute("DROP TABLE IF EXISTS message")
conn.commit()
curs.execute(TBLDEF)
conn.commit()
files = glob(FILESPEC)
self.msgids = {} # Keyed by message_id
self.message_ids = {} # keyed by id
for f in files:
ff = open(f)
text = ff.read()
msg = message_from_string(text)
id = self.msgids[msg['message-id']] = maildb.store(msg)
self.message_ids[id] = msg['message-id']
There were two relatively simple tests, one to test each of the retrieval functions in a fairly simplistic way by verifying the correspondence between the primary key values and the Message-Id headers using the msgids and message_ids dicts created during the set-up. The initial stub under test only implemented the store() function, so these tests were initially expected to fail:
def test_message_ids(self):
"""
Verify that items retrieved by id have the correct Message-ID.
"""
for message_id in self.msgids.keys():
pk, msg = maildb.msg_by_id(self.msgids[message_id])
self.assertEqual(msg['message-id'], message_id)
def test_ids(self):
"""
Verify that items retrieved by message_id have the correct Message-ID.
"""
for id in self.message_ids.keys():
pk, msg = maildb.msg_by_message_id(self.message_ids[id])
self.assertEqual(msg['message-id'], self.message_ids[id])
Steve and I had both run this code under much the same conditions as the students would, and verified that the tests did indeed fail due to the AttributeError exceptions raised by the missing msg_by_id() and msg_by_message_id() functions. Later steps have the student implement these functions, which makes the tests pass.
We were somewhat surprised to find when Steve ran his final checks that the tests were now passing, even though he had reverted to the original module with no implementations of the message retrieval functions! It took me the best part of an hour chatting with Steve to eliminate everything I could think of that might be wrong: no .pyc files left lying around, no odd path settings that allowed import from other copies of the code, and so on.
We finally tracked the issue down to something stupidly simple, as is often the case with bugs that have you scratching your head for an extended period: for production purposes the data files had been moved to an area where all students could share them (each student has their own V: directory). This meant that the setUp() method was not inserting any rows into the newly-created message table. The empty table in turn meant that neither test_message_ids() nor test_ids() was running the the body of the for loop, and consequently no AttributeError exceptions were being raised. The tests were passing even though the functions they were supposed to test had not been implemented!
My solution to this was to add a further test to verify that the table was not empty. That way, even if the other tests passed, this one would fail:
def test_not_empty(self):
"""
Verify that the setUp method actually created some messages.
If it finds no files there will be no messages in the table,
the loop bodies in the other tests will never run, and potential
errors will never be discovered.
"""
curs.execute("SELECT COUNT(*) FROM message")
messagect = curs.fetchone()[0]
self.assertGreater(messagect, 0, "Database message table is empty")
The check could have been added to one of the other tests but it seemed to make more sense to keep it separate, since several tests relied on the table having been populated.
In this case the error condition was that the test were passing! I am happy about this because it clearly demonstrates the value of test-driven development, even though the result I was getting was normally the desired goal of testing. It has also taught me to be more careful about tests in loops: if there is no guarantee that the loop body will execute then the tests inside it can be completely useless.
December 3, 2009
Google Thinks I'm Naughty
I've been doing quite a lot of web stuff lately, and this has involved doing link checking on my site. Since it now has AdWords in it, this means my (totally unsophisticated) web crawler has been following the advertisers' links. Oops.
This afternoon I get an email from WellKnownSearchEngine saying:
This afternoon I get an email from WellKnownSearchEngine saying:
You can imagine the rest. Dire Warnings, Serious Threats, and so on, coupled with a refusal to offer specifics. So I just thought I'd ask: what tools do you use use for link-checking, and how do you avoid this issue?It has come to our attention that invalid clicks have been generated on your Google ads, posing a financial risk to our AdWords advertisers. Please note that any activity that may artificially inflate an advertiser's costs or a publisher's earnings is strictly prohibited by our program policies.
June 27, 2009
Another United Web Screw-Up
So, wanting to answer a question about my flight times for EuroPython I clicked on an email link to "My Itineraries". Thanks, United, this couldn't be clearer.
April 11, 2008
Tweaking the Bytecode
Ned Batchelder has come up with an interesting hack for increasing the granularity of his code coverage testing. It involves modifying the code object associated with the code being modified: specifically, the co_lnotab member contains a starting line number followed by a sequence of (code offset, line number delta) pairs.
Ned's technique involves rewriting the member so that each bytecode appears to be on a single line on its own. While this allows him to ensure that all bytecodes are exercised, he hasn't yet ironed out the quirks in how to use that to present coverage information (which is hardly surprising, since most Python programmers could care less about the structure of their bytecode).
It seems to me, however, that by mapping the modified co_lnotab back to the original you could at least flag lines whose code isn't 100% covered by your tests, and this is surely valuable information. Ned's already thought of this, and includes the throwaway remark "The original line number information is lost. It would have to be stored off to the side to make this useful" towards the end of his blog entry.
So I believe we can expect coverage.py to start producing even more useful information soon.
Ned's technique involves rewriting the member so that each bytecode appears to be on a single line on its own. While this allows him to ensure that all bytecodes are exercised, he hasn't yet ironed out the quirks in how to use that to present coverage information (which is hardly surprising, since most Python programmers could care less about the structure of their bytecode).
It seems to me, however, that by mapping the modified co_lnotab back to the original you could at least flag lines whose code isn't 100% covered by your tests, and this is surely valuable information. Ned's already thought of this, and includes the throwaway remark "The original line number information is lost. It would have to be stored off to the side to make this useful" towards the end of his blog entry.
So I believe we can expect coverage.py to start producing even more useful information soon.
March 14, 2008
Practical Applications of Agile (Web) Testing Tools
This was a practical tutorial by Titus Brown and Grig Georghiu, two accomplished presenters who have given popular presentations at the last two PyCons. Titus opened the presentation, and told us he wanted us to be "test infected" - to make testing a necessary part of our development process. He confessed he feels a sense of physical discomfort when he writes untested code.
He gave us a number of practical guidelines, the most interesting of which were: start small; focus on actual problems; use continuous integration. The goal of testing should be to fail as quickly and as hard as possible: failure is good, because it reveals a need for revision. You should plan for testability, and as your experience level increases you will find that you naturally tend to do this.
Use a hierarchy of tests. The initial tests should run in under 20 seconds; only if those tests pass should you run the full test, which will take longer; then run regression tests, and then (if you have defined them) the acceptance tests. The point is that much of this work isn't done on your time: your continuous integration may be an overnight run, but you don't have to stay while they run. Titus feels that any test that takes over a couple of hours has limited (though definitely not zero) value, due to the lack of immediacy in the feedback it provides.
Taxonomy of testing: unit tests, functional tests, regression tests, UI tests, acceptance tests, integration tests, continuous integration, performance tests. Do you even know what all these are? Which to you need to define and use?
The more complicated it is to run the right tests, the higher the probability they won't get run. Titus talks about different strategies: TDD (Test-Driven Development) and SDT (Stupidity Driven Testing) , which he equated with TED (Test Enhanced Development). The general myth is that you are a bad person if you don't use TDD, but SDT is very important because you can ensure that bugs get properly squashed by adding specific tests after discovering one of the gruesome faults we all put in ur software from time to time.
Tests can be used to constrain your code; they force you to document your expectations, both internally and externally. internally, use more assert statements. Externally, write unit, functional and regression tests as "black box" tests to verify external behavior.
A "test umbrella" removes the need for memory. Everything should report back to a single source. This gives you a way to tell a new developer "run this; if it doesn't work, you have something to fix". You just have to remember to run the tests, not go through some complex sequence of commands that need separate documentation. This automation investment will reap big dividends in both time and quality.
Testing really helps when you want to refactor.
While code coverage may not be a good metric, it's definitely the case that if you don't run every line of code at least once in your tests then the possibility of failure becomes much higher. Line coverage is necessary, but not sufficient. Continuous integration helps you discover less obvious errors, like changes in process you forgot to document.
WSGI intercept is a useful test mechanism because it monkey-patches the Python web libraries.
[Interesting demos; I went for water for the speakers just before they started; my tests failed, and I never discovered whether they were supposed to or whether I had missed some crucial step while I was out].
The scotch tool is a recording proxy that will record everything that passes between the browser and the server: exceptionally useful for capturing AJAX interactions. It can also replay recorded sessions, and ensure that the responses that come back are "the same" (with minor exceptions such as ignoring date changes).
Figleaf is a code coverage recording tool. It saves it results in a pickle-based format, and can perform intelligent comparisons between tests, taking the union and intersection so you can easily discern what still needs testing.
Nose is a system that allows testing with a much lighter footprint than unittest. It will also all other doctests and unittests it can find if properly encouraged to do so, and can be used to add module-level setup and teardown.
Grig then took over and first described and then demonstrated Selenium, a testing tool written in Javascript that runs inside the browser (and probably one of the best-known web testing tools). It's one of the few tools that can adequately test AJAX functionality, and it's the default testing tool at Google). It has several components.
The core is augmented by an RC (remote control) module, and IDE (a Firefox add-on), and a Grid module that seems to relate to parallelized testing.
The biggest issue in defining Selenium tests is locating the required elements in the page. It would be much easier if web authors gave all elements IDs, but in practice you often end up using xpath, which has downsides. Many times there is no other way.
The IDE component allows you to record interactions with web servers, but again you have to be careful to use repeatable event sequences (e.g. pasting is not repeatable); the IDE is probably not as easy to use as Twill, but some AJAX applications are almost pure client-side, with very few server round-trips, and Twill is useless with these as it can't intercept the traffic.
Firebug is an essential component in developing the tests, as is the XPath Checker add-on, which helps greatly in generating the element locators.
The basis of Seleniun tests is a long string of locators and assertions. Selenium RC runs as a reverse HTTP proxy, bypassing the "same origin" cross-site scripting protections that normally insist that scripted content has to come from a single domain. By fooling the browser this way remote control allows you to write test of a remote server inside the local proxy. It also allows you to build simple looping constructs. Of interest to readers will be its ability to export test cases as (rather ugly) Python code. Google has a whole team working on Selenium, and others are also working to improve it, and extend it to broader browser support.
Grig then went on to describe buildbot, a continuous integration testing tool that allows a master system to run build (or indeed any other) commands on remote slaves and aggregate reports from the clients in a simple, readily available web format.
Although some people perform continuous integration on a single platform, buildbot was designed from the start to allow the master to drive the same tests across multiple clients and thereby discover cross-platform bugs that might otherwise go undetected.
Greg ran us through a configuration file. The learning curve can be steep, but once your buildbot is configured it usually needs little maintenance. He also highlighted the fact that someone has to keep on top of buildbots and avoid bitrot by displaying the currrently parlous state of the Python buildbot farm. Many clients are down, and those that are running appear to be failing their tests.
The evening closed with a brief discussion of Fitnesse, which is a higher-level tool that has you define tests declaratively in a language that allows you to refer to the concepts of the application domain; this makes them easier for the customer to comprehend, for example. The tests are maintained in a wiki.
I was beginning to flag a bit by now, so I can't really give you an intelligent summary better than Grig's "it's a Wiki that runs things". Ideally the client will at least understand the test descriptions.
All in all an excellent and enjoyable presentation that not only gave me some great ideas for testing but also left me more determined to introduce it into my web projects.
He gave us a number of practical guidelines, the most interesting of which were: start small; focus on actual problems; use continuous integration. The goal of testing should be to fail as quickly and as hard as possible: failure is good, because it reveals a need for revision. You should plan for testability, and as your experience level increases you will find that you naturally tend to do this.
Use a hierarchy of tests. The initial tests should run in under 20 seconds; only if those tests pass should you run the full test, which will take longer; then run regression tests, and then (if you have defined them) the acceptance tests. The point is that much of this work isn't done on your time: your continuous integration may be an overnight run, but you don't have to stay while they run. Titus feels that any test that takes over a couple of hours has limited (though definitely not zero) value, due to the lack of immediacy in the feedback it provides.
Taxonomy of testing: unit tests, functional tests, regression tests, UI tests, acceptance tests, integration tests, continuous integration, performance tests. Do you even know what all these are? Which to you need to define and use?
The more complicated it is to run the right tests, the higher the probability they won't get run. Titus talks about different strategies: TDD (Test-Driven Development) and SDT (Stupidity Driven Testing) , which he equated with TED (Test Enhanced Development). The general myth is that you are a bad person if you don't use TDD, but SDT is very important because you can ensure that bugs get properly squashed by adding specific tests after discovering one of the gruesome faults we all put in ur software from time to time.
Tests can be used to constrain your code; they force you to document your expectations, both internally and externally. internally, use more assert statements. Externally, write unit, functional and regression tests as "black box" tests to verify external behavior.
A "test umbrella" removes the need for memory. Everything should report back to a single source. This gives you a way to tell a new developer "run this; if it doesn't work, you have something to fix". You just have to remember to run the tests, not go through some complex sequence of commands that need separate documentation. This automation investment will reap big dividends in both time and quality.
Testing really helps when you want to refactor.
While code coverage may not be a good metric, it's definitely the case that if you don't run every line of code at least once in your tests then the possibility of failure becomes much higher. Line coverage is necessary, but not sufficient. Continuous integration helps you discover less obvious errors, like changes in process you forgot to document.
WSGI intercept is a useful test mechanism because it monkey-patches the Python web libraries.
[Interesting demos; I went for water for the speakers just before they started; my tests failed, and I never discovered whether they were supposed to or whether I had missed some crucial step while I was out].
The scotch tool is a recording proxy that will record everything that passes between the browser and the server: exceptionally useful for capturing AJAX interactions. It can also replay recorded sessions, and ensure that the responses that come back are "the same" (with minor exceptions such as ignoring date changes).
Figleaf is a code coverage recording tool. It saves it results in a pickle-based format, and can perform intelligent comparisons between tests, taking the union and intersection so you can easily discern what still needs testing.
Nose is a system that allows testing with a much lighter footprint than unittest. It will also all other doctests and unittests it can find if properly encouraged to do so, and can be used to add module-level setup and teardown.
Grig then took over and first described and then demonstrated Selenium, a testing tool written in Javascript that runs inside the browser (and probably one of the best-known web testing tools). It's one of the few tools that can adequately test AJAX functionality, and it's the default testing tool at Google). It has several components.
The core is augmented by an RC (remote control) module, and IDE (a Firefox add-on), and a Grid module that seems to relate to parallelized testing.
The biggest issue in defining Selenium tests is locating the required elements in the page. It would be much easier if web authors gave all elements IDs, but in practice you often end up using xpath, which has downsides. Many times there is no other way.
The IDE component allows you to record interactions with web servers, but again you have to be careful to use repeatable event sequences (e.g. pasting is not repeatable); the IDE is probably not as easy to use as Twill, but some AJAX applications are almost pure client-side, with very few server round-trips, and Twill is useless with these as it can't intercept the traffic.
Firebug is an essential component in developing the tests, as is the XPath Checker add-on, which helps greatly in generating the element locators.
The basis of Seleniun tests is a long string of locators and assertions. Selenium RC runs as a reverse HTTP proxy, bypassing the "same origin" cross-site scripting protections that normally insist that scripted content has to come from a single domain. By fooling the browser this way remote control allows you to write test of a remote server inside the local proxy. It also allows you to build simple looping constructs. Of interest to readers will be its ability to export test cases as (rather ugly) Python code. Google has a whole team working on Selenium, and others are also working to improve it, and extend it to broader browser support.
Grig then went on to describe buildbot, a continuous integration testing tool that allows a master system to run build (or indeed any other) commands on remote slaves and aggregate reports from the clients in a simple, readily available web format.
Although some people perform continuous integration on a single platform, buildbot was designed from the start to allow the master to drive the same tests across multiple clients and thereby discover cross-platform bugs that might otherwise go undetected.
Greg ran us through a configuration file. The learning curve can be steep, but once your buildbot is configured it usually needs little maintenance. He also highlighted the fact that someone has to keep on top of buildbots and avoid bitrot by displaying the currrently parlous state of the Python buildbot farm. Many clients are down, and those that are running appear to be failing their tests.
The evening closed with a brief discussion of Fitnesse, which is a higher-level tool that has you define tests declaratively in a language that allows you to refer to the concepts of the application domain; this makes them easier for the customer to comprehend, for example. The tests are maintained in a wiki.
I was beginning to flag a bit by now, so I can't really give you an intelligent summary better than Grig's "it's a Wiki that runs things". Ideally the client will at least understand the test descriptions.
All in all an excellent and enjoyable presentation that not only gave me some great ideas for testing but also left me more determined to introduce it into my web projects.
January 8, 2008
Windmill Web Testing Framework
If what this page says is true then Windmill will be very well worth some investigation. Of course the major problem with the Python web world is that so much is going on it's simply impossible to keep up with everything! Of course WSGI lays the foundation for much else, and it appears that Zope 2 now plays well with others.
I think I might start trying to make shorter blog entries just to note the most interesting developments as PyPI (Cheese Shop) contributions fly past at a rate of knots.
In the past I have looked at TurboGears, and was impressed how easy it was to implement the functionality of another web site. Recently I have been doing a similar thing with Django. We are very fortunate to have two such capable frameworks available for our web developments, and I am going to try and make sure that they both get some coverage in Python Magazine.
I think I might start trying to make shorter blog entries just to note the most interesting developments as PyPI (Cheese Shop) contributions fly past at a rate of knots.
In the past I have looked at TurboGears, and was impressed how easy it was to implement the functionality of another web site. Recently I have been doing a similar thing with Django. We are very fortunate to have two such capable frameworks available for our web developments, and I am going to try and make sure that they both get some coverage in Python Magazine.
Subscribe to:
Posts (Atom)