At my office we do a lot of functional testing – interacting with our software as an end user would. This, in itself, is enough of a problem: much of what we do would be better tested with unit tests or automated functional tests (see Selenium for example). But it’s hard to retrofit comprehensive unit tests into a large, established code base (that wasn’t created with unit testing in mind) and it takes time to convert from written functional tests to automated functional tests. With all the test writing, editing and performing I’ve done, I have a few notes on what you should (and should not) do with your written functional tests.
Please, for your sanity, use a text based file format for your tests. If you don’t, version control (CVS, Subversion, etc.) won’t work for your tests. It’s a good thing to be able to see what changed between revisions of a test and you simply can’t do that if your version control system can’t do a meaningful diff of two versions of a file. Even more importantly, if you use a text based file format, your version control software can merge differences when you update your working copy. With a binary format, only one person can edit a file at once. This means no Word tests!
Use a source format that uses cross referencing and handles (nested) lists automatically. I like LaTeX, but as long as the software takes care of lists and cross referencing for you, you’ll be happy. I’ve spent a lot of time renumbering tests and updating references; it just isn’t a sensible use of anyone’s time.
Make your tests compartmentalized. It’s OK to have a set of setup steps for a set of tests, but if a test changes anything so that it’s different from that base line it should change things back when it’s done. It might take a little longer to run a test that cleans up properly but there is no reason that you should have to track down what the test you ran half an hour ago did to sabotage the test you’re running now.
Ultimately, don’t depend too much on human executed functional testing. It’s useful for end-user facing interfaces but it is too prone to human error to use on a very large scale. Automated tests and unit tests – those where the instructions can’t be misread or steps can’t be forgotten – are easier to maintain, faster and, because of their speed, can be run more often (which will shorten your development cycle and catch bugs sooner rather than later).