29.3.11. Needlessly adding complexity in tests in order to hide their complexity

Some tweets from DHH on testing:

I respect the guys behind it and I'm all for experimentation, but the proliferation of rSpec and Cucumber makes me sad. (source)

RSpec offends me aesthetically with no discernible benefit for its added complexity over test/unit. (source)

Cucumber makes no sense to me unless you have clients reading the tests. Why would you build a test-specific parser for English? (source)

The important thing is of course that we get people testing, so tools shouldn't matter too much. But the added complexity still upsets me. (source)

I agree, and I'm glad that these kinds of tests have never really caught fire in the Python world. There are implementations of the RSpec and Cucumber ideas, but they don't seem to be as fully embraced. In my opinion, the dark side of testing in the Python world is the abuse of doc tests, thinking that they make both good test cases and good documentation, when in fact they're neither. There are good use cases for doc tests, but I think they've been horribly abused by certain sects within the Python world. However, even when they've been horribly abused, the people writing them seem to go after pretty broad code coverage, and they don't waste a lot of time trying to be cleverly concise (in fact, the verbosity of these large doc tests is what makes them so awful when they're also treated as documentation).

One of my main issues with RSpec and Cucumber as I've seen them in the wild is that there seems to be very few tests, and they're not terribly useful. They seem to be repeats of the classic "baby's first use case", which is "user logs in". Maybe their usage outside of the open-source world is different, but the few projects I've seen which use them have very few test cases that always has me going "that's it? you think you're tested?"

And as David points out, the complexity going on behind the scenes to make the tests just seems silly. Granted, 'Unit Tests' aren't always that easy to read, but they offer a finer example of API interactions. And if you want clarity, just add some comments. Take a cucumber-esque line like "verify that the file is encoded in UTF-8" and "now the file is encoded as latin-1" and put them as comments above the test/assert/verify statements that prove that line.

RSpec and Cucumber feel like the kinds of fiddly things that get in the way of doing real work - you can spend a lot of time writing all the back-end support to get a single test to read like an english Haiku. Or you can spend time writing a good battery of tests that actually get good coverage of the system.