Unit testing microskills

In response to my Why we Test posts, George Dinwiddie had this to say:

Good collection of "whys." I'm also looking for "hows."

— George Dinwiddie (@gdinwiddie) September 12, 2015

The connection between why and how is important, but the details are not obvious. I'll pick a few values that people hope (unit) tests might offer, and give my thoughts on how to practice testing to deliver this value. (This is certainly not a complete analysis of the subject.)

prevent regressions due to future work

Most people pick up on this one right away: as long as you can get a green bar before making changes, and another green bar when you're done, your tests catch bugs before they get checked in. Great!

Speed, readability, and granularity of tests aren't as important as good coverage. They don't even have to be unit tests - any tests will do. Reliability with a clear pass/fail result is important, so that bug-induced test failures actually get recognized.

If a piece of code is a completely obvious expression of a business requirement, you still need to write a test for it, since the tests call out the intentional behavior.

"prevent regressions" does not appear to require test-first. In fact, teams that focus on this value tend to write many of their tests afterwards. Because the code isn't written for testability, it's hard to test (duh). Either we don't bother testing it, or we bend over backwards writing horrible tests that are hard to understand, and lock down implementation details, making future refactoring harder.

a safety net during refactoring

Readability and granularity of tests aren't as important as good coverage and speed. Slow tests mean you won't run as often, which means you won't catch mistakes as quickly, which makes refactoring more expensive. That changes the cost/value/risk equation for refactoring, so you won't refactor as often.

Test speed includes any time spent analyzing results and rerunning flaky tests, so make test results obvious and rock-solid.

Many organizations are nervous about the risk of bugs from refactoring, even though they tolerate bugs from feature work. In that context, great coverage is particularly important for the refactoring safety net.

In an effort to improve coverage, teams that focus on the refactoring safety net will often test implementation details, including breaking encapsulation and injecting mocks to access those details. In the process, they lock down those details, making refactoring more difficult. That's Irony Number One.

Getting proper coverage, for both "prevent regressions" and "refactoring safety net" can be difficult. Applying the Three Rules of TDD is an effective way to get the coverage that you actually need. As long as you avoid testing implementation details, you'll necessarily have to decouple your code to make this happen. So you'll naturally end up with a code base that is at least moderately well-factored, even before you try to use the tests as a refactoring safety net. That's Irony Number Two.

make DRY problems visible

DRY problems become visible in TDD when you find yourself writing the same test repeatedly. My favorite example is file path case insensitivity in Windows. Consider:

    if (File.GetExtension() == ".cs")

There's a bug here: if the file is named ".CS" then I want the software to work the same as ".cs". I can fix it locally, by switching to a case insensitive string comparison. And I diligently write a test for it. But then tomorrow I write another file extension check in another piece code, and I write another test. I may end up with a thousand expressions of this rule, and (if diligent) a thousand corresponding unit tests.

The rule I'm trying to test here is "File extensions are case-insensitive". I want to have exactly one test that describes and enforces that rule. Which means that rule must be expressed in exactly one place. That's DRY.

The correct response to "I'm testing this idea multiple times" is "extract the duplicated behavior from all the places it's used, and merge them to one place, and test that one place."

Note that test execution time is irrelevant here; you don't ever have to run your tests to get this value! However, responding to this design feedback leads to code that is factored in a way such that tests are naturally very fast (Irony Number Three!).

Readability is important: you have to be able to read the test to understand what requirement it's describing, to be able to detect the duplication.

Granularity is important: tests must each describe exactly one requirement, or the duplication won't be visible.

DRY reduces bugs, as it eliminates the risk of updating only 999 of the 1000 places a rule is expressed. DRY (along with Great Names / Coupling /Cohesion) is far more effective at eliminating bugs in shipped software than tests that are intended to catch bugs. (Irony Number Four)

Why we test, Part 8: Because we are competent professionals

#15 in my list of reasons why we (unit) test, which I learned from James Shore:

Refactoring without tests is inherently unsafe, because of the risk of introducing bugs. As a professional, I would never take such risks. Therefore, I would only refactoring when I know I have good tests. In this way, TDD makes refactoring possible.

I may not be representing his idea with perfect fidelity; for that I apologize.

My comments:

1. There is a class of programming languages* for which there exist reliable refactoring tools. With these tools I can safely refactor even without tests.
2. The reliable tools work by following a recipe. If a human follows the same recipe carefully, they'll get the same result. That would work in strongly typed languages that lack good tooling.
3. Plenty of people who make their careers as programmers ("professionals") do sloppy work, but not those who are competent.
4. The tests have to be good. If you only write tests when it's easy, they won't give you enough protection. The only way I know to get this kind of test coverage is if you strictly follow the Three Rules of TDD.
5. When naive** TDDers aim for 100% test coverage, they go to extreme lengths in their tests, including bad mocks and test cases that don't correspond to any business value. These common problems lock down implementation, which makes refactoring far more difficult; the opposite of Jim's goal.

* It's C# and Java

** most programmers

*** mocking is fantastic for Tell, Don't Ask, and problematic without TDA.

Why we Test, part 7, The Dead Horse

I've seen a wide range of practices that the practitioner claimed was TDD. (Arlo Belshee identifies 7). Obviously, outcomes vary.

People claimed that TDD was or was not effective in some way based on those results. To make it worse, I see wide variation in the stated purpose of TDD. If we don't see the same purpose, then we aren't measuring effectiveness the same way. For example:

1. ensure correctness of new code
2. prevent regressions due to future work
3. point me directly at my mistake
4. be fast enough to run often
5. a safety net during refactoring
6. the only way to be sure my tests are comprehensive
7. to stop me from writing code I don't need
8. make code coupling obvious
9. make DRY problems visible
10. support cohesion
11. create the context for entering a Flow state
12. regular rewards as I make progress
13. confidence (possibly false!) that my program will work
14. explain to another human what my code is intended to do

(I sometimes group these into Bugs, Design Feedback, Psychological Benefits, and Specification.)

If you start by practicing TDD a certain way, and see it succeed at one of the above, you'll be tempted to argue that is the "true purpose" of TDD.

If you start with a belief about the true purpose of TDD, and select a practice that doesn't do that, you'll think TDD doesn't work. (See We Tried Baseball...)

I say all this because I hope people will shift to an "all of the above" mindset, and adjust their understanding and practice to make that happen.

Why we Test, part 6: BDD vs. TDD

I've seen BDD advocates say "BDD is just TDD done right." (e.g. here and here) They seem to be saying "It's important to write your unit tests at the appropriate level of abstraction, using language from the problem domain, phrased for a human reader. Domain experts (e.g. users, business analysts) should be able to read, and perhaps write the tests."

More recently, I've seen "TDD is just BDD done right." (e.g. here and here) These people seem to be saying "It's important to use your unit tests to drive the design of your code. BDD is missing that important action." I think they're noticing that BDD doesn't include a Red-Green-Refactor cycle.

I think they're both right. Striving to write tests for humans provides the best guidance for refactoring, carrying the Ubiquitous Language deep in to the system and improving DRY and Cohesion in the system.

Why we Test: part 5: Two reasons for regression testing

In Part 1, I wrote: "I count on tests to catch mistakes before our customers do" and "Having tests means I can refactor safely".

In both cases, I want tests to catch my mistakes, but I now realize I should consider these separately.

Regression

In the first case I'm relying on the tests to confirm that I have written my code correctly, or that future functionality changes don't break previous functionality changes. I have written bugs plenty often, and I'm looking to the tests to tell me about. I'm definitely going to keep writing new features and fixing old bugs and shipping software.

When we decide to change old functionality, we'll want to change the old tests. So they should be malleable to provide their value. They should be readable and granular, so when they fail I can decide whether to change the product or change the test.

Pinning

In the second case, my decision about whether to refactor is heavily influenced by whether I have those tests. In a legacy (i.e. tightly coupled) system without good tests, most people will just leave things as-is instead of refactoring.

If I'm just looking for a safety net while I refactor, I can use Pinning Tests. They tests don't need to be malleable, since the product behavior is not changing. If they are fast, they don't need to be granular, since I can run them really often. They do need to be very reliable. They need to cover as many cases as I can manage, but only in the sections of code I'm touching. It's OK if the tests are ugly, if I'm just going to delete them at the end of my refactoring session (when my decoupled code is now amenable to unit testing.)

(In this context, when I say "refactor", I don't mean "a highly disciplined process of changing code without changing behavior, according to a recipe", or "using a high-fidelity automated tool that will safely change code without changing behavior". You could say I mean "tiny rewrites.")

Why We Test, part 4: Specification

Warning: this post doesn't feel great to me. A bit disorganized and ill-balanced. But I needed to get it out for completeness. Feedback welcome, as always!

Llewellyn Falco (here) and Arlo Belshee (here) both talk about the way that tests can provide the value of "specification": that tests can explain to another human what the program is supposed to do.

How does this compare to the values of catching bugs, informing design, and psychological reward? There's a subtle way in which focusing your energy on specification is hugely important.

What does it mean for to be a good spec?

• Name of a test is business value oriented.
• Expresses a single example of a business rule.
• Uses terminology from the problem domain.
• Meant to be read by a human. (programmer, not customer)
• At the appropriate level of abstraction for a human reader.
• Test doesn't make any non-business-value demands

This gets you the design feedback you need. You can only meet the goal of "test as spec" if you listen to the design feedback. You can't have a bunch of setup code (including mocks); that would distract from the core message of the test. The correct terminology gets pushed in to the system under test. These short, simple, straightforward tests are only possible when code is decoupled, and when each business rule is expressed in exactly one place (DRY).

Simple, decoupled tests are inherently fast.

When they fail, they tell you clearly why the failure matters.

It also gets you the comprehensive safety net: if you're focusing your attention on writing and satisfying the spec, all your code will have purpose and will will be covered by tests.

Because the test only makes demands for for business value, you are free to refactor without unnecessarily breaking tests.

This value appears when you focus on how tests are read, but also delivers value when tests are run and written.

Why We Test, part 3: Psychology

Warning: I think this is a bit of a crappy blog post. I needed to get the ideas out there for completeness, but I haven't thought through this part thoroughly.

In "TDD as the crack cocaine of software", Jef Claes talks about the way that TDD (with really fast tests) can create the preconditions for Flow. Flow is emotionally rewarding, so that becomes its own reason to write tests this way, in addition to the desirable outcomes of catching bugs and informing design.

Other psychological (not technical) reasons to do TDD or other types of testing:

• Confidence. 

Knowing that I have tests makes me feel safe that I can make changes and ship working software. 

Note that this confidence may be false! For example, if I base that confidence on reported code coverage, even though code coverage often does not correlate with quality. (Even worse, if I ignore other quality-ensuring activities because I focus my attention on code coverage, my quality will suffer while my confidence increases.) It's very tempting to celebrate coverage numbers. Don't do it.

Interestingly, following The Three Rules of TDD will tend to result in very high coverage and good quality.

• Incrementalism

TDD with a tiny Red/Green/Refactor cycle helps you take small steps. You get the feeling of making progress all that time. You're always a minute or so away from a reverting to a green bar.

• Tracking progress/status

If I get interrupted, I can look at my most recent unit tests to remind myself what I was doing. I can write my next failing test as a note to my future self about what I wanted to do next. (While I'm away, I'll probably change my mind, but the note is still valuable.)

Also, if I commit each passing test, another programmer can read the history to see the path I took. But I'm getting off topic for this post.

Maybe you can think of more examples, or a better way to organize these ideas - let me know!

Why we Test, part 2: Design

Previously, I talked about the perspective that the reason to write tests is to catch bugs, and this is a good thing all around. Now I want to talk about code design - about using tests to help me design my code well. Some people argue this is the "true meaning" of "Test-Driven Development".

Unit tests can point me towards good design: if its difficult to write a good test, it means the code is poorly factored, especially that the thing I want to test is inappropriately coupled with something I don't care about right now. Introducing indirection at this point can open my code up to a valuable abstraction.

The "good test" that is "easy to write" will look something like:

    testSubject = new Foo(/*initial state*/);
    result = testSubject.Bar(..);
    Assert(result...);

That's Arrange-Act-Assert, with one line of each. No need to write comments to that effect; no need for blank lines. (There are a few other similar forms in the 2-4 line range.)

This kind of test is only possible if the code under test is not tightly coupled to the rest of the system. More than just "programming to interfaces" so I can inject dependencies, I look to eliminate dependencies. I don't have much setup code. I don't use mocks because I don't need to.

It's not just about "testing a class", it's also about "testing a business rule". If I can test my business rules this way, they are DRY: each piece of knowledge has exactly one canonical expression in my codebase. I minimize emergent phenomena, so my whole system is easier to reason about.

Since I have unit tests that express my business rules, I use terminology from my business domain in the tests. That terminology will flow in to my system-under-test, giving rise to ubiquitous language (within my bounded context of course.)

This kind of test will naturally be super-fast and completely reliable, which supports the "catch bugs" value described before. But I also write a lot fewer bugs, because I have well-factored, well-named, decoupled, DRY code that is easy to reason about. Writing fewer bugs is more effective than trying to find-and-fix the bugs with tests.

I can treat bugs as another kind of design feedback: I can ask what made it easy for this bug to appear, and look for a way to eliminate the whole class of bugs. I may use a unit test for this purpose, but simply writing a test for the specific bug is not enough - I want to address the whole class.

Refactoring is still really important, and (unless I have great tools in C# or Java) I must count on the tests to protect me while refactoring. But now I have the advantage that a) my code is relatively well-factored already, and b) my tests are helping me figure out good ways to refactor, so refactoring is much more fruitful.

You only get this value if you listen to the feedback your tests are giving you.

This value appears when tests are written.

Why we Test, part 1: Bugs

I've noticed a small disagreement in the Agile world around the "true purpose" of unit tests? Mostly the two camps are "to catch mistakes" and "to direct design". I want to explore these ideas a bit further. Arlo Belshee gathered a bunch of great perspectives at What Makes a Good Test Suite?, and part of what I'm doing here is reorganizing those ideas, especially Llewellyn Falco's answer.

Value #1: Bugs.

Bugs ruin software, nullifying the value we work so hard to create. Tests catch bugs (sometimes called "checking" or "regression" or "validation"), so our users and our reputations are not harmed. If another person (or my future self) works on this code later on, I count on tests to catch mistakes before our customers do.

Having tests means I can refactor safely (for some definitions of refactoring). Refactoring makes future work easier. Programmers are happier. We can say "yes" to our customers more often. When refactoring, tests are especially important in languages without great refactoring tools (basically everything except C# and Java).

Speed matters. Faster tests => I run them more often => less has changed since the last run, and what has changed in fresh in my brain => easy to understand what a failure means.

Granularity matters. When a test fails, a granular test will tell me what is broken without a lot of investigation. (Some tests can also provide good diagnostics around a failure, which helps in similar ways.)

Reliability matters. If tests are flaky or broken, you either ignore them (so they deliver 0 value) or you rerun them (which acts as a multiplier on runtime).

Coverage matters. Luckily, strictly following TDD means you won't write any untested code, so you can be confident in your coverage, which is especially important in manual refactoring. Sticking with TDD requires discipline.

When I do find a bug, the responsible thing to do is add a test for it when I fix it. Now I can be sure I'll never have that bug again.

In this mindset, mocks are a great tool, because they let me unit test my code in isolation, which makes them faster, more reliable, and easier to write. I'm likely to introduce indirection ("program to interfaces") and use dependency injection, and maybe even the Service Locator pattern.

You only get this value if the have the right tests.

The bug-catching value appears when tests are run.