Unit testing microskills

In response to my Why we Test posts, George Dinwiddie had this to say:

Good collection of "whys." I'm also looking for "hows."

— George Dinwiddie (@gdinwiddie) September 12, 2015

The connection between why and how is important, but the details are not obvious. I'll pick a few values that people hope (unit) tests might offer, and give my thoughts on how to practice testing to deliver this value. (This is certainly not a complete analysis of the subject.)

prevent regressions due to future work

Most people pick up on this one right away: as long as you can get a green bar before making changes, and another green bar when you're done, your tests catch bugs before they get checked in. Great!

Speed, readability, and granularity of tests aren't as important as good coverage. They don't even have to be unit tests - any tests will do. Reliability with a clear pass/fail result is important, so that bug-induced test failures actually get recognized.

If a piece of code is a completely obvious expression of a business requirement, you still need to write a test for it, since the tests call out the intentional behavior.

"prevent regressions" does not appear to require test-first. In fact, teams that focus on this value tend to write many of their tests afterwards. Because the code isn't written for testability, it's hard to test (duh). Either we don't bother testing it, or we bend over backwards writing horrible tests that are hard to understand, and lock down implementation details, making future refactoring harder.

a safety net during refactoring

Readability and granularity of tests aren't as important as good coverage and speed. Slow tests mean you won't run as often, which means you won't catch mistakes as quickly, which makes refactoring more expensive. That changes the cost/value/risk equation for refactoring, so you won't refactor as often.

Test speed includes any time spent analyzing results and rerunning flaky tests, so make test results obvious and rock-solid.

Many organizations are nervous about the risk of bugs from refactoring, even though they tolerate bugs from feature work. In that context, great coverage is particularly important for the refactoring safety net.

In an effort to improve coverage, teams that focus on the refactoring safety net will often test implementation details, including breaking encapsulation and injecting mocks to access those details. In the process, they lock down those details, making refactoring more difficult. That's Irony Number One.

Getting proper coverage, for both "prevent regressions" and "refactoring safety net" can be difficult. Applying the Three Rules of TDD is an effective way to get the coverage that you actually need. As long as you avoid testing implementation details, you'll necessarily have to decouple your code to make this happen. So you'll naturally end up with a code base that is at least moderately well-factored, even before you try to use the tests as a refactoring safety net. That's Irony Number Two.

make DRY problems visible

DRY problems become visible in TDD when you find yourself writing the same test repeatedly. My favorite example is file path case insensitivity in Windows. Consider:

    if (File.GetExtension() == ".cs")

There's a bug here: if the file is named ".CS" then I want the software to work the same as ".cs". I can fix it locally, by switching to a case insensitive string comparison. And I diligently write a test for it. But then tomorrow I write another file extension check in another piece code, and I write another test. I may end up with a thousand expressions of this rule, and (if diligent) a thousand corresponding unit tests.

The rule I'm trying to test here is "File extensions are case-insensitive". I want to have exactly one test that describes and enforces that rule. Which means that rule must be expressed in exactly one place. That's DRY.

The correct response to "I'm testing this idea multiple times" is "extract the duplicated behavior from all the places it's used, and merge them to one place, and test that one place."

Note that test execution time is irrelevant here; you don't ever have to run your tests to get this value! However, responding to this design feedback leads to code that is factored in a way such that tests are naturally very fast (Irony Number Three!).

Readability is important: you have to be able to read the test to understand what requirement it's describing, to be able to detect the duplication.

Granularity is important: tests must each describe exactly one requirement, or the duplication won't be visible.

DRY reduces bugs, as it eliminates the risk of updating only 999 of the 1000 places a rule is expressed. DRY (along with Great Names / Coupling /Cohesion) is far more effective at eliminating bugs in shipped software than tests that are intended to catch bugs. (Irony Number Four)

Why we Test: part 5: Two reasons for regression testing

In Part 1, I wrote: "I count on tests to catch mistakes before our customers do" and "Having tests means I can refactor safely".

In both cases, I want tests to catch my mistakes, but I now realize I should consider these separately.

Regression

In the first case I'm relying on the tests to confirm that I have written my code correctly, or that future functionality changes don't break previous functionality changes. I have written bugs plenty often, and I'm looking to the tests to tell me about. I'm definitely going to keep writing new features and fixing old bugs and shipping software.

When we decide to change old functionality, we'll want to change the old tests. So they should be malleable to provide their value. They should be readable and granular, so when they fail I can decide whether to change the product or change the test.

Pinning

In the second case, my decision about whether to refactor is heavily influenced by whether I have those tests. In a legacy (i.e. tightly coupled) system without good tests, most people will just leave things as-is instead of refactoring.

If I'm just looking for a safety net while I refactor, I can use Pinning Tests. They tests don't need to be malleable, since the product behavior is not changing. If they are fast, they don't need to be granular, since I can run them really often. They do need to be very reliable. They need to cover as many cases as I can manage, but only in the sections of code I'm touching. It's OK if the tests are ugly, if I'm just going to delete them at the end of my refactoring session (when my decoupled code is now amenable to unit testing.)

(In this context, when I say "refactor", I don't mean "a highly disciplined process of changing code without changing behavior, according to a recipe", or "using a high-fidelity automated tool that will safely change code without changing behavior". You could say I mean "tiny rewrites.")

Why We Test, part 4: Specification

Warning: this post doesn't feel great to me. A bit disorganized and ill-balanced. But I needed to get it out for completeness. Feedback welcome, as always!

Llewellyn Falco (here) and Arlo Belshee (here) both talk about the way that tests can provide the value of "specification": that tests can explain to another human what the program is supposed to do.

How does this compare to the values of catching bugs, informing design, and psychological reward? There's a subtle way in which focusing your energy on specification is hugely important.

What does it mean for to be a good spec?

• Name of a test is business value oriented.
• Expresses a single example of a business rule.
• Uses terminology from the problem domain.
• Meant to be read by a human. (programmer, not customer)
• At the appropriate level of abstraction for a human reader.
• Test doesn't make any non-business-value demands

This gets you the design feedback you need. You can only meet the goal of "test as spec" if you listen to the design feedback. You can't have a bunch of setup code (including mocks); that would distract from the core message of the test. The correct terminology gets pushed in to the system under test. These short, simple, straightforward tests are only possible when code is decoupled, and when each business rule is expressed in exactly one place (DRY).

Simple, decoupled tests are inherently fast.

When they fail, they tell you clearly why the failure matters.

It also gets you the comprehensive safety net: if you're focusing your attention on writing and satisfying the spec, all your code will have purpose and will will be covered by tests.

Because the test only makes demands for for business value, you are free to refactor without unnecessarily breaking tests.

This value appears when you focus on how tests are read, but also delivers value when tests are run and written.

Why We Test, part 3: Psychology

Warning: I think this is a bit of a crappy blog post. I needed to get the ideas out there for completeness, but I haven't thought through this part thoroughly.

In "TDD as the crack cocaine of software", Jef Claes talks about the way that TDD (with really fast tests) can create the preconditions for Flow. Flow is emotionally rewarding, so that becomes its own reason to write tests this way, in addition to the desirable outcomes of catching bugs and informing design.

Other psychological (not technical) reasons to do TDD or other types of testing:

• Confidence. 

Knowing that I have tests makes me feel safe that I can make changes and ship working software. 

Note that this confidence may be false! For example, if I base that confidence on reported code coverage, even though code coverage often does not correlate with quality. (Even worse, if I ignore other quality-ensuring activities because I focus my attention on code coverage, my quality will suffer while my confidence increases.) It's very tempting to celebrate coverage numbers. Don't do it.

Interestingly, following The Three Rules of TDD will tend to result in very high coverage and good quality.

• Incrementalism

TDD with a tiny Red/Green/Refactor cycle helps you take small steps. You get the feeling of making progress all that time. You're always a minute or so away from a reverting to a green bar.

• Tracking progress/status

If I get interrupted, I can look at my most recent unit tests to remind myself what I was doing. I can write my next failing test as a note to my future self about what I wanted to do next. (While I'm away, I'll probably change my mind, but the note is still valuable.)

Also, if I commit each passing test, another programmer can read the history to see the path I took. But I'm getting off topic for this post.

Maybe you can think of more examples, or a better way to organize these ideas - let me know!

Why we Test, part 2: Design

Previously, I talked about the perspective that the reason to write tests is to catch bugs, and this is a good thing all around. Now I want to talk about code design - about using tests to help me design my code well. Some people argue this is the "true meaning" of "Test-Driven Development".

Unit tests can point me towards good design: if its difficult to write a good test, it means the code is poorly factored, especially that the thing I want to test is inappropriately coupled with something I don't care about right now. Introducing indirection at this point can open my code up to a valuable abstraction.

The "good test" that is "easy to write" will look something like:

    testSubject = new Foo(/*initial state*/);
    result = testSubject.Bar(..);
    Assert(result...);

That's Arrange-Act-Assert, with one line of each. No need to write comments to that effect; no need for blank lines. (There are a few other similar forms in the 2-4 line range.)

This kind of test is only possible if the code under test is not tightly coupled to the rest of the system. More than just "programming to interfaces" so I can inject dependencies, I look to eliminate dependencies. I don't have much setup code. I don't use mocks because I don't need to.

It's not just about "testing a class", it's also about "testing a business rule". If I can test my business rules this way, they are DRY: each piece of knowledge has exactly one canonical expression in my codebase. I minimize emergent phenomena, so my whole system is easier to reason about.

Since I have unit tests that express my business rules, I use terminology from my business domain in the tests. That terminology will flow in to my system-under-test, giving rise to ubiquitous language (within my bounded context of course.)

This kind of test will naturally be super-fast and completely reliable, which supports the "catch bugs" value described before. But I also write a lot fewer bugs, because I have well-factored, well-named, decoupled, DRY code that is easy to reason about. Writing fewer bugs is more effective than trying to find-and-fix the bugs with tests.

I can treat bugs as another kind of design feedback: I can ask what made it easy for this bug to appear, and look for a way to eliminate the whole class of bugs. I may use a unit test for this purpose, but simply writing a test for the specific bug is not enough - I want to address the whole class.

Refactoring is still really important, and (unless I have great tools in C# or Java) I must count on the tests to protect me while refactoring. But now I have the advantage that a) my code is relatively well-factored already, and b) my tests are helping me figure out good ways to refactor, so refactoring is much more fruitful.

You only get this value if you listen to the feedback your tests are giving you.

This value appears when tests are written.

Why we Test, part 1: Bugs

I've noticed a small disagreement in the Agile world around the "true purpose" of unit tests? Mostly the two camps are "to catch mistakes" and "to direct design". I want to explore these ideas a bit further. Arlo Belshee gathered a bunch of great perspectives at What Makes a Good Test Suite?, and part of what I'm doing here is reorganizing those ideas, especially Llewellyn Falco's answer.

Value #1: Bugs.

Bugs ruin software, nullifying the value we work so hard to create. Tests catch bugs (sometimes called "checking" or "regression" or "validation"), so our users and our reputations are not harmed. If another person (or my future self) works on this code later on, I count on tests to catch mistakes before our customers do.

Having tests means I can refactor safely (for some definitions of refactoring). Refactoring makes future work easier. Programmers are happier. We can say "yes" to our customers more often. When refactoring, tests are especially important in languages without great refactoring tools (basically everything except C# and Java).

Speed matters. Faster tests => I run them more often => less has changed since the last run, and what has changed in fresh in my brain => easy to understand what a failure means.

Granularity matters. When a test fails, a granular test will tell me what is broken without a lot of investigation. (Some tests can also provide good diagnostics around a failure, which helps in similar ways.)

Reliability matters. If tests are flaky or broken, you either ignore them (so they deliver 0 value) or you rerun them (which acts as a multiplier on runtime).

Coverage matters. Luckily, strictly following TDD means you won't write any untested code, so you can be confident in your coverage, which is especially important in manual refactoring. Sticking with TDD requires discipline.

When I do find a bug, the responsible thing to do is add a test for it when I fix it. Now I can be sure I'll never have that bug again.

In this mindset, mocks are a great tool, because they let me unit test my code in isolation, which makes them faster, more reliable, and easier to write. I'm likely to introduce indirection ("program to interfaces") and use dependency injection, and maybe even the Service Locator pattern.

You only get this value if the have the right tests.

The bug-catching value appears when tests are run.

Logging an Item

MSBuild items seem like the should be almost the same as properties, but they are different in some odd ways. Time to add support for them.

Most of the time was just spent iterating with MSBuild, trying to understand what it expected, and how it would respond to different inputs. I'm glad I'm encapsulating that here in this class, so the rest of my Task code doesn't have to worry about it.

Loading Gist 7523672

EDIT: After posting this, I noticed a bug. I had hard-coded the name on line 7 of LogProperty. Some TDD gurus insist that you write more than one test before removing hard-coded results. If I had done that, I would have caught that bug before committing my code. I'm tired, so I'm looking for shortcuts; if I had a pair, I bet we wouldn't have let that happen.

Refactoring the test

All the TDD gurus admonish us to keep our test code well-factored.* I know I'll want to write more tests soon, so now is a good time to clean up the test I have. Let's refresh ourselves with starting code:

Loading Gist 7519763

This is a job for Extract Method. Most of the effort goes in to picking the right places to slice the method in to bits.

It's easier to know where to slice when you have a second (or 3rd!) example - it's easier to see the duplication. On the other hand, refactoring 3 is more work than refactoring 1. I took a quick stab at writing a new test, saw the duplication, deleted it, and then refactored. Call this an educated guess.

Here's what I ended up with:

Loading Gist 7522819

I'm curious about this new assert. Some TDD gurus mention custom asserts as one element of advanced TDD, so I am going to pay attention here and see if anything interesting happens.

Next up, a new feature, with a new test. Let's see if my refactoring works out.

*Arlo Belshee goes a step further: don't worry about duplication between tests as much as making the rests read cleanly. They should say exactly what you want to test, no more, no less. He says tests should be "WET" - Write Explicit Tests, while product code should be "DRY" - Don't Repeat Yourself.

Refactoring the logging

Now that the tests are working OK, I want to turn my attention back to the other goal: cleaner code for task parameter logging. Here's the current implementation:

Loading Gist 7517928

I want to refactor this to generalize. I'm not going to add additional tests just yet, even though maybe I should. At least I have enough test coverage that I don't have to spend time manually testing as I go.

I used very small steps, but the outline is:

1. Extract Method LogInputs, push it up to base class LoggingTask
   
2. Replace the hard-coded "Foo" and this.Foo with reflection via this.GetType().GetProperty("Foo").Name and .GetValue(this).
   
3. Extract method LogInput, taking a PropertyInfo
   
4. Replace GetProperty() with a loop over GetProperties(), filtering by the [Required] attribute.
   
5. Extract Method GetPropertyInfos and HasAttribute.

Here's the result:

Loading Gist 7521986

As an indication that I'm on the right track, see how small the test task is:

Loading Gist 7521995

You will also need this extension:

Loading Gist 7522018

I don't really like that he [Required] attribute is used to find parameters; that's not quite the right rule. But I'll come back to that later.

Creating the MSBuild project in-memory

Now that we're no longer running the MSBuild command line, the next isolating refactoring is to create the project file in memory, too. To do that, I used ProjectRootElement:

Loading Gist 7519460

Note this depends on the following extension method, which I introduced to reduce duplication:

Loading Gist 7519477

I have an urge to write an extension method to replace AddTask which will ensure that the project has a UsingTask already. In fact, I'll go ahead and do that, so you can see the transformation.

Loading Gist 7519648

I also made AddUsingTask generic:

Loading Gist 7519677

Which allows the following simpler test:

Loading Gist 7519763

Running MSBuild in-proc in a unit test

In the previous step, I executed my code under test (a custom MSBuild task) by running MSBuild as a separate process, with a hard-coded path to the EXE, passing a project file on the command line, and capturing standard output. An ideal unit test runs the system-under-test (SUT) in isolation, and this is far from isolated. That's fine in spike mode, but over time I want to refactor my tests to improve isolation.

To take a step in that direction, I'm am switching from Process.Start to using the MSBuild APIs directly. It took a while to figure out a way that works, but now I've got one:

Loading Gist 7519022

This requires a custom logger to gather the output, which I wrote like this:

Loading Gist 7519004

This new logger  is pretty specialized - high importance messages only. I have an urge to generalize it to let the importance be a parameter, but for now, I'm applying YAGNI and leaving it as simple as I can.

Running MSBuild in a unit test

My aim now is to write a simple, fast, reliable test that will execute my custom MSBuild task. Currently, I can run it with the command line:

PS> msbuild .\My.proj /v:d

One challenge with testing it this way is that the output has a lot of non-repeatable test, including timestamps. Just to make things easier, I'll elevate the logging importance to "high".

Log.LogMessage(MessageImportance.High, "  Foo = {0}", this.Foo);

Then I'll run with minimal verbosity, and nologo to clean up the output:

PS> msbuild .\My.proj /v:m /nologo
  Inputs:
    Foo = hi

Now I need to actually write a test. The simplest test I can think of:

Loading Gist 7518446

Note that I wrote the expected output as empty, to see the exact actual output (which is already known good, since I'm writing the test after the fact). I'll grab that actual output, make it the expected value, and the test will pass. That way of working is the basis of Llewellyn Falco's ApprovalTests, which I may switch to in a future step. But for now, make the test pass & check in:

            const string expected = @"  Inputs:
    Foo = hi
";

spiking a custom MSBuild Task

My team has a bunch of MSBuild tasks. Most of them write their input parameters to the log, like this:

Loading Gist 7517928

As of recent versions of MSBuild, this logging already happens automatically, but only in diagnostic logs. Those logs are so big as to be unusable for us, so we want this information to appear in the detailed (or perhaps normal) logs.

Today, I want to explore two ideas:

1. Using TDD on MSBuild tasks, and
2. Removing duplication of this logging code.

I'll start with a spike, to confirm that I do indeed know how to write & run a custom MSBuild task. The task implementation is presented above. Next, I write a small MSBuild project:

Loading Gist 7518019

And then run it on the command line (the 2nd run is with diagnostic verbosity, so you can see what the built-in logging looks like)

Loading Gist 7518058

Great, I know how to do something. :-)

Next up, turn this in to a test.