Mega Mobbing

This week we ran a #mobprogramming session with 35 people. Here are some notes about how that went:

READ MORE

Port And Transport And Port

I use Ports-and-Adapters to abstract away my application’s interactions with external systems. I bend the dependency’s interface to the shape that I want for my domain. This makes it easier to think about my code and to unit test it.

READ MORE

New Blog

I'm experimenting with a new blog. I hope it will be easier to right better technical posts if I can use Markdown.

Check it out: http://jay.bazuzi.com/

Fast tests for integration points

Ports-and-Adapters is a good design approach for separating business logic from external dependencies, aka Mine vs. Thine.

Like all good designs, Ports-and-Adapters makes things more testable. Everything is tested in a tight edit/build/test cycle except for the "real" adapters. The "real" adapters don't change very much, so we test them on a slower cadence.

"Don't change very much" isn't very reassuring though. I don't think about the real adapters much, but I at least want something to tell me that my real adapters aren't changing. If they need to change, grab my attention so I can run the focused integration tests.

Arlo Belshee suggested record/reply tests. Here's an example, in C# with HTTP:

Record-and-passthrough integration testing

Test->HttpClient: HTTP Request
HttpClient->Pass-through Recorder: HTTP Request
note right of HttpClient: Record the request
Pass-through Recorder->Some service: HTTP Request
Some service->Pass-through Recorder: HTTP Response
note right of HttpClient: Record the response
Pass-through Recorder->HttpClient: HTTP Response
HttpClient->Test: HTTP Request

While developing the adapter we run "focused integration tests", testing the adapter it against the real dependency. For each test we record the HTTP requests and responses.

Since these tests are slow/flaky/expensive we don't run them in the edit/build/test cycle, but only when actively working on the adapter.

Verify-and-replay isolated testing

Test->HttpClient: HTTP Request
HttpClient->Player/Verifier: HTTP Request
note right of Player/Verifier: Verify the request
note right of Player/Verifier: Return the recorded the response
Player/Verifier->HttpClient: HTTP Response
HttpClient->Test: HTTP Request

While doing development on the rest of the system, or while refactoring in the real adapter, we run the real adapter against the recorded messages. This test tells us that the adapter's behavior hasn't changed (in any way that we test for), without the speed/reliability/cost of talking to the real service.

#NoMocks

This is not a mock. How so? And why not use a mock?

A mock encodes what we know about the thing we're mocking. We write our understanding in code. If our understanding doesn't match the real service, our tests can pass but the system is will fail in production.

New requirements mean extending the mock. As the mock grows, it needs good design to keep from becoming unmaintainable. This recorder is cheap to extend: write a new test, run it, save the results.

Code

Loading Gist eb5b5fc548d253155ad113ba60898ba8....

Refactor a lot, but only when it's appropriate

Don't just refactor for fun. Refactor in service of delivering business value. If there's some terrible code that you never need to touch, then there's no reason to change it. Leave it terrible.

So, when are the right times to refactor?

• When you're changing code. Refactor to make it well-designed for its new purpose.
  
• When you're reading code. Every time you gain some understanding, refactor to record that understanding. Lots of renames.
  
• When you're afraid of code. If there's code you should be changing or reading, but you avoid because it's such a mess, then you should definitely refactor it.

Note that this refactoring is a small improvement each time, not a dramatic major rewrite. The goal is Better, not Good.

1* Agile is not nothing

Sometimes when people learn about the Agile Fluency Model they think of the first zone as "not very good". We should of course be aiming for one of the higher zones, right?

Maybe. You have to figure out which is the best zone for your context. Focus on Value is awesome in itself. As describe by Arlo Belshee:

• Plan and work in units of value, not in technical components.
• Deliver their work by unit of value, even when that cross-cuts with the technical design.
• Regularly focus on the 20% of the work with the highest value and drop the rest.
• Regularly inspect and adapt. Own their own people, practices, platform, and data, and improve them constantly.

It's also not easy. Many teams are not there and would need a lot of work to get there. Don't take it for granted.

You really should get rid of those strings.

If you're looking for something to improve in your code, the #1 thing is not in this blog post. It's Naming is a Process.

The #2 might be eliminating the Primitive Obsession of string parameters to functions. The short recipe is:

1. Pick a string parameter to some method.
2. Write a new trivial value type* that contains that string.
3. Fix the method.
4. Fix all callers.
5. Let the type propogate.

You can often figure out the name of the new type from the name of the parameter:

string attributeName;   // <-- suggests an `AttributeName` type

string previousZipCode; // <-- suggests a `ZipCode` type

Why?

- It gives you a place to hang behaviors, eventually growing into a Whole Value.

- Eliminates bugs where you pass a CustomerId where only an OrderId is allowed (assuming strong types)

This isn't a strict rule. Don't go out and do this to every string parameter you have right now. Do it a bit at a time, in code you already need to change or understand, when you're pretty sure what the new class should be.

*Value Type in the DDD sense and not in the C# sense

Prevent infinitely many bugs with this one simple trick

Here's a way to quickly find a bug in any legacy code:

Step 1: Find a case-insensitive string comparison.

boost::remove_erase_if(attributeNameAndValuePairs, [](const auto& nameAndValue) {
    return boost::iequals(nameAndValue.name, "foo");   // <-- here
});

Step 2: Trace the use of those strings to find somewhere they are compared with the default comparison. Bug!

if (attributeName == "bar") // <-- bug!

Sometimes it's less obvious:

return std::find(allowedAttributes.cbegin(), allowedAttributes.cend(), attribute.name) != allowedAttributes.cend(); <-- hard to see bug!

I've written this kind of bug many times, and so has everyone else, judging by how often I see it. Recently I wrote it again, and decided to dig a little deeper instead of just fixing this one new case.

Ways to see the problem

Is this a testing problem?

Should we have caught this bug sooner by testing more carefully?

I could write a unit test, asserting that this function handles casing correctly, for each of the dozen or so places that could get this wrong. But if I can remember to write that test, then I can remember to write the code the correct way. Both my code and my unit tests are limited by imagination. How do I catch all the cases?

Getting a second person to do the testing can help: they'll see things I don't see, and that is fantastic. But still, testing is only as good as the imagination of the tester. Also, while that may help us catch the bug before we ship, I want to catch the bug before I check in. 

Test Feedback

The real power of Test-Driven Development is not in running the tests to catch defects, but the feedback the tests give you about the design of your code (if you can hear that feedback). What are tests telling us in this case?

With the current design, to eliminate this kind of bug means finding all the places we deal with attribute names, and writing a test. If there's 1000 places, I need 1000 tests. This is a test's way of saying "your business rule is duplicated all over the place". We can look at this as a DRY problem, and can improve the design by refactoring to eliminate that duplication.

Primitive Obsession

We could also look at this as a Primitive Obsession problem: dealing with strings instead of a type that represents a concept from the business domain. Here, the concept is "Attribute Name". The Whole Value pattern says "make and use new objects that represent the meaningful quantities of your business".

Test-as-Spec

Yet another way to look at this problem is from the point of view as "test-as-spec". I want my unit tests to make sense to a person familiar with the problem domain.

The test I want to write is "Attribute names are case-insensitive". With the original design, there's no way to do that, both because the rule is written in many places in the code, and because the rule is embedded in other functions which implement some other business rule.

In order to get test-as-spec, the rule will need a single, canonical home, decoupled from the rest of the system.

Solution

Based on any of the above points of view, we can create a type that represents an attribute name:

struct AttributeName

{

    std::string Value;

}

And extract the equality check:

bool operator==(const AttributeName& lhs, const AttributeName& rhs)

{

    return boost::iequals(lhs.Value, rhs.Value);

}

And now we can write a single test, named after the business rule that it describes (test-as-spec):

void AttributeNamesAreCaseInsensitive()

{

    CPPUNIT_ASSERT(AttributeName{"foo"} == AttributeName{"FOO"});

}

Then we replace any instance of std::string attributeName with AttributeName attributeName and let the compiler tell us what to fix next. Keep chasing compiler errors until they are all gone.

Is this code perfect?

No, but it's better: all the places that compared attribute names before are now slightly simpler / easier to read / easier to write / easier to test.

It's not impossible to write a case-sensitivity bug for attribute names, it's now much harder to do the wrong thing, and much easier to do the right thing.

In the process of carrying out this refactoring I found another instance of this kind of attribute name case-sensitivity bug, which testing had not yet caught. Double win!

What do you think?

Releases per bug

Traditional teams count "# of active bugs" and "# of bugs fixed per week" and the like. These drive the wrong behaviors, rewarding create/find/fix over eliminating the underlying causes of bugs.

As the bug injection rate approaches zero, you can shift how you work and how you think about bugs. A few metrics that I like for BugsZero teams:

• # of releases since the last bug
• # of user stories completed since the last bug
• # of user stories shipped since the last bug

Slightly better every day

If you have legacy code, you should strive to make things slightly better every day. (Sometimes referred to as the scout rule).

There's a lot of nuance in here that is important to understand.

Things

Only improve what matters for the work you're doing today. 

- Reading. If you need to read some code, record your understanding of that code by renaming something or extracting a method.

- Writing. If you need to change some code, refactor it to make the change you're doing a little easier and safer.

- Not changing. If the right place for this change is in module A, but you know better than to touch A because it's so incredibly terrible, so instead you hack the change in to module B: go refactor a little bit in A so that maybe in the future you can put the next change where it belongs.

- Tools. Anything that affects your ability to get work done is potentially in scope. For example, if you find yourself waiting for the build a lot, then do something to optimize the build.

Slightly

You find the code you need to work in. It's a mess. You know how you would like it to work. You've been itching to rewrite it for a while, and then it would be pretty good and not suck.

Don't.

Making it all the way good would take too long. You need to get your work done. And there are other parts of the system that need attention, too - focusing on this could would mean leaving that other code in a bad state.

Make it only slightly better now. Trust that if you need to touch it again tomorrow, that's when you'll make it slightly better again. 

Better

Extracting a method can make code better. But the feature or bug fix you're putting in will make things worse. To actually leave the code better than you found it, you must make more improvement than you degrade it. 

Code quality is difficult to measure, but we can measure things like cyclomatic complexity and lines of code and build/test/deploy duration, and I do mean that these should improve over time, even as the system gains more capabilities and delivers more business value. 

Every day

There are exceptions. Maybe you need to get this bug fix done in a hurry, and can't see a quick, safe way to improve the code. But most of the time, things are better at the end of the day than at the beginning. And there will be days when things are on fire; but most of the time, things are better at the end of the week than at the beginning.

Three kinds of code

I propose a refactoring "Extract Open Source Project".

We build software systems to some purpose. But when I read code, I see that some of that code directly serves that purpose while other code does not. I see three categories:

Features

This is the stuff you and your customers care about. It's the reason your software system exists.

In an e-commerce system, that's code that says "when a customer uses a discount code, the discount is applied to the order."

If you learn about code smells, great names, and duplication, and then refactor with those in mind, you'll find that some code is explicitly the feature and some that is not. That leads to:

Utilities

Code that helps you write code, but has nothing to do with the problem domain you're working in.

It's often write in the middle of the rest of your code, but as you refactor to improve readability and reduce duplication, it can become visible. For example, consider this refactoring sketch:

Loading Gist ae5bd627269cb00ceaa9f9c3d3294dea...

If you refactor mercilessly, you'll end up with a lot of this stuff. It's not part of the value you offer, and it would be useful to the programming community. Factor it out to be an open source project and share with the world.

Some examples of this are Boost in C++ and Rails in Ruby.

Domain Libraries

You'll also have some code that is specific to your domain, but is not the feature you're creating. This is code that lets you describe your feature. A library for building features in this domain. Maybe a DSL.

In that e-commerce system, it might be class like "Money".

While this is not the value you are offering, it is key to offering that value. You get to decide whether to release it as open source (so other people can build more systems in the same domain), or keep it under wraps (so your competition has to build their own).

Test-as-spec and assertion syntax

I like to say that tests should, first and foremost, be a human-readable spec. Let's look at what that can mean for how we write assertions.

Suppose we're writing a card game, and we want to assert that a deck of cards is sorted the way you'd find them when you first open the box. (I'm using this simple example as a proxy for the kinds of more complex problems that we see in legacy code. It's up to you to map these ideas to that context.)

An approach I see in a lot of code is to iterate over the cards to assert. Perhaps something like:

Loading Gist aa1a17188812f640eba4d94495de892e...

This kind of code makes it obvious that an AssertEquals would be valuable, so that on failure you can see the expected and actual values in the test results.

If this test fails, you only know about one incorrect card. If there are more, you won't know until you fix the current error and rerun the test.

A richer assertion library might offer AssertSorted. It could even take a set of 1 or more sort key selectors. The result might look like:

Loading Gist aa1a17188812f640eba4d94495de892e...

(That's C++ lambda syntax, if you haven't seen it before).

Both of these approaches are "computer science" solutions - they work in the solution domain, and use the language of computer code. If I want my test to be a human readable spec, I need to use the language of the problem domain. I could take a step in that direction by extracting a method, giving:

Loading Gist aa1a17188812f640eba4d94495de892e...

But we're also doing TDD. In TDD, we want the tests to give us feedback about the design of the code. And this test is saying "the notion of being sorted that is missing from the code under test". Taking an intuitive leap, the class that should hold that notion is a "deck of cards", which is also missing from the code under test. That leads to:

Loading Gist aa1a17188812f640eba4d94495de892e...

I like the improvements to the design of the code and the way the test reads, but I am sad to lose the ability to provide a detailed report when this assertion fails. I'm not sure how I would fix that, or if it would ever actually matter.

It's interesting to me that we're back to the bool-only assertion from the first example.

Micro-ATDD

I strive to make all my tests be both microtests and acceptance tests, an idea I learned from Arlo Belshee.

When I say this to people, they are usually confused first, then doubtful when I explain. I don't think I'm ready to address the doubt, but maybe I can address the confusion today.

Microtests

Coined by GeePaw Hill (see his article for his original definition), a microtest is like a unit test, but it has all the qualities I wish all unit tests had. It's fast and focused. It answers the question "does this little piece of code do what I intend it to do?"

Because microtests talk directly to the system under test, they are written in terms of the SUT.

It's obvious that a microtest can only be used on parts of the code that are simple and decoupled and isolated. An integration test is never a microtest.

Acceptance Test

An acceptance test describes expected software behaviors from the point of view of a user or other stakeholder. It answers the question "does this system meet the requirements I expect of to meet?"

Because acceptance tests are written in conversation with that user, they are written in the language of that user. They are organized like a spec.

My ideal tests

I want tests that hold to all of the above. My ideal tests are super fast, 100% reliable, simple, isolated, written in the language of the user, and easy to read. (This requires the SUT to be decoupled, cohesive, well-named, and DRY - properties I already want.)

Every test is both an acceptance test and a microtest.

The doubt

The usual objection I hear is "while isolated unit tests tell you about each of the little pieces, you still need some kind of integration test to confirm that all the parts work when you put them together." 

Well, that's true for most programs, but it's only necessary because of how your code is organized. "parts work when you put them together" means "the desired behaviors of the program (the acceptance criteria) are emergent properties of the system". But we know how to refactor. If two parts of the system need to work together, we can put them together in the code, and then use a microtest to assert that desired behavior.

AONW2017: Amazing Distributed Teams

My new job involves teams that are distributed over a bunch of locations, mostly on the West Coast of the USA. Each team has people at multiple locations.

I went to Agile Open NorthWest 2017 with the question "How can we make distributed teams awesome?" Here's what we came up with:

There are a bunch of known good practices to help distributed teams not suck too much. Doing them won't get us to "awesome", but at least we can get up to "not sucky". So let's start by writing down these practices:

• Communicate a lot
• Don't let remote people be 2nd-class. Make everyone equally remote, even if some are in the same building.
• Chat room for all communication, even within an office
• Experienced people who don't need to learn as much are at less of a disadvantage when remote*
• Because remote pairing is more tiring, be deliberate about taking breaks. 
• use Pomodoro
• Do your homework before coming to meetings, so you don't need to rely as much on awkward VTC communication
• Show up to meetings on time
• Don't let people get blocked on questions. If someone raises a question in chat, don't leave them hanging.
• Have the whole team mob for 1 hour to start the day, to get alignment on the day's work
• Synchronize time of work
• Meet face-to-face regularly. Have a budget to bring people together.
• Many companies save money by having people work from home. Direct some of that savings to equipment, travel, etc. 
• Consider paying out-of-pocket to make remote more awesome, and then ask for reimbursement if it helps.
• Remember that patience online is short, and accommodate that fact.
• Create team agreements - they're at least as important as for teams that sit together
• Recognize the expression of Conway's Law: limited communications affect software architecture
• Telepresence robots can help. Make sure they are human size (short robots get treated like children)
• Retro often, with a relentless focus on the things that make remoting difficult
• Build the team
• Play games together remotely (poker, Halo)
• Friday beers in VTC

Yet-unsolved problems that generally make remote work suck:

• There is no good remote whiteboard
• Estimating remotely is particularly bad (plug for #NoEstimates)
• When tools/tech stop working right when we need them, we have a bad time
• It's hard to influence the org beyond the team / hard to influence culture

And then we looked at advantages to remote work / distributed teams - how they can be better than teams that sit together:

• 50 people can write on a Google Doc at once, while only a couple people can write on a whiteboard at once.
• Better ergonomics are possible. No crowding around a single screen.
• Remote breaks are real breaks. When you step away, no one can reach you. Go outside!
• You have access to a broader pool of talent.
• It increases diversity, even compared to the same people sitting together
• Enables a 24-hour development cycle
• Can accommodate people in new ways. For example, a person with a partial hearing loss can turn up their headphone volume, instead of asking everyone to remember to speak up. 
• Can accommodate varying communication styles

We measure how awesome a team is with two questions:

1. Are we delivering (steadily increasing) value?
2. Are people happy?

*I don't think this is true, but it come up in the session, so I put it here in the list.

Safely extract a method in any C++ code

Moved here for better formatting: http://jay.bazuzi.com/Safely-extract-a-method-in-any-C++-code/

Pinning Tests

I wrote this on the C2 Wiki, with the hopes that other people would help improve it. But now that site is down, so I'm posting it here:

Definition: A simple-minded automated test that locks down the behavior of existing code that otherwise is not well-tested, as a safety net while refactoring.

Example: Run some code and collect logs as a baseline. Each time you make a change, run the program again and compare the logs against the baseline. As long as there is no difference, you have some confidence that things are still working.

Pinning tests can make it safer to refactor. (Pinning tests can never make refactoring completely safe, because you'll forget important cases in your pinning tests. For safety, use #3 or #4 from Various Definitions of "Refactoring"). Pinning tests are a safety net, just in case.)

The most important features of pinning tests are:

1. Give an obvious, definitive pass or fail result.
2. Good coverage. Professional testers get really good at this; ask them to help.
3. Faster is better, so you can run them often.

Non-requirements for pinning tests:

Robustness. Professional testers get really good at making robust tests that work on different computers, or at different screen resolutions, or across UI changes. Ask them to refrain - these tests are short lived, and the behavior of the system won't be changing (by definition of "Refactoring").

You don't need to run your pinning tests in every environment that you ship. For a GUI, it's fine to record mouse clicks and keystrokes.

Long-lived. The goal is to hold behavior constant for just long enough to ReFactor.

Clean code. Hacking the test together is OK. For example,

• Use the C preprocessor to redirect troublesome API calls to write to a log instead.
• Edit your HOSTS file to hijack accessing a network resource.

Proposed Refactoring: Introduce Parameter in Lambda

Given a lambda with a captured local variable,

1. Add a new parameter to the lambda
2. Inside the lambda, replace uses of the local with uses of the new parameter
3. Where the lambda is called, pass in the local.
   

Loading Gist 9bb6b3066f60fb7aaa7ad1f39af3cd1d...

I believe this is a refactoring: I believe that this transformation has no effect on the behavior of the code. But I'm not completely certain.

This operation is not allowed if the value of the local is changed inside the lambda.

This is almost the same operation as Introduce Parameter.

Proposed refactoring: extract and execute lambda

Given a statement block, wrap it in a lambda assigned to an Action variable, and execute it immediately.

Loading Gist 21a92bc8770caf19bf24812dff938822...

I believe this is a refactoring: I believe that this transformation has no effect on the behavior of the code. But I'm not completely certain.

I think a similar recipe for expressions is equally valid, using a Func<> instead of Action.

This is almost the same operation as Extract Method.

"pure unit test" vs. "FIRSTness"

Sometimes we categorize tests into groups like "pure unit test", "focused integration test", "end-to-end-test", etc. That's a fine approach, and useful for a lot of cases.

For example, I find that pure unit tests are extremely valuable in giving me feedback about my code design, especially coupling and duplication. I don't even have to run the tests to get that value! Other types of tests have their value, but they don't give me that feedback.

Another categorization I sometimes find useful is based on the FIRST Properties of Unit Tests. You should read that link for the full story, but I'll summarize here:

Fast
Isolated (tests have a single reason to fail. One aspect of behavior = one test)
Repeatable (same result every time)
Self-verifying (tests report an unambiguous pass/fail)
Timely (each test is created just before it is needed)

It's common for programmers to have one set of tests that they run with every edit-build-test cycle on their dev machine. They might have another set they run to validate each checkin before it merges in to source control. Another that runs nightly or weekly. Another that runs before each release.

I've noticed is the decision about which tests fit in each of these buckets is less about "unit" vs. "integration" and more about "FIRS" (without the T). That is, if a test is fast and the results are reliable and useful, programmers will tend to run them more often. If a test is slow, or results require investigation, they will tend to run them less often.

Ideally, I'd like to see 99.9% of tests run in 1ms or less, be perfectly repeatable, with a clear pass/fail, and for each failure to make it obvious what aspect of what desired behavior is not right. You should strive for that. But today, given the tests you have, you may find value in bucketing your tests as I've described.

How to document your build process for an open source C# project

As an Open Source contributor...

I find an interesting open source project that I want to contribute to. I fork/clone the repository to my machine. Then I have to figure out how to build it.

Is there a solution file? Or a script?

I try something and the build fails. Do I need a certain SDK or Visual Studio feature installed? Which version?

I get it to build and then I try to run the tests. 1/3rd of them fail, because they are looking for something that isn't installed on my machine.

If I'm lucky (!?) I find a document in the repository that claims to be build instructions, but it is jumbled and clearly out of date. I try to follow it, but something I need to install is no longer available, or not compatible with my version of Windows. Will a newer version of that thing work OK?

Uggh, what a mess.

As an Open Source maintainer...

I put together a cool little project in my spare time and post it online. It's simple and straightforward to build and run tests. 

Then a contributor complains that they can't build it. What information could possibly be missing? It's simple and straightforward, right? I write a small text file explaining the obvious instructions. The contributor tries to follow it but is even more confused. I don't have time for this.

Uggh, what a mess.

A solution

My solution is AppVeyor. I treat AppVeyor as the reference build environment.

Here's how:

1. https://ci.appveyor.com/
2. New Project, select your project.
3. Settings -> Do what you need to get a green build + tests
4. Settings -> Export YAML. Add it to your repo.
5. Delete the AppVeyor project
6. New Project again, but this time configure nothing. It will use the settings from your repo.
7. Confirm that build + tests are still green

Now the instructions for how to build + run tests are in source control. Anyone can read them. There won't be any missing details. If a dependency changes, I won't miss updating the instructions, because AppVeyor will report my build is broken. 

No more mess.