Unit tests lie: that's why I love them

When a unit test for a method implementing some feature is green, it does not mean the feature is working. The corresponding end-to-end or integration tests reveal if it’s working or if it’s broken. To Product Owner’s point of view, end-to-end tests are all that matters. Unit tests are useless.

Unit tests are meant to lie. They rely on the often wrong assumption that the rest of the world is correctly working, but only because they are explicitly mocking it: using a fake world is a deliberate lie.

To me, that’s exactly why they are so useful.

Assert the truth, then fake it

Say you have a method somewhere like this

public SomeResults DoSomething(someInput) {
  var someResult = [Do your job with someInput];
  Log.TrackTheFactYouDidYourJob();
  return someResults;
}

DoSomething is very important to your customer: it’s a feature, the only thing that matters. That’s why you wrote a Cucumber, or Specflow, or any Gherkin compatible tool specification: you wish to verify and communicate the feature is working or not.

Feature: To be able to do something
  In order to do something
  As someone
  I want the system to do this thing

Scenario: A sample one
  Given this situation
  When I do something
  Then what I get is what I was expecting for

 

No doubt: if the test passes, you can assert you are delivering a working feature. This is what you can call Business Value.

Nevertheless, you should lie, too: suppose the rest of the universe is working (that is: all dependancies the method is using are correctly working), and assert your method is working.

In previous example, pretend Log is surely working. Then assert you are doing the right job with someInput and returning the right someResults.

In practice, do something like

public SomeResults DoSomething(someInput) {
  var someResult = [Do your job with someInput];
  FakeAlwaysWorkingLog.TrackTheFactYouDidYourJob();
  return someResults;
}

 

You can do this with Dependency Injection, or some Factory Method or any Mock Framework or just extending the class under test.

Can a lying assertion could be of any help?

Broken world

Suppose there’s some really bad bug in Log.DoSomething().
You’re lucky your Gherkin spec will find it and your end-to-end tests will fail.

The feature won’t work, because Log is broken, not because [Do your job with someInput] is not doing it’s job. And, by the way, [Do your job with someInput] is the sole responsibility for that method.

Also, suppose Log is used in 100 other features, in 100 other methods of 100 other classes.

Yep, 100 features will fail. But, fortunately, 100 end-to-end tests are failing as well and revealing the problem. And, yes: they are telling the truth.

It’s a very useful information: I know I have a broken product. It’s also a very confusing information: it tells me nothing about where the problem is. It communicates me the symptom, not the root cause.

Yet, DoSomething‘s unit test is green, because it’s using a fake Log, built to never break. And, yes: it’s clearly lying. It’s communicating a broken feature is working. How can it be useful?

(If DoSomething()‘s unit test fails, be sure: [Do your job with someInput] has some bugs.)

Tell me where, not what

Hopefully, you have a 75% unit test code coverage. I mean, hopefully, if you’re doing TDD, DoSomething() unit test is not the only unit test you wrote: you do have a unit test for Log as well. If you don’t, shame on you.

If Log is broken, its test will fail. And it will be the only failing test.

Unit tests are meant to kill dependencies

Oversimplifying, a software system can be seen as a network of cooperating modules. Since they cooperate, some of them depend on other.

Now, what if there’s a bug in B and B‘s feature is broken? A‘s feature would be broken as well. And so are all depending modules’ features.

With integration and end-to-end tests you would be able to find all the broken features.

Yet, this is not of any help in guessing where the bug is. The same system, with the same bug, would result in these unit test failures:

Now, compare the two scenarios.

• All your features using the broken Log are red
• All your unit tests are green, only the unit test for Log is red

Actually, unit tests for all modules using a broken feature are green because, by using mocks, they removed dependencies. In other words, they run in an ideal, completely fictional world. And this is the only way to isolate bugs and seek them. Unit testing means mocking. If you aren’t mocking, you aren’t unit testing.

Unit test green does not mean you can deploy

Behavioral tests tell what‘s not working, which feature is not working, and they never lie. But they are of no use in guessing where the problem could be.

Unit tests are the sole tests that tell you where exactly the bug is.

To draw this information, they lie about what’s around the method they test.

Integration tests are about business value: they are fundamental to document what can be delivered and what’s still pending or broken.
They are useful in communication with the Product Owner. They measure the progress state of the project.

Unit Tests are one of developers’ tools: they are great in quickly finding where exactly the bug is.

I think the most concise and effective analogy ever is the one Karianne Berg once sent by Twitter:

Good unit tests are like bad mothers-in-law: When you make a mistake, they immediately tell you exactly what you did wrong.

Behavioral tests are for business. Unit tests are one of developer’s tools. There’s no BDD vs TDD war. I think both are fundamental, and none of them should be neglected.

About these ads