The mytical perfect Unit Test Fixture

After years of unit testing I’m really convinced that the real complexity in writing a good Unit Test is setting up a strong test fixture. Here is an example: in Jarvis we have a test that runs fine in R# test runner, in Nunit GUI test runner, in Team City build test runner, but is red when it runs inside Visual Studio Test Runner with Nunit Test adapter.

After a brief investigation we discovered that the cause was a different handling of System.Threading.Thread.CurrentPrincipal object within the various test runners. Here is our situation

What is happening inside VS?

Here is a stupid test that confirmed our suspicions.

[TestFixture]
public class verify_principal
{
    [Test]
    public void verify_principal_is_empty()
    {
        Assert.That(System.Threading.Thread.CurrentPrincipal.Identity.Name, Is.Null.Or.Empty);
    }
}

This simple test verify that the name of the identity associated to the thread used to run the test is null or empty. This test runs green with every test runner, except with Nunit test adapter in Visual Studio.

When test is executed in Visual Studio, CurrentPrincipal identity is a valid Windows Identity and it’s equal to your current Windows user, and this makes some of our test fails when executed inside visual studio. This confirms me that something different happens inside Visual Studio and I want to fix it.

Fixture Fixture Fixture

The test that failed is this one

[Test]
public void raising_events_without_user_context_should_throw()
{
	var router = new AggregateRootEventRouter<SampleAggregate.State>();
	var ex = Assert.Throws<InvalidPrincipalException>(() =>
		{
			router.Dispatch(new SampleAggregateCreated(new SampleAggregateId(1)));
		});
}

This test verifies that: if you try to raise a Domain Event without user context, the engine should throw an exception. Clearly this test fails with Visual Studio Nunit test adapter, because CurrentPrincipal.Identity is equal to current user but it runs perfectly with other test runners.

Actually we can blame VS Test Runner for this behaviour, but the real cause of the erratic test is due to an imperfect test fixture. Let’s this for a moment on the purpose of this test; it is going to verify: how our engine behave if no user context is set. It works perfectly in all test runners except Visual Studio because they runs the test without an identity set in CurrentPrincipal, but this fact is not an assumption of nunit framework.

If you believe that the real culprit is VS Test Runner, consider what happened if, before this text executes, anohter test sets a valid Principal in Thread.CurrentPrincipal property and it does not restore it to original value when the test is finished. You will end with a failure in all test runners, but the failure happens only if the test that sets principal is executed before this one. What you have is

  1. You run all test, test is red
  2. You run the test alone, test is green (because the test that changes CurrentPrincipal is not run)
  3. You run all test again, test is red (ouch)
  4. You change the name of some test and the test returns green (maybe because the test that changes CurrentPrincipal is run after the erratic test

This is called erratic test, because it is a test that can fail due to other external conditions. Such kind of tests bring real pain when they fails, because you need to understand if the test fails because the underling assumption is wrong (you have a bug) or if it’s an external situation that makes it fail (the bug is in the test). In the long run this kind of test should be either fixed or removed from your suite of test.

The above test is better refactored in this way:

[Test]
public void raising_events_without_user_context_should_throw()
{
    var old_identity = System.Threading.Thread.CurrentPrincipal;
    System.Threading.Thread.CurrentPrincipal = new GenericPrincipal(new GenericIdentity(""), new String[] {});
	var router = new AggregateRootEventRouter<SampleAggregate.State>();
	var ex = Assert.Throws<InvalidPrincipalException>(() =>
		{
			router.Dispatch(new SampleAggregateCreated(new SampleAggregateId(1)));
		});
    System.Threading.Thread.CurrentPrincipal = old_identity;
}

This test is superior for a number of reasons

  • It explicitly clears the user context before the test.
  • The test is explicitly telling you that the engine is using System.Threading.Thread.CurrentPrincipal to check current user.
  • This test is stable and runs correctly in every test runner.
  • This test is stable even if a previous test set a valid CurrentPrincipal in current Thread for some other reason.

Lesson learned

To create stable and reproducible tests, you need to be sure that the system is the desired state in Fixture Set Up phase, and with desired state I mean every external component or piece of the system that is exercised in the text. A typical error is believeing that desired fixture was already setup for you by some other “entity” like the test runner or other tests.

Gian Maria.

Code that was never executed in production

One of the greatest risk in software development is letting untested code slip in production. The general rule is: you should always execute your code and verify that it behaves as you expected before moving to production. Clearly we should prefer automated testing over manual testing, but at least we need to be sure that every part of the code was at least executed once before promote it to production.

A real example

Suppose we have this code that represents an Id based on a simple string.

public class StringId
{
    public String Value { get; set; }

    public static implicit operator String(StringId id)
    {
        return id.Value;
    }

    public override bool Equals(object obj)
    {
       //....
    }
}

And we have also a Customer class based on this type of Id

public class CustomerId : StringId
{
    public CustomerId(String rawValue)
    {
        this.Value = rawValue;
    }
}

public class Customer
{
    public CustomerId Id { get; set; }

    public String Name { get; set; }

    public String Surname { get; set; }

    public Int32 Age { get; set; }

    public Double TotalOrderAmount { get; set; }
}

Finally we have some services that handle accessing data in Sql Database with NHibernate and we have a function that return all customers. To simplify this discussion lets consider a service that keeps data in memory.

public class CustomerService
{

    public Dictionary<CustomerId, Customer> _inMemoryRepo = new Dictionary<CustomerId, Customer>();

    public void AddCustomer(Customer customer)
    {
        _inMemoryRepo[customer.Id] = customer;      
    }

    public IEnumerable<Customer> GetAllCustomers()
    {
        return _inMemoryRepo.Values
           .OrderBy(c => c.TotalOrderAmount)
           .ThenBy(c => c.Id);
    }
}

Trivial code isn’t it? Since it is super simple probably you think that there is no need for testing. If you believe that the above code is simple and correct, you missed a bug.

Let’s try to write a simple unit test that exercise the code.

[TestMethod]
public void verify_get_all_customers()
{
    CustomerService sut = new CustomerService();
    sut.AddCustomer(new Customer() { Id = new CustomerId("c1"), TotalOrderAmount = 10.0d });
    sut.AddCustomer(new Customer() { Id = new CustomerId("c2"), TotalOrderAmount = 5.0d });
    sut.AddCustomer(new Customer() { Id = new CustomerId("c3"), TotalOrderAmount = 16.0d });

    var result = sut.GetAllCustomers();
    result.Select(c => c.Id.Value)
        .Should()
        .Have.SameSequenceAs(new [] { "c2", "c1", "c3" });
}

This test is oversimplified, but it represent a similar test that you can write if the service access database trough repository interface.

This test is green, but you have a bug!

The real problem shows up only with certain test data. This new test can highlight the bug.

[TestMethod]
public void verify_get_all_customers_with_same_amount()
{
    CustomerService sut = new CustomerService();
    sut.AddCustomer(new Customer() { Id = new CustomerId("c1"), TotalOrderAmount = 16.0d });
    sut.AddCustomer(new Customer() { Id = new CustomerId("c2"), TotalOrderAmount = 10.0d });
    sut.AddCustomer(new Customer() { Id = new CustomerId("c3"), TotalOrderAmount = 16.0d });

    var result = sut.GetAllCustomers();
    result.Select(c => c.Id.Value)
        .Should()
        .Have.SameSequenceAs(new[] { "c2", "c1", "c3" });
}

Now examine the differencies: basically it is the same test as before, but now we have two customers with the same TotalOrderAmount, and the test fails with this message.

System.ArgumentException: At least one object must implement IComparable.

And this is caused by the .ThenBy LINQ instruction. This instruction tries to order objects based on Id property, but since it is a complex type, LINQ does not know how to order based on CustomerId class. The previous test never exercised the condition specified by the .ThenBy

The solution is really trivial, just implement the IComparable interface for the StringId base class and everything is green. The real problem happens when this error is discovered in production and you need to do some quick fix.

Lesson learned

This kind of problem is caused by letting slip in production code that was never executed, nor manually, nor with Unit Testing. This kind of code constitutes a time bomb in your system that can explode in every moment.

A possible solution is trying to cover all lines of code with Unit Test code coverage, but code coverage is no perfect. In the previous example if you run only the first unit test (the one that succeeds) you got 100% code coverage for CustomerService class. This happens because the code that was really not covered is inside the implementation of the ThenBy LINQ operator.

This can be a classic example on how difficult is writing Unit Tests that are able to discover every bug in your code.

Gian Maria.

What is

Programmers should speak the language of domain experts to avoid miscommunication, delays, and errors. To avoid mental translation between domain language and code, design your software to use the language of the domain. Reflect in code how users of the software think and speak about their work. One powerful approach is to create a domain model.

The ubiquitous language is a living language. Domain experts influence design and code and the issues discovered while coding influence domain experts. When you discover discrepancies, it's an opportunity for conversation and joint discovery. Then update your design to reflect the results.

The Art of Agile Development

Is it all about terms?

Some time ago I had the pleasure to meet with some friends for a GUISA meeting.

Andrea Saltarello gave the “Ubiquitous Language Smackdown” talk in which he shared his experience. We made a very similar experience ourselves (same terms, same communication issues with our customer), but I was firmly convinced that this approach could be used only on projects and was not suitable for products where domain experts are our clients (we craft business software for SME and Public Sector).

Based on our experience every client speaks it’s own language; it can depends on the market, the geographical location (even within the same country) and company size.

Among our products there’s a Manufacturing execution system; if we talk to a mechanical parts manufacturer we can refer to “Bill of Materials”, but if we talk to a fashion manufacturer and we say “Bill of Materials” the games are over: “your product is not for us, we don’t deal with BOM”. Their bill of materials (IT: “distinta base”) is “Scheda tecnica”; I don’t know how to correctly translate it with the correct meaning in english, maybe there’s not this distinction. If we talk with a PCB assembly company, “Scheda tecnica” means “Product Sheet”.

With all these terms referring to the same piece of functionality how can we design and code our model in an Ubiquitous way?

We gave up on UL until…

No, it’s all about Meaning…

One year later I was asked to do a three days class on CQRS+ES, usually I drop these kind of requests because it’s not our business, our goal is to delivery (working) software to our customers; teaching others to do the same is community stuff.

This time I made an exception, and something happened: “Semantic… semantic… semantic…” I found myself talking all the time about domain semantics and how domain semantics can help us to shape our models.

Probably that GUISA talk was the inception, unconsciously we started to use and profit UL in a new way (thanks again Andrea).

The whole thing emerged clearly with Evenstorming.

There's a sign on the wall but she wants to be sure
Cause you know sometimes words have two meanings

Stairway to Heaven - Led Zeppelin

We can’t use Ubiquitous Language with different customers and in our code base but we can use it on a smaller scope: a single organization.
Within a single organization Ubiquitous Language is an invaluable discovery tool; Ubiquitous Language exploration has become the cornerstone of our Evenstorming process (..more on slideshare).

…and Behaviour.

As expected, when we found that the same term is used by the people in the organization with different meanings we have probably saved many hours of work; I ask the users to write down on a wide blank post-it the meaning of every single term as soon as it materialize in any command / event / readmodel post-it.

The discussion starts and leads us very quickly to domain discovery down at the rainbow’s end!

Yes, that’s the fastest way to get to the “Pot of Gold”: term => meaning => use cases => behavior.

Stay tuned for the next post, I’ll tell you how Ubiquitous Language helped us to discover behaviour and model the “Document” aggregate in our DMS.

(spoiler: it’s not a single Aggregate root)