Why configurations are important

When you start working with .NET technologies it seems natural to place application settings inside app.config/web.config files, but this approach usually fails to be useful in long term for various reasons:

  • When a software is composed by multiple services it is good to have a single configuration point
  • Config file mixes application specific settings with .NET settings (es. assembly redirect and connection string)
  • Xml is not the best human readable format
  • Frictions for automatic deploy (web.config transform, SlowCheetah, etc)

Resist the urge for complexity

As a good rule of thumb, architectures capable of launching rocket into space should be avoided, unless launching a rocket is the real need. If you only need a better configuration manager for a set of services, simplicity should be the key.

Simplicity is the ultimate sophistication. Leonardo da Vinci

Keep It Simple, Stupid. Wikipedia

Since I love designing software architectures the risk is producing unnecessary complexity, and I’m trying to change this attitude. The goal should be designing the simplest thing that could possibly work:

Solving simple problem with complex solution, #FAIL.

Solving simple problem with simple solution, #GOOD.

Solving complex problem with complex solution, #GOOD.

Solving complex problem with simple solution, #GENIUS.

Jarvis need “now”, is moving settings away from multiple config files and having a single “configuration manager”; a json file with all the configuration is good enough to start. There are also some little pain for configuration, leading to a simple Configuration Manager class to encapsulate access to json configuration file with some facilities.

Keep in mind DevOps

Releasing sofware must be an easy and well documented procedure and Operational people should be facilitated in troubleshooting wrong configruations.

One of the pain point to address with the new configuration manager is getting rid of this code.

var sysUrl = new MongoUrl(ConfigurationManager.ConnectionStrings["system"].ConnectionString);
var sysdb = new MongoClient(sysUrl).GetServer().GetDatabase(sysUrl.DatabaseName);

This code can bite during deployment. Suppose a connection string is missing; when the service starts it chrashes without any useful log or clue of what is wrong. Logging raw exception can be good for developers, but fails to be useful for operational people. If you got a MongoException or a NullReferenceException with a stack trace you have absolutely no clue of what is wrong and where you can fix the error. One of the goal of good configuration manager is creating good logs full of useful information to troubleshoot wrong settings.

Thanks to a new configuration manager, I can simply substitute standard ConfigurationManager with the new implementation, leading to code version 2.

var sysUrl = new MongoUrl(CqrsConfigurationManager.Instance.GetSetting("connectionStrings.system"));
var sysdb = new MongoClient(sysUrl).GetServer().GetDatabase(sysUrl.DatabaseName);

The only difference is in the new CqrsConfigurationManager class; if the setting is missing I got a meaningful log: Required setting ‘connectionStrings.system’ not found in configuration: http://localhost/Jarvis/Debug/config.json. This log gives two important information: What is missing and where I should add the configuration.

Fail as first as possible (and with good log)

The situation is still not optimal because it address only missing configuration but not bad ones. What happens if the configuration is present but is wrong (es. wrong mongo connection string)? The answer is: it depends.

Usually the code will throw some exception somewhere and only when the configuration is really used. The when problem is one of the worst. If your software needs the address of a special mongo instance where it should save commands that raised exception during execution and that settings is wrong, the service starts without any error log, giving you false confidence that everything is good.

It is only when the code uses that connection string that you start having error logs, probably containing: MongoDbException: Unable to connect to server fingolfin:27017: No such host is known and absolutely no clue on how to fix it.

The rule is: I want my software to detect as soon as possible wrong configuration settings, and log every information needed to do the fix!. This lead to code version 3.

MongoDatabase sysdb = null;
CqrsConfigurationManager.Instance.WithSetting("connectionStrings.system", setting =>
{
    var sysUrl = new MongoUrl(setting);
    sysdb = new MongoClient(sysUrl).GetServer().GetDatabase(sysUrl.DatabaseName);
    sysdb.GetStats();
});

The semantic is simple; WithSetting accepts an optional lambda that should consume and/or validate the setting in a single step. If no exception is thrown or lambda returns empty string, configuration is considered to be good. If the connection string is wrong I immediately got an exception: Error during usage of configuration ‘connectionStrings.system’ - Config location: http://localhost/Jarvis/Debug/config.json - error: Unable to connect to server fingolfin:27017: No such host is known.. This log contains enough details to understand what configuration is wrong and where it should be fixed.

This is especially good if the setting is complex, ex: a comma separated list of integers.

List<Int32> parsedConfig = new List<int>();
CqrsConfigurationManager.Instance.WithSetting("coefficientList", setting =>
{
    var splitted = setting.Split(',');
    Int32 tmp;
    foreach (var number in splitted)
    {
        if (!Int32.TryParse(number, out tmp))
        {
            return "The value " + number +
                    " is not a number. I'm expecting a list of integer number comma separated.";
        }
        parsedConfig.Add(tmp);
    }
    return "";
});

If a wrong string 13,24,3O9 is found in configuration file (there is an O letter instead of zero in last number), the system will log this message: Error during usage of configuration ‘coefficientList’ - Config location: http://localhost/Jarvis/Debug/config.json - error: The value 3O9 is not a number. I’m expecting a comma separated list of integer.. Whenever a setting is consumed by the application, it is useful to consider a little variant of the famous rule found in Code for the maintainer

Always code as if the person who ends up installing your software is a violent psychopath who knows where you live!

Next post will deal on: Where do you specify location of json config file used by CqrsConfigurationManager?

Jarvis Dev stack

In the previous post we introduced Jarvis, in this we’ll talk about the development stack and the architecture.

The client is an Html5 AngularJs Single Page Application, our backend is mostly c#.

Jarvis building blocks

The core domains are implemented in eventsourcing and cqrs on top of NEventstore with a custom implementation of CommonDomain.
For NEventstore v5 we needed a better implementation of Mongo Persistence Engine so I started pushing our fixes on Github.. some pull requests after I was invited to join the team (thank you Damian).
That was our first giveback to the dev community, then I patched / enhanced the majority of the core libraries we used in this project.

That’s (imho) how open source is supposed to work, less complains and more pull requests.

Jarvis core domains are hosted in few (Topshelf based) windows services talking each other with Rebus and MSMQ.

The webapi services are hosted in IIS and secured with integrated authentication.

Data flow

Frontend api

Jarvis data flow is “realtime async”: our frontend api accepts and validate user commands, translate them in domain commands and push them to our commandbus for delivery.

Domain services

The destination worker handle the command, with automatic retry and failover, invoking the corresponding aggregate methods, it’s a sort of RPC over service bus: there is not a strict one to one mapping between commands and methods.

The target aggregate emits events in response to method calls to change its state, the events are persisted in NEventStore waiting for consumers: projections and process managers.

Process managers

Every process manager place a subscription on Rebus for the events it is interested in. To be more accurate a process manager can subscribe to events and messages; a message doesn’t have domain semantic and is exchanged with external high latency services. The process manager react to events sending new commands and messages.

High latency services

We need to rely on external services for text extraction, document conversion and analisys; all the cpu intensive and high latency tasks are hosted by a service worker installed on one or more machines with a simple round robin routing for task distribution.

Projections

Our projections service works in pull mode; projections are grouped by affinity; every group has a subscpription to a polling client on the evenstore and run in its own thread. Our querymodel is denormalized on MongoDb and Elasticsearch.

Push notifications

Query model updates are pushed to subscribers with SignalR and processed by AngularJs for interface updates.

Jarvis architecture

What’s next

This architecture worked well for our first customer, we have over 100 concurrent users, databases, frontend api and worker services on the same vmware box. Cpu is mostly between 1-2%, ram consumption flat and the whole roundript usually takes few milliseconds.

Rebuilding all the 50+ projections from 350k+ commits takes almost 10 minutes (we are single thread), we need to improve in this area.

Our querymodel is about 4gb, we expect to grow (at least) by a 100x factor in the next two months.

We want to split our domains in isolated services and remove the friction for every deploy: we deploy twice a week, some weeks every day (or twice a day).

To achieve this goal Gian Maria is working on a configurations dispatcher service and getting rid of all the connection strings and application settings palced in the web/app.config.

More in the next post.

File > New > Intranet

This is the first step of this journey, we want to tell you what we have built so far.

One year ago we started an “Intranet 2.0” project for a small bank.

The project main goal was to improve the ease of access to key informations, avoid personal (and sometimes out of sync) document folders, on user’s desktop.

We found a “Yahoo Directory” of about 4.000+ relevant documents (250.000+ pages).

To simplify document retrieval, every office developed a custom taxonomy to catalog corporate documents.

In the beginning there was Chaos,
And within this Chaos was Power,
Great Power without Form

Forget Google and try to search the internet browsing the “Yahoo Directory”.. you got it!

Our first goal was: no more phone calls to “knowledge owners” to find “that document” or endless filesystem style browsing in search of relevant informations.

So our first domain was a DMS.

library
library by kiarras, on Flickr.

Then we moved on:

  • Support Ticketing with Knowledge Base,
  • Corporate address book,
  • Home Banking workflow,
  • Point Of Sale workflow,
  • Resource scheduling system,
  • Corporate bookmarks,
  • Corporate chat & file exchange,
  • Active Directory sync,
  • Personal & team calendars,
  • Organisation chart,
  • IT asset inventory,
  • Analytics.

Some domains have been implemented with CQRS+ES, others with CRUD.

Next post I’ll cover the current architecture and our dev stack.

Stay tuned!