Jarvis Dev stack

In the previous post we introduced Jarvis, in this we’ll talk about the development stack and the architecture.

The client is an Html5 AngularJs Single Page Application, our backend is mostly c#.

Jarvis building blocks

The core domains are implemented in eventsourcing and cqrs on top of NEventstore with a custom implementation of CommonDomain.
For NEventstore v5 we needed a better implementation of Mongo Persistence Engine so I started pushing our fixes on Github.. some pull requests after I was invited to join the team (thank you Damian).
That was our first giveback to the dev community, then I patched / enhanced the majority of the core libraries we used in this project.

That’s (imho) how open source is supposed to work, less complains and more pull requests.

Jarvis core domains are hosted in few (Topshelf based) windows services talking each other with Rebus and MSMQ.

The webapi services are hosted in IIS and secured with integrated authentication.

Data flow

Frontend api

Jarvis data flow is “realtime async”: our frontend api accepts and validate user commands, translate them in domain commands and push them to our commandbus for delivery.

Domain services

The destination worker handle the command, with automatic retry and failover, invoking the corresponding aggregate methods, it’s a sort of RPC over service bus: there is not a strict one to one mapping between commands and methods.

The target aggregate emits events in response to method calls to change its state, the events are persisted in NEventStore waiting for consumers: projections and process managers.

Process managers

Every process manager place a subscription on Rebus for the events it is interested in. To be more accurate a process manager can subscribe to events and messages; a message doesn’t have domain semantic and is exchanged with external high latency services. The process manager react to events sending new commands and messages.

High latency services

We need to rely on external services for text extraction, document conversion and analisys; all the cpu intensive and high latency tasks are hosted by a service worker installed on one or more machines with a simple round robin routing for task distribution.

Projections

Our projections service works in pull mode; projections are grouped by affinity; every group has a subscpription to a polling client on the evenstore and run in its own thread. Our querymodel is denormalized on MongoDb and Elasticsearch.

Push notifications

Query model updates are pushed to subscribers with SignalR and processed by AngularJs for interface updates.

Jarvis architecture

What’s next

This architecture worked well for our first customer, we have over 100 concurrent users, databases, frontend api and worker services on the same vmware box. Cpu is mostly between 1-2%, ram consumption flat and the whole roundript usually takes few milliseconds.

Rebuilding all the 50+ projections from 350k+ commits takes almost 10 minutes (we are single thread), we need to improve in this area.

Our querymodel is about 4gb, we expect to grow (at least) by a 100x factor in the next two months.

We want to split our domains in isolated services and remove the friction for every deploy: we deploy twice a week, some weeks every day (or twice a day).

To achieve this goal Gian Maria is working on a configurations dispatcher service and getting rid of all the connection strings and application settings palced in the web/app.config.

More in the next post.

File > New > Intranet

This is the first step of this journey, we want to tell you what we have built so far.

One year ago we started an “Intranet 2.0” project for a small bank.

The project main goal was to improve the ease of access to key informations, avoid personal (and sometimes out of sync) document folders, on user’s desktop.

We found a “Yahoo Directory” of about 4.000+ relevant documents (250.000+ pages).

To simplify document retrieval, every office developed a custom taxonomy to catalog corporate documents.

In the beginning there was Chaos,
And within this Chaos was Power,
Great Power without Form

Forget Google and try to search the internet browsing the “Yahoo Directory”.. you got it!

Our first goal was: no more phone calls to “knowledge owners” to find “that document” or endless filesystem style browsing in search of relevant informations.

So our first domain was a DMS.

library
library by kiarras, on Flickr.

Then we moved on:

  • Support Ticketing with Knowledge Base,
  • Corporate address book,
  • Home Banking workflow,
  • Point Of Sale workflow,
  • Resource scheduling system,
  • Corporate bookmarks,
  • Corporate chat & file exchange,
  • Active Directory sync,
  • Personal & team calendars,
  • Organisation chart,
  • IT asset inventory,
  • Analytics.

Some domains have been implemented with CQRS+ES, others with CRUD.

Next post I’ll cover the current architecture and our dev stack.

Stay tuned!

Intro

This blog is about a journey we want to share: we are building a new software product, learning day by day something new from our users, from our mistakes, from our friends.

Why we are doing this? (Quid pro quo)

No this blog is not a marketing effort, it’s a giveback to the dev community; a giveback we hope can improve our learning journey through your feedback and maybe avoid you a headache for the same mistakes we made.

ok.. show me the code.

What you will find on this blog?

We plan to publish all the code with you (yes, the whole product.. not just code snippets) in 6-9 months when we hope to reach a full featured 1.0 release and have 500+ active paying users.

We’ll share more about the product (domain insights), tradeoffs and our current dev stack (AngularJS, .Net, NEventstore, CQRS / ES / DDD, MongoDB, Elasticsearch) in the next weeks.

So.. No code? No party! Hold on… software development is a learning process, working code is a side effetc

That’s the point, we started this journey one year ago and we are pretty happy with this way of thinking / doing things.

We want to share what we have learned so far and what we’ll learning in the next months (and yes.. some code too).

Who we are

My name is Andrea, I run a small (4 devs) software house in Italy and I’m very active in the local dev communities.

I failed twice at blogging, this is my 3rd try and the first one in English (well beyond my comfort zone).

I asked a close friend in the dev community to join us for the next months, we have a new customer and we need to scale up to 100x in terms of data, 4x in terms of active users and a backlog with a lot of new interesting stuff.

Let me welcome on board Gian Maria.

My Name is Gian Maria, I’m an independent consultant on .NET technologies, ALM and Microsoft Team Foundation Server and I’m passionate about everything about developing.

I have a huge interest about DDD and modern software architectures and I was really amazed about the product Andrea built in really short time with its company, so when he start needing help I was really excited to jump in it. The best part of starting collaborating in this project is probably the idea of publish everything online to share [ code | opinions | comments | everything ] with DDD community.

In the end…. I’m really exited and I hope this will reflects in my blog posts, stay tuned.

Wrap-up

Me and Gian Maria we’ll keep this blog up to date with our journey, and we hope to have guest posts from our friends sharing a similar path (and some code) on other projects. Yes, we are looking at you Stefano and Alessandro.

Thank you guys

This journey wouldn’t be possible without:

Our chats about aggregates, process managers, ubiquitous language, eventstorming, CQRS and Domain Driven Design are the building blocks of this journey.

A special mention goes to Rinat Abdullin for inspiring this blog with his HappyPancake post.

Curious? Suggestions? we are waiting your feedback and we have room in our blog backlog.