I wrote a while ago an article (available in French only, but Google translate should be enough to understand it) about monitoring and pull vs push where I talked about these topics and about tools like Riemann and Vector.

My main point was that push-based monitoring systems are not dead and are sometimes simpler to operate than pull-based systems (Prometheus). Also, I like how push-based platforms can be used by teams autonomously, especially with tools like Kafka: everyone can subscribe to a topic containing logs or metrics and do whatever they want with the data.

Riemann was a revelation for me when I first used it (in 2016 I think ?). The tool is powerful, super flexible, with tons of good ideas. I contributed to it a lot at some point, but found a couple of things which were hard to solve without a rewrite.

I thought a lot about it (and discussed it in the previously linked article), and finally (after multiple experimentations in the last two years) started my own project inspired by Riemann (but still different, I will explain where later) 9 months ago.

Mirabelle is inspired by Riemann. Some code was also reused, like the TCP server for example. Mirabelle work with the same event format than Riemann (it means tools like Collectd, Telegraf, syslog-ng, languages clients…​ for Riemann also work with Mirabelle).

I like simple systems. I think the world needs tools which are easy to run, operate, and with decent performances.
I don’t want to be forced to deploy Kafka, Hadoop, Zookeeper, or something else. I think most users do not need all of that and want simple tools.

Mirabelle is not a distributed application. Is it an issue ? Maybe for some people, but a lot of popular tools are in this situation today.

You can still have high availability with Mirabelle by using external tools, which is I think the way to go (a simple tool which can be combined with others if needed):

In the future, I would really like to write my own proxy with consistent hashing which could be deployed in front of Mirabelle, a bit like the proxy in Veneur. It should not be too difficult to build and could really help HA setups.

I’m also thinking about stateful computation and how to maintain internal states between crashes. Today, Mirabelle is able to store events on a disk queue and replay it when it starts to rebuild its state (see documentation about it).
But I’m also thinking about dedicated actions, maybe backed by a store like rocksdb, with a "checkpoint" system to rebuild the state…​ To be honest, I think a lot of users don’t need that. But I may work on something soon.

Let’s go back to our event from the beginning of the article. It represents a counter which is always increasing. Let’s use Mirabelle to compute the rate of requests for our applications:

I wrote in this example two streams:

We now want to test it. The tap action is here for that. It does nothing by default, but will register events in test mode. This is my test:

Tests are also defined as EDN. Each test contains a list of events in :input, and in :tap-results the expected content of taps. All tests run in isolation, side effects (I/O) are automatically discarded.

We inject here two types of events, one for :path "/foo" and one for :path "/bar" (to check that our by action works as expected).

It passes ! Our HTTP request rate is computed as expected (for example, for "/foo" we have :metric 110 at :time 1, :metric 210 at :time 11 so we expected a rate of 10 during this period.

Riemann tests were sometimes difficult to write, and I think one of Mirabelle strengh is its test system. Being able to easily test transformations, alerting rules…​ is important.

Let’s go back to publish! action. Mirabelle (like Riemann) has a Pub-Sub system, with a couple of new features.

You can subscribe to events (using websockets) in Mirabelle. It can be used to quickly get information about some kind of events, build real-time dashboards (here is a Riemann plugin for Grafana which also work with Mirabelle).

Once Mirabelle starts, let’s subscribe to the :http_request_rate channel (the value passed to :publish!. A client is available in the Mirabelle repository.

Now, let’s send some events to Mirabelle

I get in my websocket client shell:

It works ! You can even pass a query to the websocket client to only get some events (queries have the same syntax that where clause):

This subscription will only keep events with :metric greater than 10 for :service "http_request_rate". You could for example publish all events to some channel (it should not impact performances), like that you know that you can easily extract some events quickly using the websocket client.

I haven’t talked about it in this article but Mirabelle also has a component named the Index, which is instantiated per stream. You can put events into this index, subscribe to it, and detect events expiration (this is where the :ttl field is used). See the documentation for more information.

Mirabelle ships with 65 built-in actions (and I constantly add new ones) to work on events (filtering, aggregating events, various kinds of windows, math operations…​). It supports Pagerduty, ElasticSearch, and InfluxDB as I/O (I would like to add TimescaleDB, Graphite…​ but writing integrations is a lot of work).

I only showed you simple examples in this article, but monitoring distributed systems and today’s infrastructure is possible with Mirabelle. I plan to write an article where I monitor an infrastructure.

I also didn’t demo the HTTP API. As explained before, you can use it to manage streams dynamically. You can also push events to Mirabelle through HTTP, which can simplify integrations with some systems.

If you are interested in Mirabelle, don’t hesitate to test it and give me feedback. If you are a vendor and want to integrate your solution with it, you can also contact me.
I spent a lot of time writing the documentation, which should contain all you need to get started (but if you are blocked or have questions, you can open an issue on Github).

I hope you enjoyed the article.