Mirabelle 0.9.0: hello graphviz !
I released today the version 0.9.0 of Mirabelle, a stream processing engine I built. This release adds a new command named graphviz
which generates a .dot
file representing your streams configurations.
An example
Let’s take the commented stream example available on the Mirabelle main documentation page:
(streams
(stream {:name :multiple_branches :description "Example configuration"}
(where [:= :service "http_request_duration_seconds"]
(with :ttl 60
;; push everything into influxdb
(push-io! :influxdb)
;; index events in memory by host and service
(index [:host :service])
;; by will generate a branch for each :host value. Like that, downstream
;; computations will be per host and will not conflict between each other
(by [:host]
;; if the metric is greater than 1 for more than 60 seconds
;; Pass events downstream
(above-dt {:duration 60 :threshold 1}
;; pass the state to critical
(with :state "critical"
;; one alert only every 60 sec to avoid flooding pagerduty
(throttle {:duration 60 :count 1}
(push-io! :pagerduty)))))))))
The comments should be enough to explain what’s this stream is doing:
-
Filteting events with
:service
equal tohttp_request_duration_seconds
-
Setting the
:ttl
field to 60 if not set -
forwarding all events to InfluxDB and storing them into the stream index.
-
Then, we send an alert to Pagerduty if the duration is greater than 1 during 60 seconds.
Cool. Let’s now run the new graphviz subcommand: mirabelle graphviz /tmp/graph.dot
. This .dot file can now be converted to png using dot -Tpng graph.dot > graph.png
.
This is the content of graph.png
Another example
Lets now execute the same command on another example:
(streams
(stream {:name :main :default true :description "All events will arrive into this stream"}
;; inject events into the :alert and :influxdb streams
(reinject! :alert)
(reinject! :influxdb))
(stream {:name :alert :description "Send alerts to pagerduty"}
;; keep only critical events
(critical
;; split the actions by host/service
(by [:host :service]
;; let only one event pass every 60 sec
(throttle {:duration 60 :count 1}
;; send to pagerduty
(push-io! :pagerduty)))))
(stream {:name :influxdb :description "Send all events to InfluxDB"}
;; handle influxdb errors
(exception-stream
;; set the :source key to "mirabelle"
(with :source "mirabelle"
;; push to influxdb
(push-io! :influxdb))
;; set the :service key to "mirabelle-influxdb-error"
(with :service "mirabelle-influxdb-error"
;; inject events into the :alert stream
(reinject! :alert)))))
I have in this configuration 3 streams:
-
The first one named
:main
has:default true
so it will be used by default for events arriving on Mirabelle and not targeting a specific stream. This stream will send all events (usingreinject!
to the streams:alert
and:influxdb
. -
The second one named
:alert
will keep only events with:state = critical
, and then send alerts to Pagerduty (with aby
¨+throttle
to avoid spamming Pagerduty too much: only one event will be send to Pagerduty every 60 seconds for each :host/:service combination). -
The third one named
:influxdb
will forward all events to InfluxDB. If pushing to InfluxDB fails (for example, the service is unavailable) an event containing the error will be sent to the:alert
stream thank toexception-stream
.
Let’s generate the graphical representation of these streams:
We can see how the streams are connected between each other.
Conclusion
I’m sure the .dot file generated by mirabelle graphviz
can be improved to make it prettier, but I think this feature shows again the benefits of "compiling" the Mirabelle DSL into an intermediate datastructure (EDN) which can be then be consumed if needed (see the documentation for more information about this format).
I’m sure it should be easy for example to do the opposite: building a graphical editor for streams and then generating from the graphical representation the corresponding Mirabelle configuration !
Happy monitoring !
Add a comment
If you have a bug/issue with the commenting system, please send me an email (my email is in the "About" section).