Cabourotte 0.12.0 released: new features, why synthetic monitoring matters
I released last week Cabourotte 0.12.0. This release brings a couple of new features that I will detail in this article. I will also discuss why I built Cabourotte and what’s next.
Cabourotte and synthetic monitoring
I built Cabourotte to have a simple yet powerful way to define various kind of health checks on infrastructures components. Why ? Detect micro (or long) downtime.
People usually define health checks on public endpoints, using tools like Pingdom or StatusCake for example. It’s a start, but doing it on internal endpoints (internal services, databases, messages bus…) is to my experience not common.
Moreover, internal probes are usually executed from a central component (a Prometheus blackbox exporter instance for example). But today, networking is hard. Infrastructures get bigger and bigger. We’re adding layers and proxies on the network stack, especially in the container ecosystem (encapsulation protocols like VXLAN, services mesh, iptables wizardry…). We’re also working a lot with distributed systems.
We can quickly get in situations where a service is up, the local agent monitoring it (Consul or Kubelet checks for example) sees it up, the central Prometheus blackbox exporter also sees it up, but the thing is actually unreachable from some nodes or applications of the infrastructure.
Who never experienced a random timeout, a "network hiccup" ? It could happen during a network configuration change, the restart of an application, an outage… being able to detect quickly these kind of things (which can be rare, especially on low-traffic services) in order to fix them (implement graceful restarts/upgrades of applications for example) is important.
This is why I built Cabourotte.
Specifications
I had several criteria in mind before building Cabourotte:
-
I wanted a lightweight tool, that’s why I used Golang. Remember that Cabourotte could be deployed everywhere to verify services health from a lot of sources.
-
I wanted a modern tool: Everything should be configurable and available through the API (although a configuration file should also be available), the tool should expose metrics about itself and the health checks (in Prometheus format, with good labels), logs should be in json, hot reload on SIGHUP should work as expected…
-
A lot of options should be available to configure health checks
-
Exporters to push health checks results to external systems.
Cabourotte has all of that, and even more.
I also wanted to have "one-off" health checks support. By default, health checks configured in the configuration file or using the API are executed on a configured interval (every 10 seconds for example). But I also wanted to execute arbitrary checks if needed.
Let’s say you deployed Cabourotte on every host of your infrastructure, and that you want to quickly execute an health check (not already running) on all instances.
Cabourotte allows sending an health check definition to its API, and will immediately execute it. This could replace the traditional "SSH + netcat/curl" used by a lot of sysadmin, and allows people to execute the same check on a fleet of machines in seconds.
Want to know more about Cabourotte ? Check the website.
Cabourotte 0.12.0
This new release brings several new things.
Command health checks
A new health check type named command was added. This allows users to execute arbitraty commands, the check being considered failed if the status code is not zero.
Basic auth and security
I added support for basic authentication on the HTTP server. Cabourotte already supports mutual TLS which brings security but basic auth could I think be useful in addition.
I also added the ability do disable parts of the API (like the health check one) in the configuration file. I think some users may want to only configure health checks using the configuration file and not expose the API at all.
As said previously, with mutual TLS, basic auth, and by running Cabourotte in an isolated environment (non-privileged user, using a container…), exposing the API is I think not risky. Remember that we are in a world where a lot of agents exposing an API run as root (hello Kubelet). The Cabourotte API is also its killer feature.
But if you really want to, you can disable it.
More options for the HTTP health check
I never added for unknown reasons the ability to configure HTTP checks for HTTP methods other than GET. You can now use other methods if you want to (but I forgot to add DELETE support and saw it after the release (╯°□°)╯︵ ┻━┻, so expect a new release with DELETE support soon).
What’s next
I want to rewrite completely the frontend (which is terrible) exposing health checks results. I also want to add the ability to allow only some Common Names on the HTTP server if mutual TLS is enabled.
I should also really start working on a CLI (curl is nice but I have a lot of ideas to improve the tool usability), but as always, finding the time is difficult.
Add a comment
If you have a bug/issue with the commenting system, please send me an email (my email is in the "About" section).