Earlier this year, the OpenTelemetry team announced the deprecation of span events. I'll explain in this article why I think it's a bad idea.

I've always been fond of Observability, whether in the companies I've worked for, where I always raised the bar on Observability best practices, in my free time by building Observability-related projects or contributing to existing ones, or on this blog by writing about it.

Over the years, I've followed with interest all the discussions around wide events. More specifically, I dove deep into tracing and implemented it successfully at scale. I've also had the opportunity to train lots of people on those topics, and of course, at some point, the discussion always turns to span events and how they compare to logs.

Span events: an essential feature of tracing

Today, my production services emit almost no logs: I've replaced almost all of them with span events, which are better logs than logs. Actually, I don't even look at logs anymore: almost all issues can be debugged using traces alone.

Why jump between tools when I can have, on a single span:

  • Duration information (that's why we generate spans in the first place)
  • Rich attributes (mostly product-related in my case) for context
  • The span status (Ok, Error) for quick filtering
  • All information about the original request (the whole trace the span belongs to)
  • Span events for arbitrary information (what we used to use logs for before) and, of course, errors (including stack traces)

Now, when debugging an issue, I just have to look at the right trace (either from the ID directly or from any business-related ID that could come from the support team, for example: user ID, entity ID... thanks to attributes), find the span in error with its associated error message and stack trace, and that's it. Our log platform isn't even involved.

I also derive my SLIs (and compare them to my SLOs) from spans (for example: top-level HTTP spans for a given service). Numbers not good? I already have dashboards ready, listing, for each SLO, the spans in error, where I can immediately understand why the request failed. No metric or log platform involved, only traces and spans.

To me the future was clear: we finally had a way to represent an event (the span) from which we can derive logs and metrics, while still being able to see all the information related to a span (including span events: specific timestamps and messages that belong to it) in a single view. One debugging unit to rule them all.

Until the OpenTelemetry team decided to deprecate them.

The issue with the deprecation

The GitHub issue about the deprecation has already brought up some interesting discussion points.

First, lots of companies don't store traces (and so spans) and logs in the same system, or don't use the same vendor for both, or don't even use the OpenTelemetry log transport. That's my case, for example: traces are stored in ClickHouse, whereas logs are stored either in Elasticsearch or Loki. For lots of people, this change will mean losing the ability to display events alongside their span in a single view.

Second, sampling will become a mess (look at the GitHub issue), since spans and span events will somehow have to keep the same sampling rules. The issue thread offers some theoretical solutions, but I'm not convinced.

Third, to me, it's a step backward: in lots of systems, a trace-first approach, where other signals are derived from spans, and so can be stored, queried, and visualized from a single data source that can then apply transformations and aggregations on demand, works perfectly.

Granted, in rare cases logs can be useful, especially when we want to make sure to get the log before the span ends (some cron jobs, for example), but that's not a good reason to dynamite all the other use cases (which are, based on my experience, the majority of applications) where having events on spans is exactly what everyone wants.

Separating spans and span events will ultimately create friction during debugging, and add burden to SRE teams, who will have to maintain scattered pipelines and data stores while still trying to always display the data belonging to a span together, for a gain I don't even understand.