In the ever-evolving world of technology, the QCon London 2026 conference offered a glimpse into the challenges and opportunities of self-hosted observability. Colin Douch, a site reliability engineer at DuckDuckGo, took the stage to delve into this complex yet fascinating topic.
Douch's presentation was a thought-provoking exploration of the 'complexity demon' that haunts modern observability. He highlighted a paradox: while observability tools are meant to simplify debugging, the stack itself can become a complex beast. This paradox raises an intriguing question: is self-hosted observability a necessary evil, or a strategic choice with hidden benefits?
The Self-Hosted Observability Dilemma
Douch's warning was clear: self-hosted observability is not for the faint-hearted. It demands significant resources, including extra engineering talent and financial investment. Yet, he argued, it's a path worth considering for those who have exhausted other options.
The typical self-hosted observability stack includes metrics, logs, and traces, each with its own set of tools and challenges. Douch's preferred choice for metrics is Prometheus, despite its horizontal scaling challenges, or VictoriaMetrics. For logs, he suggested VictoriaLogs or Loki, depending on the project's needs and design philosophy.
The Modular Ecosystem and Its Trade-offs
In practice, self-hosted observability often involves a modular ecosystem of loosely coupled systems. This flexibility comes at a cost: increased operational overhead. Douch reviewed the open-source tooling landscape, endorsing OpenTelemetry for traces but advising against its use for metrics and logs. He emphasized the importance of structuring logs and storing them efficiently, warning against the pitfalls of a 'soup of unusable data'.
Connecting the Dots: The Value of Coherent Telemetry
Douch's key insight was that the value of observability lies in connecting logs, metrics, and traces, not treating them as separate entities. Logs are a subset of traces, and metrics are aggregations of the same underlying data. Building an observability platform, he argued, is about designing a coherent telemetry pipeline, not just selecting individual tools.
A Step Back: The Broader Perspective
What makes this discussion particularly fascinating is the broader implications it has for the tech industry. Self-hosted observability is not just a technical challenge; it's a strategic decision with potential competitive advantages. By taking control of their observability infrastructure, organizations can gain deeper insights into their systems, potentially leading to innovative solutions and a stronger market position.
In my opinion, Douch's presentation at QCon London 2026 was a valuable reminder of the complexities and opportunities that lie beneath the surface of modern technology. It's a testament to the ever-evolving nature of our field and the need for continuous learning and adaptation.
