dusanstanojevic

Hi, sorry for not responding sooner this one slipped through the cracks. I've tried to explain my reasons for starting traceway and what I've been building with it. It's not aimed to be the fastest ingestion tool out there, but it's backed by clickhouse and does minimal processing of the data, you can expect the perf to be as close to clickhouse as possible.

I'm working on a comprehensive benchmark of Traceway performance on different hardware configurations. The most I've tested with was the smallest managed ch instance with 250k traces per sec, handled it without a hiccup (but that's empirical). You can checkout the traceway git, there is an issue I've opened for benchmarking and you can subscribe/comment on it if you're interested. I'm benchmarking across sqlite, self hosted clickhouse and managed clickhouse. I am a huge fan of systematic, realistic and most of all reproducible benchmarks, so I am really excited about the progress on that.

Anyhow, you can checkout traceway and see what it offers, it's aimed at providing SLOs out of the box, session replays, alerting, configurable dashboards and great exception tracking (automatic symbolication) etc...

Got it, sorry, your comment just looked like a bunch of others and felt extremely out of place as nobody mentioned hitting any limits, especially with the Victoria stack (that I could see).

The comment read out of place/generic and given my previous experience I incorrectly assumed it was another generic bot - my bad.

Hopefully no hard feelings and Victoria looks great.

What do you like the most about it and how was your experience scaling it?

Hi, first of all, thank you for your response, I really appreciate actual Victoria team commenting.

This is probably the 5th comment (almost identical) I have seen about VictoriaMetrics, mostly on Reddit. I engaged with a few trying to learn more about your product and eventually just gave up. If you really want you can comb through my reddit comments, but be warned, I have commented on a lot of things... a lot...

You should be proud of what you have built, I've looked a bit more and your product looks incredible. I personally think that a sales person might have been testing an automation tool, but if it was actual customers that just shows how good the product is!

Traceway is not working on addressing the problems y'all are solving, it is more focused on having an out of the box experience with preconfigured dashboards, SLOs, integrations, automatic endpoint ranking, frontend session replays/RUM, symbolication etc.

Again, thank you for your comment.

Hi, creator of Traceway here.

I have not used SigNoz or ClickStack. I believe both are very good products that focus on slightly different things.

With Traceway I am trying to focus on providing a pre configured system that works out of the box, tells you whats wrong and what to fix. It comes with a great issue tracker, session replays/RUM, preconfigured Dashboards and it's easy to host. It has an alerting integrations with Slack and Github. The idea is to be proactive rather than reactive when you start growing, so rather than waiting for a failure to build out an SLO it comes with them included.

Based on what you're looking for Traceway may or may not be the best option for you, but all feedback is welcome and I am working on improving it every day. You can checkout the github + it's super easy to self host and I am always down to chat about how it works in the Traceway Discord.

This is refreshing to see, an actual comment about VictoriaMetrics that explains why and when it's good. Thank you!

I am the creator of Traceway, after reading that I think we're in adjacent vertices.

Traceway is aimed at providing teams with no dedicated SREs with preconfigured SLOs, a preconfigured dashboard and a really powerful exception tracking. It integrates with git to automatically open issues as well as slack and others. The idea is to get the experience of Datadog/Sentry in the open source space.

I've focused a lot on session replays, RUM and now symbolication for native, flutter and frontend applications - this might be a potential place where VictoriaMetrics could benefit from integrating with Traceway.

If any of that sounds potentially interesting let me know. Again, thank you for your comment.

Creator of Traceway here. Sorry for not responding sooner, didn't realize this HN post existed.

I saw it recently, I think it looks amazing, I haven't looked into it enough to know of any downsides. I am currently heads down in building as I have the roadmap cut out for the next few months, I will circle back to them as soon as I have a bit more time.

If you're familiar with their platform feel free to checkout Traceway and let me know if there are any incredible features you'd like to see in Traceway or anything they're missing. I am always looking for feedback!

Hi, I'm the creator of Traceway.

I have created Traceway because I looked at that stack and decided that I'm not going to add 7 more services to my stack that could all fail that I now have to maintain as well. Here is the list: Grafana, Otel Collector (to forward metrics), Prometheus, Loki, Tempo, Mimir, K8s.

This is not maintainable in production, unless you have a person to manage it. My app had about 500-1000 req/sec, this sounds like a lot but it's extremely light from the observability perspective. Why would I add 7 more points of failure and services to monitor for proper resource allocation for something like this? To add insult to injury I would have to keep building my SLOs, they wouldn't be tracked automatically by default, I would have to keep paying for Sentry because the issue tracking is quite lacking on Grafana. Oh almost forgot, I would also have to get an alerting provider or pay for that (maybe I'm wrong, it was 6 mo ago).

Anyhow, Traceway is a 60mb binary in Go, it works with Clickhouse or Sqlite and the data is stored on S3 when not used. That means you can host it with sqlite on the 2$ server or even free tier and have it working for your side projects, you can host it with managed clickhouse and get auto scalability on the db level.

The goal is to provide full observability and tools to fix issues directly for developers. What we have so far: alerts, notifications, SSO (google & github), integrations, metrics, preconfigured SLOs, distributed tracing, RUM/session recordings (js & flutter).

Almost forgot, you'd need a symbolicator as well, or your fe/mobile exception stack traces will be messed up in Grafana, I don't even know which tool they have for that, but it's always a new service to host and maintain...

I honestly think you are a bot. When ever I see Victoria mentioned it is always the same, always asking about hitting a scaling problem + promoting it, never responding to any comments. Hope I'm wrong, but it's been one too many. I refuse to use a product that is this dishonest.

No need to guess, I'll tell you the exact story of why I made Traceway!

Last Dec I had a customer complaint, took me 2 days to find the issue. I had to pay $800 for Sentry and a bit more for New Relic. The issue was a locking problem that happened only in very very specific cases, erroring in diff places and timing out in others, unfortunately power users were running into it often. I had two systems, no SLO to catch this and they were completely disconnected. Super annoying.

Anyhow, I spent a day looking at those and eventually went, screw this, I'm gonna just make this actually work. So I spent a few hours, hooked it up, no auth or anything nice, pulled the traces and found the issue. Turns out it was locking due to a long transaction existing in a scheduled task, it existed for years.

The big things for me is it automatically flagging issues, prioritizing them and taking into account: errors, response codes, timing. That's why I'm making it, no venture capital, funded by actual revenue from the start (not paying for Sentry or New Relic anymore). It's really a dev focused tool to help smallish teams find and fix issues before customers even have time to complain.

Anyhow, hope that explains it, kinda related to cloud costs, mostly just my personal frustration with existing tools. Also I did NOT want to host a 5 service stack (grafana, otel collector, prometheus, mimir, loki, k8s) for something that can be done in a 60mb go binary that runs on a 3$ server...

Hi, creator of Traceway here. Sorry for the late response, I didn't know this got posted and then my account was rate limiting on comments.

A lot of tools in this space, most pretty good. The goals when I started Traceway were: - simple to host and reason about - cheap to host - comes pre configured for sub 15 dev teams - completely open source, no paid ad-ons

It's not aimed at teams that can afford SREs (yet), the idea was to provide a good tool for smaller teams and startups in the sub 15 dev range.

The base of Traceway is Clickhouse, nothing special there, if you want you can run it with sqlite for self hosting. Sessions are also stored in S3 so the costs are minimal.

It is opinionated, it comes with preconfigured SLOs for flagging issues with endpoints and it will never try to sell you an AI SRE, you can file your exceptions/slo issues with the git integration and run what ever AI you want on it (I was sick of observability tools trying to sell me an AI). The goal is to have a one line setup, for OpenTelemetry, that gets you everything you need in Traceway without anything needing to be additionally configured. It's Datadog/Sentry but combined and fully open sourced.

I'm a huge fan of open source, here is what we've done so far for making existing solutions better:

1 - Session Replays/RUM

Session replays are usually a premium/expensive feature. With Traceway you can self host them and add them to your app in minutes. I am working on making this a standalone feature that ties into the otel sdks for mobile/js so that you can get your spans/logs/metrics/exceptions from any platform connected to your session replays in Traceway. At one point I got nerd snipped into making it work with Flutter, so we are the only solution I know of that has affordable usable session replays for Flutter.

2 - Symfony Otel

Symfony, the php framework, had no library that offered a few line setup and worked out of the box with open telemetry. We wrote one, you can use it with any tool out there.

3 - Symbolicator

We're working on a symbolicator that will be Open Telemetry Collector compatible, so that you can get your stack traces for Js/Flutter/Android/iOS resolved back. From what I can tell no good solution exists for this currently.

I will make a proper HN post at some point with more info on the project, right now I am focusing on building. If you have any ideas or things you'd like to see feel free to comment, join our discord community or open the issue in our git, we're always happy to accept PRs.

Clickhouse is just a database, it has a really neat feature that infrequently accessed data is pushed back to S3 minimizing the costs. It also heavily compresses the data when storing it.

I am the creator of Traceway and it's my all time fav database. Having said that the repositories in Traceway are completely modular, I've implemented the sqlite version so that I can skip docker containers locally and to simplify self hosting for side projects (it runs on like 2$ servers without issues). This is why it's uniquely suitable for telemetry data and why I've used it as the base of Traceway.

They've acquired HyperDX because it was a major Clickhouse user because their whole platform was telemetry on top of Clickhouse. I hope they don't fully pivot into the space as it would be quite awkward, but there are alternatives and I can always redo repositories with a diff storage engine/db.