super_ar
Joined 2 years ago
Karma
43
Submissions
42
Member since
Sep 19
Recent submissions
Comment · 6 months ago
Comment · 7 months ago
I am seeing this pattern a lot lately. Teams start with a simple flow:
logs/metrics → Vector → ClickHouse
Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine.
Very typical issue that I see:
Time window limits: By default vector handles windowing in-memory. So with a higher load, it becomes too heavy to run there.
Missing support: When running in prod env, I have seen teams under pressure because there is no support available (except for Datadog customers). But most people I know run it self-hosted.
Scaling hits ceiling: I keep hearing similar numbers: 250k to 300k rec/sec per instance. Even by adding more resources, things do not scale. The consequences are: backpressure, latency spikes, etc.
At that point, it is no longer a “log pipeline.” It is a streaming system. Just not treated like one.
I wrote a deeper breakdown of this here if anyone’s curious:
https://www.glassflow.dev/blog/when-vector-becomes-your-stre...
Curious how people here are handling this.
Are you still pushing more logic into Vector, or have you split it out elsewhere?
Comment · 10 months ago
Really cool!
Comment · 10 months ago
Looks cool! Do you have any idea who "good" it is at detecting AI-generated text?
Comment · 10 months ago
This is interesting. Just wondering about your traffic volume and how long you have been running lcoalpdf?
For us, it is more like 5% of the traffic from GEO, but we have been running the company for 2 years and have created a lot of handwritten content for devs.
Where we saw friction with Flink was mainly: 1.) Operational overhead (jobs, state backends, checkpointing) 2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)
We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.
So Flink is more general, this is more opinionated and focused on this specific use case.