PeterCorless

I am so disgusted by this current regime.

This is actually a great article. We, as an industry, need to make sure that we address the points raised in this article. Especially What is, and what isn't a "streamhouse?"

How would you know you've achieved "streamhouse?" What objective requirements does it need to meet?

Disclosure: I work at Redpanda, which was part of the launch, and I will be hosting a webinar soon with some of the companies part of the technical working group. I'll try to raise some of the points made in this blog as part of the discussion if I have the chance.

This is really cool stuff. Amazing the efficiencies Pinot brings to data systems.

[Disclosure: I used to work at StarTree. Still a fan!]

I think that's the issue: a human can't. We'll need to set up "radar" agents to find out new tools for agents.

> Agents are cardinality-hungry. They want the high-cardinality data you'd normally drop: individual trace IDs, per-request attributes, full tag sets. They are very patient. They will sift through it.

The agents themselves are not likely going to be doing the high cardinality queries or they will keel over. They have limited memory buffers. They will take many seconds to return results. They are likely going to be limited in terms of QPS.

From the blog: > Apache Iceberg, with data stored as Parquet on S3, and most of the system implemented in Go

You have just ensured that queries will have a p99 >1 second. This is kind of antithetical to having an agent be fast.

You couldn't run any sort of real-time service, where hundreds of thousands to millions of events were occurring per second, and you needed to adjust to that in milliseconds.

The terms "p99" and "QPS" do not occur anywhere in the article. Which leaves the question of scalability to a user's imagination.

I applaud the direction. I am looking for objective evidence.

Vera does what NVIDIA calls Spatial Multithreading, "physically partitioning each core’s resources rather than time slicing them, allowing the system to optimize for performance or density at runtime." A kind of static hyperthreading; you get two threads per core.

It's somewhat different from how x86 chips do simultaneous multithreading (SMT),