benesch
Joined 14 years ago
Karma
1,289
Submissions
214
Member since
Dec 12
Recent submissions
Comment · 1 month ago
It is working! Check back soon for progress updates. The graphs will update once we have the companion dev log entry explaining the perf optimization.
As you can see from the dates we're a few weeks behind our publishing schedule. Hard to pull ourselves away from the perf hacking to write the dev log entries.
Comment · 1 month ago
An automatically generated internal ID: (segment ID, doc ID). The user-provided primary key (the field called `id` in the document) turns into a secondary index at the storage layer.
Comment · 2 months ago
Yeah, we've had a PlanetScale cluster since the early days that powers the dashboard experience (turbopuffer.com/dashboard). That's a fairly standard webapp (customers, orgs, users, API keys, namespace stats) that benefits from the relational model.
When we built the cplane it was natural to just store the cplane ops in MySQL too. cplane ops belong to clusters which belong to customers, so it's actually rather nice to be able to foreign key it all together [0].
[0] Well, not actually enforced foreign keys, because Vitess, but you get the idea. It lets us apply settings at the customer level that flow to all ops for that customer's clusters.
Comment · 5 months ago
Yes, that’s right.
Comment · 9 months ago
I agree our sample may not be representative but we try to stay focused on the current and next crop of tpuf customers rather than the software industry as a whole. So far "CI prohibits network access during tests" just hasn't come up as a pain point for any of them, but as I mentioned in another comment [0], we're definitely keeping an open mind about introducing an offline dev experience.
(I am familiar with Bazel, but I'll have to save the war stories for another thread. It's not a build tool we see our particular customers using.)
Comment · 9 months ago
Thanks, appreciate this! Jotted down some notes on our roadmap.
Comment · 9 months ago
Yep, we're well aware of the selection bias effects in product feedback. As we grow we're thinking about how to make our product more accessible to small orgs / hobby projects. Introducing a local dev environment may be part of that.
Note that we already have a in-your-own-VPC offering for large orgs with strict security/privacy/regulatory controls.
Comment · 9 months ago
The exact CPU depends on the region/cloud provider, but this Granite Rapids CPU is representative: https://www.intel.com/content/www/us/en/products/sku/240777/...
Comment · 9 months ago
So I can note this down on our roadmap, what's the root of your requirement here? Supporting local dev without internet (airplanes, coffee shops, etc.)? Unit test speed? Something else?
Comment · 9 months ago
My point is it's enough of a hassle to set up that I've yet to see that level of restriction in practice (across hundreds of CI systems).
Comment · 9 months ago
> in many CI environments unit tests don't have network access, it's not purely a price consideration.
I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services.
We've considered the possibility of a local simulator before. Let me know if it winds up being a blocker for your use case.
Comment · 9 months ago
Often not a dealbreaker, actually! We can spin up new tpuf regions and procure dedicated interconnects to minimize latency to the on-prem network on request (and we have done this).
When you're operating at the 100B scale, you're pushing beyond the capacity that most on-prem setups can handle. Most orgs have no choice but to put a 100B workload into the nearest public cloud. (For smaller workloads, considerations are different, for sure.)
Comment · 9 months ago
For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing
Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.
Comment · 1 year ago
It’s hard to overstate the amount of service Ian provided to the Go community, and the programming community at large. In addition to gccgo, Ian wrote the gold linker, has blogged prolifically about compiler toolchains, and maintains huge swaths of the gcc codebase [0]. And probably much, much more that I’m not aware of.
I’ve had the pleasure of trading emails with Ian several times over the years. He’s been a real inspiration to me. Amidst whatever his responsibilities and priorities were at Google he always found time to respond to my emails and review my patches, and always with insightful feedback.
I have complicated feelings about the language that is Go, but I feel confident in saying the language will be worse off without Ian involved. The original Go team had strong Bell Labs vibes—a few folks who understood computers inside and out who did it all: as assembler, linker, two compilers, a language spec, a documentation generator, a build system, and a vast standard library. It has blander, corporate vibes now, as the language has become increasingly important to Google, and standard practices for scaling software projects have kicked in. Such is the natural course of things, I suppose. I suspect this cultural shift is what Ian alluded to in his message, though I am curious about the specific tipping point that led to his decision to leave.
Ian, I hope you take a well-deserved break, and I look forward to following whatever projects you pursue next.
[0]: https://github.com/gcc-mirror/gcc/blob/master/MAINTAINERS
Comment · 1 year ago
Yes, the three major open table formats are all quite similar.
When AWS launched S3 Tables last month I wrote a blog post with my first impressions: https://meltware.com/2024/12/04/s3-tables
There may be more in depth comparisons available by now but it’s at least a good starting point for understanding how S3 Tables integrates with Iceberg.
github.com/benesch
[email protected]
[ my public key: https://keybase.io/benesch; my proof: https://keybase.io/benesch/sigs/XTirUcJbA7dNwrQRe_-fze5YiXsPDPIs2sMq862fyDE ]