sensodine

Joined 2 years ago

Karma
20
Submissions
8
Member since
Nov 26
building s2.dev
View on Hacker News

Recent submissions

Comment · 1 month ago
> every data system that doesn't need <100ms latency is moving to object storage

I think the opportunity extends below 100ms too, particularly given the existence of faster object storage tiers like S3 express or more recently GCS rapid bucket (both only offering single-zone durability, so still need to do quorum writes to get region-level durability as with standard tiers).

One of the tensions of course is how long to linger before flushing to object storage - you have to trade off directly between latency and cost of your API ops for PUTs.

When building the serverless offering of s2.dev (which is in a similar space, full disclosure!), we designed around stateful backend processes capable of constantly flushing multi-tenant objects (i.e., containing records from many streams), allowing streams to offer low ack latencies (~50ms p99 from same region) without blowing up the unit economics.

Congrats on the launch btw!

Comment · 4 months ago
> I was wondering if I could use this for multiple processes (codex, nvim) on the same machine

Could be useful for time-travel I suppose (in the same sense that git is, even if the repo only exists machine local). Right now that isn't very developed in opbox, other than the ability to clone an existing workspace at a particular historical moment (ob clone --as-of "2026-07-03T22:00:00-07:00"), but could be neat to "rewind" a workspace to any prior save state automatically.

Comment · 9 months ago
> it leaves me completely baffled about what s2 does, what problem s2 is trying to solve, or who the intended audience of s2 is

Regarding S2 generally (not just s2-lite), the intent behind it is to turn the core data structure from streaming platforms (like Kafka) into a serverless primitive -- kinda similar to what object storage did for file storage.

So if you are already in the world of working with streaming platforms, S2 gives you a simpler API, bottomless storage (S2 itself uses object storage for durability), and no limits on the quantity of streams you can create and work with. Streams also all have URIs and are directly accessible over REST with granular access controls.

This enables new types of patterns for working with streams, other than just the traditional ones where people typically reach for streaming platforms (like CDC, ETL pipelines, etc). An agent can have its own stream to serialize state onto, for instance; you can use a stream as a durable transport layer -- e.g., you want to reliably provide a flow of data (tokens from a model, financial ticker data, etc) to a user and allow them to resume from exactly where they left off if they are disconnected, for instance; you could use streams as a durable ingest buffer, for collecting data that will eventually reside in an OLAP like Clickhouse.

Comment · 1 year ago
Comment · 1 year ago
Comment · 1 year ago
(S2 team member here)

> I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it's still an object-backed streaming service, right?

This is how it works essentially, yes. Architecting the system so that chunks that are written to object storage (before we acknowledge a write) are multi-tenant, and contain records from different streams, lets us write frequently while still targeting ideal (w/r/t price and performance) blob sizes for S3 standard and express puts respectively.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton