Horos
Joined 1 year ago
Joined 1 year ago
in my case the problem doesn't arise because control plane and data plane are separated by design — metadata and signals never share a concurrency primitive with chunk writes. the data plane only sees chunks of similar order of magnitude, so a fixed worker pool doesn't overprovision on small payloads or stall on large ones.
curious whether your control and data plane are mixed on the same path, or whether the variance is purely in the blob sizes themselves.
if it's the latter: I wonder if batching sub-1MB payloads upstream would have given you the same result without changing the concurrency primitive. did you have constraints that made that impractical?
From my pov, the worker pool's job isn't to absorb saturation. it's to make capacity explicit so the layer above can route around it. a bounded queue that returns ErrQueueFull immediately is a signal, not a failure — it tells the load balancer to try another instance.
saturation on a single instance isn't a scheduler problem, it's a provisioning signal. the fix is horizontal, not vertical. once you're running N instances behind something that understands queue depth, the "unfair scheduler under contention" scenario stops being reachable in production — by design, not by luck.
the FrankenPHP case looks like a single-instance stress test pushed to the limit, which is a valid benchmark but not how you'd architect for HA.
If you fix N workers and control dispatch order yourself, the scheduler barely gets involved — no stealing, no surprises.
The inter-goroutine handoff is ~50-100ns anyway.
Isn't the real issue using `go f()` per request rather than something in the language itself?
A few thoughts after your answers:
The E2E file sync part has existing solutions, and your access rights system is really a signed append-only log rather than a blockchain (no consensus, no decentralization) — which is fine, but the term might create misleading expectations.
What I'm more curious about is the access model itself. How are access tokens created and transferred? Who consumes them, and how does authorization propagate? Have you considered a salted API where each user carries a unique identifier, so the whole grant/revoke/delegate flow goes through a single unified mechanism regardless of what's being accessed?
The SQL sync layer is what actually caught my eye — I've worked on similar problems for specific use cases, and encrypted database sync between peers is a genuinely hard problem. That feels like your real differentiator.
On that note: does the SQL layer reference the file content or file paths? I'm guessing you built both interfaces because they're correlated — the SQL holds structured data that points to the encrypted files. If so, that's worth making explicit, because right now they look like two unrelated features rather than two sides of the same system.
my claude drive his own brave autonomously, even for ui ?