_ben_
Joined 10 years ago
Karma
47
Submissions
51
Member since
Oct 1
Recent submissions
Comment · 6 months ago
Comment · 6 months ago
Zindex is a stateful diagram runtime for agents. Agents create diagrams by sending structured operations through the Diagram Scene Protocol (DSP) - the platform validates, normalizes, and renders durable scene state rather than one-shot output.
Comment · 10 months ago
Thanks for the question. You’re right that S3 itself is simple and reliable, and yes, most engineers *can* write HTTP retries and multipart uploads. EdgeMQ isn’t trying to replace S3’s API, it’s what you need around S3 when you have lots of producers on the public internet.
It gives you:
* edge HTTPS endpoints (auto-scale, multi-region HA) * a WAL so accepted events aren’t lost * segmentation + compression * explicit commit markers for consumers * backpressure instead of silent data loss * and a standardized way every team lands data in S3
You could build that yourself on top of S3; many companies do. EdgeMQ exists for folks who wants that behavior but dont want to operate a custom HTTP to S3 ingest service forever.
Its also worth noting that its in the early stages and the next features to be developed are transformations whereby you can input format a (say, JSON) and deliver in s3 as format b (e.g. csv, parquet etc).
Comment · 3 years ago
Thanks for the questions. At a very high level, the AI uses statistical models that learn in real-time and estimate how frequently the data on the database is changing. The TTL's get set accordingly and are set per SQL query. The model looks at many inputs such as the payload sizes being returned from the database as well as arrival rates.
If PolyScale can see mutation queries (inserts, updates, deletes) it will automatically invalidate, just the effected data from the cache, globally.
If you make changes directly to the database out of band to PolyScale, you have a few options depending on the use case. Firstly, the AI, statistical based models will invalidate. Secondly, you can purge - for example after a scheduled import etc. Thirdly, you can plug in CDC streams to power the invalidations.
Feel free to ping me if you would like to dig in deeper (ben at) and this document provides more detail on the caching protocol: https://docs.polyscale.ai/how-does-it-work#caching-protocol
This blog also goes in to detail on how invalidation works: https://www.polyscale.ai/blog/approaching-cache-invalidation...
Comment · 3 years ago
At PolyScale [1] we tackle many of the same challenges. Some of this article feels a little dated to me but the data distribution, connectivity and scaling challenges are valid.
We use caching to store data and run SQL compute at the edge. It is wire protocol compatible with various databases (Postgres, MySQL, MS SQL, MariaDB) and it dramatically reduces query execution times and lower latency. It also has a JS driver for SQL over HTTP, as well as connection pooling for both TCP and HTTP.
Comment · 3 years ago
PolyScale [1] focuses on many of these issues. It provides a globally distributed database cache at the edge. Writes pass through to the database and reads are cached locally to the app tier. The Smart Invalidation feature inspects updates/deletes/inserts and invalidates just the changed data from the cache, globally.
Comment · 4 years ago
Yes, in the early versions of the automated invalidation, the logic cleared all cached data based on tables. That is no longer the case. The invalidations only remove the affected data from the cache, globally. You can read more here: https://docs.polyscale.ai/how-does-it-work#smart-invalidatio...
Comment · 4 years ago
For database caching outside of PlanetScale, PolyScale.ai [1] provides a serverless database edge cache that is compatible with Postgres, MySQL, MariaDB and MS SQL Server. Requires zero configuration or sizing etc.
What I'm solving for is AI-generated diagrams are usually one-shot: Mermaid/SVG/PNG gets emitted, but there’s no durable diagram state to update, validate, diff, or reuse.
Zindex makes the diagram itself structured state. Agents use a Diagram Scene Protocol (DSP) to patch nodes, edges, groups, relationships, constraints, and revisions; Zindex handles validation, layout, rendering, versioning, and storage.
So for Temporal/DBOS/Restate/Cloudflare Workflows, I’d imagine Zindex sitting beside the durable execution engine: the engine remains the source of truth for execution, while Zindex maintains the persistent, inspectable visual model derived from code or execution history.
[1] https://zindex.ai/