no_circuit
Joined 6 years ago
Joined 6 years ago
The problem space and solution has been around for a while in big tech, and now there is a handful known products publicly known, and probably a couple dozen still secret ones. It is just now with AI/agents volume, there probably needs to be an easier solution for quick narrowly focused VCS views.
For filesystem mount, usually FUSE-FS, of a version control system to enable multiple viewers without transferring a lot of data see some current/previous implementations:
- Google: Piper via CitC (Clients in the Cloud) often used with Cider (web IDE)
- Meta: Sapling on EdenFS (from what I read, never worked there)
- Rational Clearcase, anyone else remember mounting VOBs?
The main issue I see is with the site -- it just seems like a big blob of AI-generated text I need to understand what is going on. The cool part wasn't even shown off: your GitHub UI clone that you can get to from seeing the benchmark code.
FYI, I also think the 4-way arrows logo has been used before, and still might be in use. I tried searching, but I think I saw a multi-colored one, maybe in a UK-based IT corporate training company's class I attended.
I think the approach of multi-format, multi-UI, and new (to you) programming language isn't optimal even with AI help. Any mistake that is made in the API design or internal architecture will impact time and cost since everything will need to be refactored and tested.
The approach I'm trying to take for my own projects is to create a polished vertical slice and then ask the AI to replicate it for other formats / vertical slices. Are there any immediate use cases to even use and maintain a UI?
So a few comments on the code:
- feature claims rate limiting, but the code seems unused other than in unit tests... if so why wasn't this dead code detected?
- should probably follow Google/Buf style guide on protos and directory structure for them
- besides protos, we probably need to rely more on openapi spec as well for code generation to save on AI costs, I see openapi spec was only used as task input for the AI?
- if the AI isn't writing a postgres replacement for us, why have it write anything to do with auth as well? perhaps have setup instructions to use something like Keycloak or the Ory system?
The MCP community is just reinventing, but yes, improving, what we've done before in the previous generation: Microsoft Bot Framework, Speaktoit aka Google Dialogflow, Siri App Shortcuts / Spotlight.
And interactive UIs in chats go back at least 20 years, maybe not with an AI agent attached...
The next thing that will be reinvented is the memory/tool combination, aka a world model.
git clone https://github.com/googleapis/googleapis.git
cd googleapis
find . -name '*.proto' -and -not -name '*test*' -and -not -name '*example*' -exec grep '^message' {} \; | wc -l
I think this more speaks to the tradeoff of not having an IDL where the deserializer either knows what type to expect if it was built with the IDL file version that defined it, e.g., this recent issue:https://github.com/apache/fory/issues/2818
But now I do see that the 4096 is just arbitrary:
If schema consistent mode is enabled globally when creating fory, type meta will be written as a fory unsigned varint of type_id. Schema evolution related meta will be ignored.So to reflect the real‑world practices, the benchmark code should then allocate and give the protobuf serializer an 8K Vec like in tonic, and not an empty one that may require multiple re-allocations?
I can see the source of an 10x improvement on an Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz, but it drops to 3x improvement when I remove the to/from that clones or collects Vecs, and always allocate an 8K Vec instead of a ::Default for the writable buffer.
If anything, the benches should be updated in a tower service / codec generics style where other formats like protobuf do not use any Fory-related code at all.
Note also that Fory has some writer pool that is utilized during the tests:
https://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
Original bench selection for Fory:
Benchmarking ecommerce_data/fory_serialize/medium: Collecting 100 samples in estimated 5.0494 s (197k it
ecommerce_data/fory_serialize/medium
time: [25.373 µs 25.605 µs 25.916 µs]
change: [-2.0973% -0.9263% +0.2852%] (p = 0.15 > 0.05)
No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
Compared to original bench for Protobuf/Prost: Benchmarking ecommerce_data/protobuf_serialize/medium: Collecting 100 samples in estimated 5.0419 s (20k
ecommerce_data/protobuf_serialize/medium
time: [248.85 µs 251.04 µs 253.86 µs]
Found 18 outliers among 100 measurements (18.00%)
8 (8.00%) high mild
10 (10.00%) high severe
However after allocating 8K instead of ::Default and removing to/from it for an updated protobuf bench: fair_ecommerce_data/protobuf_serialize/medium
time: [73.114 µs 73.885 µs 74.911 µs]
change: [-1.8410% -0.6702% +0.5190%] (p = 0.30 > 0.05)
No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
2 (2.00%) high mild
12 (12.00%) high severehttps://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
https://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
It seems if the serialization object is not a "Fory" struct, then it is forced to go through to/from conversion as part of the measured serialization work:
https://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
The to/from type of work includes cloning Strings:
https://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
reallocating growing arrays with collect:
https://github.com/apache/fory/blob/fd1d53bd0fbbc5e0ce6d53ef...
I'd think that the to/from Fory types is shouldn't be part of the tests.
Also, when used in an actual system tonic would be providing a 8KB buffer to write into, not just a Vec::default() that may need to be resized multiple times:
https://github.com/hyperium/tonic/blob/147c94cd661c0015af2e5...
TXSE's goal is to provide greater alignment with issuers and investors and address the high cost of going and staying public.
The alignment part translates IMO to avoiding political / social science policy issues like avoiding affirmative action listing requirements like the Nasdaq Board Diversity Rules that was just recently repealed: https://corpgov.law.harvard.edu/2025/01/12/fifth-circuit-vac....So it is as one might imagine, the formation was probably for similar reasons why owners are moving their company registration out of Delaware.
https://github.com/currentspace/capn-rs/blob/a816bfca5fb6ae5...
Yet there is a public interface that allows for initialization with "0":
https://github.com/currentspace/capn-rs/blob/a816bfca5fb6ae5...
It's like the LLM was able to predict that 1 is needed in the protocol, but wasn't relevant to check in the boilerplate.
I don't have a problem with newtype-all-the-things to ensure correctness in some areas, but no comments/constants does not lead to confidence.
But asking the community to review an AI-generated implementation week-old announced protocol, is more or less putting the recently coined-term AI "workslop" upon others. It doesn't really matter if it happens to be a good implementation or not.
There are two main issues I can think of right now:
1) Work going into the protocol is only useful for your implementation of it. The capnweb-core crate depends on the tokio runtime, and parts of the protocol/definitions are in the client crate:
https://github.com/currentspace/capn-rs/blob/a816bfca5fb6ae5...
What if someone wants to leverage work into the core parts of the protocol to use a different runtime or no-std?
2) The project has namespace squatted/swiped the best name for the official implementation of the project. I understand Rust/Crates-IO allows for this free-for-all, but isn't it entirely possible that Cloudflare already has Rust crates for this that they might open source? Or if someone else wants to make a competing implementation? Maybe it's just me, but I like to put my organization prefix on all my crates in the event I ever open source any of them.
Would you offer to transfer the crate names to Cloudflare if they were going to offer an implementation -- just like what happened with protobuf/Google?
For lower level libraries there is also the more downloaded SQLite library, rusqlite [2] who is also the maintainer of libsqlite3-sys which is what the sqlite library wraps.
The most pleasant ORM experience, when you want one, IMO is the SeaQl ecosystem [3] (which also has a nice migrations library), since it uses derive macros. Even with an ORM I don't try to make databases swappable via the ORM so I can support database-specific enhancements.
The most Rust-like in an idealist sense is Diesel, but its well-defined path is to use a live database to generate Rust code that uses macros to then define the schema-defining types which are used in the row structs type/member checking. If the auto-detect does not work, then you have to use its patch_file system that can't be maintained automatically just through Cargo [4] (I wrote a Makefile scheme for myself). You most likely will have to use the patch_file if you want to use the chrono::DateTime<chrono::Utc> for timestamps with time zones, e.g., Timestamp -> Timestamptz for postgres. And if you do anything advanced like multiple schemas, you may be out of luck [5]. And it may not be the best library for you if want large denormalized tables [6] because compile times, and because a database that is not normalized [7], is considered an anti-pattern by project.
If you are just starting out with Rust, I'd recommend checking out SeaQl. And then if you can benchmark that you need faster performance, swap out for one of the lower level libraries for the affected methods/services.
[1] https://github.com/launchbadge/sqlx/commit/47f3d77e599043bc2...
[2] https://crates.io/crates/rusqlite
[3] https://www.sea-ql.org/SeaORM/
[4] https://github.com/diesel-rs/diesel/issues/2078
[5] https://github.com/diesel-rs/diesel/issues/1728
But the centralized method, as in secretspec, not everyone will accept reading secrets in environment variables, as is also done with the 1password cli run command [1]. They also may need to be injected as files or less secure command line parameters. In the Kubernetes world one solution the is External Secrets Operator [2]. Secrets may also be pulled from an API as well from the cloud host. I won't comment on how that works in k8s.
To note, the reason for reading from file handles is so that the app can watch for changes and reload, e.g., key/token rotations without restarting the server.
But what could be useful to some developers is a secretspec inject subcommand (the universal version of the op inject command). I use op inject / dotenvy with Rust apps -- pretty easy to manage and share credentials. Previously I had something similiar written in Rust that also handled things like base64 / percent-encoding transforms.
If you aren't tied to Rust, probably could just fork external-secrets and get all the provider code for free.
[1] https://developer.1password.com/docs/cli/reference/commands/...
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
[1] https://github.com/seaweedfs/seaweedfs
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...