Joined 15 years ago
I've been working on a suite of skills and a tiny MCP (also SQLite + SQLite-vec based) where the focus is on making it easy to produce "atoms" from quick brain dumps.
The chunking problem is "bypassed" by declaring each section a chunk, and having the LLMs rewrite drafts to sections that chunk well. That means lots of redundancy, and no "As explained above".
The intended reader isn't a human, but rather agents that generate human-friendlier prose, for different target audiences. By assuming the reader is an "expert", the idea is that it's much cheaper to mass-produce reviewed "atoms".
Itching to try that workflow with Atomic or Tolaria.
I'm not at all affiliated, just a happy consumer of their material.
I've worked on something in the same space, with a focus on reliable but flexible synchronization to many consumers, where logical replication gets impractical.
I have a mind to do a proper writeup, but at least there is code at https://github.com/cognitedata/txid-syncing (MIT-licensed) and a presentation at https://vimeo.com/747697698
The README mentions …
> A long-running transaction in the same database will effectively "pause" all event handlers.
… as the approach is based on the xmin-horizon.
My linked code works with involving the MVCC snapshot's xip_list as well, to avoid this gotcha.
Also, note that when doing a logical restore of a database, you're working with different physical txids, which complicates recovery. (So my approach relies on offsetting the txid and making sure the offset is properly maintained)
The memory space reserved for locks is finite, so if you were to have workers claim too many queue items simultaneously, you might get "out of memory for locks" errors all over the place.
> Both advisory locks and regular locks are stored in a shared memory pool whose size is defined by the configuration variables max_locks_per_transaction and max_connections. Care must be taken not to exhaust this memory or the server will be unable to grant any locks at all. This imposes an upper limit on the number of advisory locks grantable by the server, typically in the tens to hundreds of thousands depending on how the server is configured.
– https://www.postgresql.org/docs/current/explicit-locking.htm...
Deep merging two JSON parsed objects is innocuous enough everywhere else that most don’t think twice about doing it. Lots of widely used libraries that provide deep merging utilities have had security vulnerabilities because of this.
I guess you could argue that the wat is that objects coming out of JSON.parse don’t have null as its prototype.
>> JSON.parse("{}")["__proto__"]["A"] = "T"
"T"
>> W = {}
Object { }
>> W.A
"T"Java has its “gadget chain” class of vulnerabilities, where the presence of certain jars can turn object deserialisation into RCEs. I’d argue that Javascript has “pollution gadgets”.
Some years ago I struggled making lodash – which almost any non-trivially sized Javascript project has at least a transitive dependency on (possibly multiple versions of) – fix its “gadget” in its template function. It’s since been patched, and the conversation unfortunately deleted - https://github.com/lodash/lodash/pull/4518
Here’s two real world examples of turning a prototype pollution into an RCE: https://hackerone.com/reports/852613 and https://hackerone.com/reports/861744
(If you’re unfamiliar with prototype pollution: This is possibly a security bug in Javascript: `obj[a][b] = c` if a user controls a, b and c.)