Also exploring mechanisms that try to pre-emptively keep attention-draining distractions/anti-patterns out of the context, things like when a model edits a file, we take the cache hit of removing the stale versions it read to make the modifications, replacing them with a reference syntax that the model can access in a sort of sandboxed auto-fork of the context.
That's going a little slowly because I'm trying to strike a balance between working 'reasonably' with extant models, and providing a mechanism to SFT/lorafy a model to make best use of it.
That's going a little slowly because I'm trying to strike a balance between working 'reasonably' with extant models, and providing a mechanism to SFT/lorafy a model to make best use of it.