AlexC04
Joined 16 years ago
Joined 16 years ago
Increasingly I've been asking the LLM to do things that are only seconds of work to do manually on my own. Earlier in the process of learning how to use these things I think that was a somewhat reasonable decision in the name of learning how these things work.
Now... It's maybe the case that getting a bit more practice in with keeping on top of how we do things the manual way ... might still have value? both for the token cost and perhaps for the value that gets unlocked through understanding your systems a little better?
I'm also a little torn since the leaders in the field of "tokens for everything" like Cherny and Karpathy are pretty vocal about not writing any code on their own anymore. While they also don't really pay for tokens the way the rest of us do (I'm out of pocket for mine) is there something to be said for joining that frontier wave of developers?
https://docs.vllm.ai/en/latest/api/vllm/model_executor/layer...
vLLM apparently already has an implementation of turboquant available - which is said to losslessly reduce the memory footprint required by 6x and improve inference speed by 8x.
From what I understand, the steps are:
1. launch vLLM 2. execute a vLLM configure command like "use kv-turboquant for model xyz" 3. that's it
I've got two kids under 8 years old, a full time job, and a developer-tools project that takes like 105% of my mental interests... so there's been a bit of a challenge finding the time to swap from ollama to vLLM in order to find out if that is true.
SO buyer beware :D - and also - if anyone tries it, please let me know if it is worth the time to try it!
have you or had you considered pursuing a vscode extension? (and by extension cursor?)
In my free time I'm also building out a set of agentic tools and I've found that there's so much you get "for free" by working in the space where developers already are.
Git integrations, text editing, highlighting, etc...
Now - I'm still in pre-alpha local development only so who knows what my adoption story will look like after I actually get it onto the vscode marketplace, but I dunno - just making converstaion I guess :D
Anyways, this really seems like an interesting project and I'll take a deeper look outside of office hours! Thanks for sharing.
One of the things it made me think about is whether it made sense for using when editing large markdown files would it be more efficient to convert a document form markdown to DOM then back again for the purposes of editing a large markdown file via code agents? (or a json)
The theory being that agents are always asking me for pemission to use sed in bash to edit markdown files -- could tree-sitter do the same thing using its code-editing capabilities? And would that difference be materially impactful? Could I lower the token cost of writing an extensive plan by choosing a format that allows me to use tree sitter?
I really haven't explored that much yet since I've been working on other things but it was more just one of those things that make you go hmmmmm... maybe someone else knows :)
nice share. thanks.
OH! I commented before I realized it was a product pitch ... literal seconds before she introduced ACE.
If I'm honest, the idea of a collaborative agentic coding interface is pretty novel and interesting. Still impressed with the article just the nature of my appreciation has changed.
(edit: I hope this is an obvious joke. less facetiously these are pretty jaw dropping numbers)
> Feels like a fundamental bottleneck for production agent systems, so would love to compare how you're thinking about the latency vs accuracy tradeoff.
I'm really not focusing on latency right now. My short term goal is to prove the thesis that `ail` can improve same-model performance on SWEBench Pro vs. their own published results.
Can I run swebp with GLM-4.6 and get a score better than their published `68.20` https://www.swebench.com/?
The argument is that the latency right now just isn't the part we should worry about. If we're reducing the time to code something from ~6 weeks to 1 hour... then does it really matter tha we add an other 30 minutes of tool calls if we get it 100% right vs. 80% right?
Make it work -> Make it right -> make it fast.
I'm still on the first one tbh :rofl-emoji:
So before each turn is sent into the LLM we (potentially) run a local process to assemble a bespoke context of only what is required for that specific turn.
If a tool call is not going to be needed on the prompt, we don't include it in the system prompt on that round.
I'm still formalizing the spec at the moment and think I'm about six months to a year out before I have a full human ready UI running.
This is the foundational paper I'm basing the tool on: https://github.com/AlexChesser/ail/blob/main/docs/blog/the-y... while the spec starts here: https://github.com/AlexChesser/ail/blob/main/spec/core/s01-p...
Essentially I'm trying to build an artificial neocortex and frontal lobe to provide a complete layer of Executive Function that operates on top of our agents - like Claude Code (or whatever else).
I'm basing the roadmap on the about 100 years of cognitive science. We've legitimately had names for all these failure modes (in humans) since the 1960's. We have observations of what we're witnessing in agents from 1848.
We have the roadmap from Psychology.
Keep it up! you're on the right track.
Hong, K., & Chroma Research Team. (2025). Context rot: How increasing input tokens impacts LLM performance. Chroma Research. https://research.trychroma.com/context-rot
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
I'm going do a deeper read of your work in a bit. I'd love it if you took a look at my theory of artificial cognition The YAML of the Mind https://alexchesser.medium.com/the-yaml-of-the-mind-8a4f945a..., dropped in to the `ail` project and let me know what you think.
I just have to get the kids to school and I'll pop back into cortex later
I appreciate that it's pretty long feel free to point your LLMs at it
I love the startup news, the weekend projects that do cool stuff, and the overall "ethic" of the place.
I've done a few small projects recently, I keep knocking them out in order to try and find that thing that will "stick" (or just to have a bit of fun).
Here are some recent ones: http://fstr.net http://onemorelike.com http://itsalldiet.com http://chesser.ca
I used to be a high school teacher. Worked for five years in the United Kingdom teaching the little mytes. It was wonderfully fulfilling.
I'd love to write a film one day. Maybe I should spend my time doing that rather than all these little web projects .. they're fun too and I can see the results faster :)
alex dot chesser at gmail dot com