popzxc

I guess this is a feature worth announcing

I think it's a great way to stop understanding what's going on in your terminal.

IMHO the value of these nuggets is that you understand what you're doing and why; opening yourself to large amounts of non-default behavior likely will end up in a less than pleasant setup.

Not taking into account that different users might disagree on what is convenient and what is not, which is basically the point of making things configurable.

The exact shape is TBD (I target proc macros support for 0.3.0, whereas 0.2.0 will be about completeness/editors support), but in short -- I'm thinking about "plugin"/DSL architecture where proc macro _effects_ can be described beforehand and applied by the LSP without actual code execution. Executing proc macros is a lot of effort, but LSP mostly cares about the externally observable effects, which is a much smaller subset.

On the first question -- auto imports is an example of a "heavy" functionality since we indeed need to scan more than is imported in the project, which is why I'm still working on this to optimize properly (it's decently fast, but I want it to be faster before I'll enable it, since it's also something we want to see in completions, and completions must be _very_ fast). Still, we don't need "everything": we only need _reachable_ items in _direct_ dependencies; typically project has much less direct dependencies than transitive ones (e.g. 20 direct dependencies can fan out to 1000 transitive deps), and each dependency usually has less exported items than total amount of items. Finally, for that you only need semantic data (e.g. items) rather than bodies. So while it's a big task (though finding references is still significantly bigger and tougher, because that's where it gets to "it can be anywhere in reverse dependencies bodies), it's not "we need the whole world".

As for the size -- the Rust Glancer artifacts for Rust Glancer itself currently take 225mb. And that's proportial to the project size only, e.g. it doesn't grow over time on its own.

Yup. Basically, I have been combining both a manager and IC role for years now, which certainly helps working with LLMs: I can "delegate", and I know when to "intervene". It certainly is a skill, but the issue with LLMs is the confident voice you hear at the end of the turn: "the task is done, all is good"; and given that LLMs tend to praise its own code and tend to avoid global architecture reworks (I guess it's a training artifact), you need to understand the situation and resist hearing "nah, the architecture is fine, the rework is too much of work" phrased very convincingly. It's kind of peer pressure combined with both authority and glazing, which is a weirdly deceptive (to the point of being manipulative) mix.

I certainly enjoy working with humans much more than working with LLMs. Working with humans is usually "how can we get this done together", while working with LLMs is pretty much is "how can I make this model do what I need", which is something I don't like. An personal persistent thought I have now is not to transfer my relationship with LLMs to my relationship with human beings, and I don't really like it.

> So you would essentially fully index only once per project, + whenever you upgrade dependencies or upgrade rustc?

This is when you do full reindexing.

> Is that only an incremental indexing with normally low RAM usage?

Yup, on save you only update the crates that were invalidated.

Exactly. You pay higher RAM usage price once, for 5-30 seconds at the very beginning of the project (or if you make changes that invalidate the dependency graph, which is rather rare). In 95% of cases and 99.999% of idle time using the editor, you enjoy lower RAM usage.

And note that on dirty buffers Rust Glancer doesn't do reindexing at all: it reuses the last available analysis, plus it does syntax-based shallow overlay that is sufficient to be useful but doesn't necessarily detect semantic changes. It's a tradeoff, but this tradeoff makes Rust Glancer competitive in terms of latency with rust-analyzer without compromising RAM and while keeping your CPU cool.

> it completely reindexes the single file that’s saved

This is true, though I have to mention that change in the file might invalidate its reverse dependencies, which can make the partial analysis bigger than just one file/crate, but it's still very fast in practice.

I would say that LLMs have really good understanding of LSPs, but they are not necessarily good at building them. Had I blindly followed the proposed flow, Rust Glancer wouldn't have reached a stage where it is at least remotely usable. At some point the size of the project becomes too big for LLM to fit in its context window, and with the tendence to add code rather than remove, the bloat can explode really quickly. At a pretty early stage, I did not catch a situation where LLM suggested an extremely stupid design (because I wasn't familiar with the scope enough at the moment), and it implemented a whole new parallel hierarchy of functionality that was already implemented but in a _slightly different_ form. When I realized it, it took nearly two weeks to unfuck the situation.

So all in all -- yeah, LLMs can be good _domain experts_ when you build an LSP, but a) I wouldn't trust them blindly, and b) the quality of code is still very much your responsibility.

Storing structures mmapp'd is actually very tricky. I have experimented with rkyv initially having this idea in mind, but gave up because the machinery just to power the archive types was causing complexity to explode. The thing with zero-deserialization frameworks is that they are very limited in what they can abstract away, and it gets ugly pretty quickly.

So I don't deny the idea, just stating that it's probably _significantly_ more complex to implement than it sounds.

I will work on creating a more or less fair benchmark soon-ish, but right now the initial indexing typically consumes more RAM than rust analyzer does, but not awfully so.

The difference, however, is that with Rust Glancer you don’t need full reindexing often, so it probably compensates for that to a degree.