matklad
Joined 12 years ago
Karma
1,382
Submissions
211
Member since
Feb 17
https://github.com/matklad
View on Hacker NewsRecent submissions
Comment · 2 months ago
The goal always was to refactor rustc to make compiler-based LSP feasible. The idea of original RLS, "we'll just use the compiler", is fundamentally sound. It's just that you'll need to turn the compiler sideways to achieve that, which is a hard work to motivate without having a real example demonstrating the benefits (or having political clout in the project to just mandate that ^^)
Comment · 2 months ago
I can shed some light here! This is going to be longish comment, but hopefully by the end of it you should understand _why_ we decided to avoid using the disk initially, even if you don't agree with that decision.
Historically, the decision to not use disk traces back to this comment https://github.com/rust-lang/rfcs/pull/1317#issuecomment-150..., which is perhaps the single GitHub comment that influenced my life most. Very high impact, thanks dgrunwald! Specifically,
>Don't store anything to disk. It's likely the oracle can be fast enough without doing this; and unnecessary complexity creates bugs. "Have you tried deleting the .ncb file?" (I remember having to do this a couple times per day when using VS, ca. 2005)
>Use lazy evaluation. The IDE is only interested in very specific bits of information, almost always restricted to a couple of lines around the cursor. Avoid calculating stuff that might never get used before it gets invalidated by the next code change.
>At least for C#, laziness saves so much time that incremental compilation is unnecessary for IDE purposes
The other part of historical context was that the motivation for creating rust-analyzer was that I didn't want to write a second Rust compiler (having been doing that for a couple of years at JetBrains). So it was explicitly an experimental project to prototype the right architecture for an IDE, to ultimately change how rustc works internally, so that, down the line, an IDE and a command-line compiler could use the same core. Given that rust-analyzer is now effectively a separate rust compiler, it's safe to say I am not good at achieving my life's goals!
In that context, I believe that avoiding disk was the _right_ decision:
* It's not really germane to the problem space, if all you need is literally a cache, it can always be added later.
* Disk is a can of worms of data consistency problems. They can be overcome with engineering effort to ultimately give better user experience, but user experience wasn't the primary goal. And using disk wouldn't actually illuminate the interesting aspects of the architecture, the intended primary goal.
* Finally, _not_ using disk would be a forcing function to keep analysis fast enough, to not make startup prohibitive.
The last one was a particularly big argument in my mind --- I didn't want to reach out for "easy" solutions prematurely, to avoid avoiding hard problems. And, again, my recollection is probably not 100% correct, but, until we added support for proc macros and build scripts, it was fine-ish from the perspective of startup time (RAM usage is a different story). The problem with proc_macros and build.rs is that they need to run the rust code, so they have to run the real rustc compiler, so all our usual IDE tricks ("information ... restricted to a couple of lines around the cursor") just don't apply.
The reason why we didn't add it later was that it seemed a relatively lower priority task than the work to share the parser between rust-analyzer and rustc. So that's what I was focusing on, though, I didn't deliver that. I still think we should do it! There's no _insurmountable_ technical reasons why the parsers can't be shared! It's just (a lot of) engineering work. And, while the parser is the boring part of compiler, it's the interesting part of an IDE.
Anyway, that explains how we ended up where we are.
That being said, I don't think that "just adding disk cache" is the right approach --- the salsa in-memory data structure is very sparse and pointy. Dumping that to disk would help somewhat, but wouldn't be a great long term solution. What is needed (I also explain this in https://matklad.github.io/2026/08/21/rust-glancer.html) is to design a compact, first class data format for representing analysis information about the crate, and than teaching rust-analyzer to be polymorphic in the source of data. For current workspace, you want to use a lazy incremental in-memory data structure (I do think we sadly need incrementally for Rust, given its compilation unit structure). For dependencies, you want to work off a compact on disk index. And, if the user "goes to definition" and mutates its file in place, we want to transparently switch between the two. The _pre requisite_ for that was to define a backend agnostic analysis API, and that work was always slowly progressing in the background (https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ), but it generally took the backseat, while sharing the parser was the main focus.
Comment · 2 months ago
To clarify, author is https://github.com/popzxc, not me! My thoughts are here: https://matklad.github.io/2026/08/21/rust-glancer.html
Comment · 3 months ago
Some thoughts on this here: https://matklad.github.io/2025/12/23/static-allocation-compi...
Comment · 3 months ago
To add more context around lifetime errors and TigerBeetle's particular style guide:
>Many projects opt to answer these kinds of questions through a style guide. TigerBeetle's TigerStyle is an example in Zig and Google's 31,000 word C++ style guide is another. The challenge with style guides is enforcement.
TigerStyle[1] is a bit more than just a style guide. The key rule for this discussion, uplifted straight from of NASA[2], is *static memory allocation*: all memory is allocated in the startup phase, and there's absolutely zero `alloc`s afterwrads . This plus crash only[3] design means that we never call `free`.
This rule is self-enforcing and compositional, in Zig. There's no global memory allocator, so the code after startup simply hasn't the API to allocate. You can't circumvent this by accident. Of course, if the programmer is byzantine, they can stuff allocator in the global, or just directly `mmap` and `unmap` pages of memory, but, at our scale, we don't have problems with that. This is a similar in kind (not degree) to Rust, where untrusted code generally can circumvent safety guarantees, even without literally spelling `unsafe`.
And, naturally, never `free`ing goes a long way towards solving many memory errors by construction. Empirically, they just haven't been a problem for TigerBeetle. It's hard to untangle contribution of static allocation in particular from everything else we are doing, but it would make sense for it to play a leading role.
(As a footnote, we aren't actually do static allocation to avoid memory errors, we use it as a linter to check that every quantity has a known _logical_ static limit, the main property we care about)
[1]: https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TI...
[2]: https://spinroot.com/gerard/pdf/P10.pdf
[3]: https://www.usenix.org/legacy/events/hotos03/tech/full_paper...
Comment · 5 months ago
To clarify, those are things an LLM considers to be issues, and LLMs can make mistakes.
Some of those are clear false positives, others I need to revisit tomorrow to say one way or another.
Comment · 8 months ago
Ha! I also use `line_number = line_index + 1` convention!
Comment · 9 months ago
Thanks, I haven't considered this! My history is usually naturally project-scoped, but I bet I'll find ~/make.ts useful now that I have it!
Comment · 10 months ago
I assume you ment to write `assert(subexpression != undefined)`?
This is resilient parsing --- we are parsing source code with syntax errors, but still want to produce a best-effort syntax tree. Although expression is required by the grammar, the `expression` function might still return nothing if the user typed some garbage there instead of a valid expression.
However, even if we return nothing due to garbage, there are two possible behaviors:
* We can consume no tokens, making a guess that what looks like "garbage" from the perspective of expression parser is actually a start of next larger syntax construct:
``` function f() { let x = foo(1, let not_garbage = 92; } ```
In this example, it would be smart to _not_ consume `let` when parsing `foo(`'s arglist.
* Alternatively, we can consume some tokens, guessing that the user _meant_ to write an expression there
``` function f() { let x = foo(1, /); } ```
In the above example, it would be smart to skip over `/`.
Comment · 10 months ago
In
const recv_buffers = try ByteArrayPool.init(gpa, config.connections_max, recv_size);
const send_buffers = try ByteArrayPool.init(gpa, config.connections_max, send_size);
A) clean up individual allocations on failure:
const recv_buffers = try ByteArrayPool.init(gpa, config.connections_max, recv_size);
errdefer recv_buffers.deinit(gpa);
const send_buffers = try ByteArrayPool.init(gpa, config.connections_max, send_size);
errdefer send_buffers.deinit(gpa);
const recv_buffers = try ByteArrayPool.init(arena, config.connections_max, recv_size);
const send_buffers = try ByteArrayPool.init(arena, config.connections_max, send_size);
const recv_buffers = ByteArrayPool.init(gpa, config.connections_max, recv_size) catch |err| oom(err);
const send_buffers = ByteArrayPool.init(gpa, config.connections_max, send_size) catch |err| oom(err);
fn oom(_: error.OutOfMemory) noreturn { @panic("oom"); }
Comment · 10 months ago
>If you look at MITRE's top 25 most dangerous software weaknesses, the top four (in the 2025 list) aren't related to UB in any language (by the way, UAF is #7).
FWIW, I don't find this argument logically sound, in context. This is data aggregated across programming languages, so it could simultaneously be true that, conditioned on using memory unsafe language, you should worry mostly about UB, while, at the same time, UB doesn't matter much in the grand scheme of things, because hardly anyone is using memory-unsafe programming languages.
There were reports from Apple, Google, Microsoft and Mozilla about vulnerabilities in browsers/OS (so, C++ stuff), and I think there UB hovered at between 50% and 80% of all security issues?
And the present discussion does seem overall conditioned on using a manually-memory-managed language :0)
Comment · 10 months ago
They do make a lot of sense in other contexts :-) From the actual rules, only #2 (minimize preprocessor) and #10 (compiler warnings) are C specific. Everything else is more-or-less universally applicable.
Comment · 10 months ago
There's some reshuffling of bugs for sure, but, from my experience, there's also a very noticeable reduction! It seems there's no law of conservation of bugs.
I would say the main effect here is that global allocator often leads to ad-hoc, "shotgun" resource management all other the place, and that's hard to get right in a manually memory managed language. Most Zig code that deals with allocators has resource management bugs (including TigerBeetle's own code at times! Shoutout to https://github.com/radarroark/xit as the only code base I've seen so far where finding such bug wasn't trivial). E.g., in OP, memory is leaked on allocation failures.
But if you manage resources manually, you just can't do that, you are forced to centralize the codepaths that deal with resource acquisition and release, and that drastically reduces the amount of bug prone code. You _could_ apply the same philosophy to allocating code, but static allocation _forces_ you to do that.
The secondary effect is that you tend to just more explicitly think about resources, and more proactively assert application-level invariants. A good example here would be compaction code, which juggles a bunch of blocks, and each block's lifetime is tracked both externally:
* https://github.com/tigerbeetle/tigerbeetle/blob/0baa07d3bee7...
and internally:
* https://github.com/tigerbeetle/tigerbeetle/blob/0baa07d3bee7...
with a bunch of assertions all other the place to triple check that each block is accounted for and is where it is expected to be
https://github.com/tigerbeetle/tigerbeetle/blob/0baa07d3bee7...
I see a weak connection with proofs here. When you are coding with static resources, you generally have to make informal "proofs" that you actually have the resource you are planning to use, and these proofs are materialized as a web of interlocking asserts, and the web works only when it is correct in whole. With global allocation, you can always materialize fresh resources out of thin air, so nothing forces you to do such web-of-proofs.
To more explicitly set the context here: the fact that this works for TigerBeetle of course doesn't mean that this generalizes, _but_, given that we had a disproportionate amount of bugs in small amount of gpa-using code we have, makes me think that there's something more here than just TB's house style.