drob

Sometimes, but I think there are a few problems with this:

1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.

2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.

3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.

We should publish some stats. The fix rates on bugs are around 45% baseline, increasing over time as we learn from behavior within a given codebase. The merge rates for the codebase health work we do (e.g. deleting dead code) are very high, at least 90%.

Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.

Should work fine. Ping me [email protected] or @danlovesproofs with your account info so we can look into it?

We've been thinking about this too. We have some ideas. Thanks for the comment, in any case – gave us a lot to chew on.

Other auth providers for sure. We'll be adding shortly.

Using an alternate auth provider won't even prevent you from scanning non-public GitHub code. There's a GitHub OAuth App just for auth (which is what you're seeing here), and a separate GitHub App that you need to install either way to give Detail access to the right repos. We can swap out the former for Google/Okta/pw if you want to avoid this warning. GitHub Apps (the half that manages repo access) have a much finer grained permissions model.

We've run it on a few firmware repos and gotten good results. A lot of firmware code tends to have really poor type-safety which means lots of low-hanging bugs.

We should be able to handle cross-compilation. Want to try it? Ping me in any direct channel ([email protected] / @danlovesproofs) and we can keep an eye on your repo.

Just github for now, but purely for reasons of plumbing. We'll add gitlab and others.

We support java, c/c++, kotlin, ruby, and swift as well. Did you have something specific in mind?

Github only for now. Out of curiosity, is yours on gitlab? Something else?

We should be able to find something interesting in most codebases, as long as there's some plausible way to build and test the code and the codebase is big enough. (Below ~250 files the results get iffy.) We've just tested it a lot more thoroughly on app backends, because that's what we know best.

Fix is deploying, sorry about that!