adamkurkiewicz
Joined 7 years ago
Joined 7 years ago
The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.
Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).
Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).
The most important thing I've found is to ask Claude to thoroughly audit the reply (to find all hallucinations). I usually ask it to give me an enumerated list of all facts and all legal cases quoted, and then I give it to a new instance to carefully validate each one.
Newer models are getting much better at not hallucinating German case law though :)
For slightly out of date founder bios (both Adam and Iva) were also co-founders here:
We've used the following CLI to do the freeagent upload:
https://github.com/anjor/freeagent-cli
How are you dealing with finding the receipts? Would you like to try a receipt finder that grabs them from your mailbox/ google drive?
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.
Many issues with the MIT game were discussed here:
https://physics.stackexchange.com/questions/43695/how-realis...
While the thread lists multiple issues, the issue that I found was with modeling temporal aspects of relativistic doppler (I also reported it on the github repo of the "slower speed of light" project, but that doesn't appear to be maintained):
https://github.com/MITGameLab/OpenRelativity/issues/17
In this game, it looks like the temporal component of the Doppler effect is modelled correctly.