Comment

Comment by danpalmer

This project falls foul of the same issues as almost every one of these personal assistants: there's no indication that it works.

Each part of this is very easy to implement, even if writing the code by hand. A chat interface, plus memories, plus semantic search, a little database, some embeddings, cron triggers. It's all trivial to implement... unless you want it to work, and more than that, if you want to know for sure that it works.

Because the harness is key, the prompting is key, the exact wording of the skills matters, the way the memories work, the embeddings used, all of these have influence over how well the system actually performs at being a personal assistant.

I've plugged together the dozen libraries necessary to do this sort of thing, and then it sorta works and sometimes doesn't. I realised that the whole problem is actually one of evaluation. If you don't have evals that say that it recalls memories with a certain performance, if you don't have evals that say that it actually completes tasks some percentage of the time, then you don't know if it actually works.

I've learnt not to bother investing my time in anything like this unless there's evidence that it actually works beyond "on my machine" from the devs.

parent danpalmer 21 days ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton