sksksk

The agent will tell me in the WhatsApp chat, and send a screenshot.

I have a VNC client app on my phone, so I log in and take control. But my next step is to setup a web based VNC client, and have the agent send me a link to it (with time based with), so that I can do it from within WhatsApp

For browsing, I use camoufox, it tries to hide the fact that it is an AI running a bot and seems to be fairly successful.

I set the browser window dimensions as the same is my phone, and added a VNC server. so if there are things I need to intervene on (like a captcha), I can log in and manually take control.

I'm always a bit wary about doing write up on things like this, as I kind of just muddle things together and definitely don't follow best practices.

I used OpenClaw a while back, I did find it useful, but it wasn't as reliable.

This time around, I found hermes a lot easier to use and configure. One thing I think has improved it is building apps to "pre-compute" data.

For example, with my email, instead of accessing my email servers directly, I have an IMAP server running locally; with mbsync running to download all the mail from all my email accounts locally. Then, when I ask hermes for something in my email, it's much quicker, doesn't hit any rate limits, and gets the full email (gmail sometimes doesn't send the full body of the email).

With my travel and purchase tracking. I parse an e-mail once as it arrive (I use a frontier model to generate a script which can parse e-mails from that merchant in the future). The parsed data is added to a SQLite database.

I ask hermes what's the address of the place we're staying at next week, It generates a SQL query, and queries the database.

Much cheaper/quicker/reliable than searching my e-mails for booking confirmations, and then parsing the e-mail.

My (unscientific) theory, is that doing things like that is much more important than the model or agent you use.

If there is something that the AI can't do, I don't use it for that.

I've given some examples in a sibling comment.

If you want to get started, I'd recommend just finding one thing to automate, and do what's necessary to get to that point. Once you've got that going, you can iterate.

For me, the thing I started with was the train tracking.

- In order to get train tracking working, it needed email access to get my trips.

- To get my email, I sync my email inboxes to a local IMAP server

- To check train times, it needs access to a train data API, I got an API key from realtimetrains.

- To apply for train delay compensation, it needs access to a browser, so I set up camoufox.

- It needs to log in to the compensation portal, so I set up a 1Password vault and gave it access to select credentials.

- To send me updates, it needs a gateway, so I set up a WhatsApp bridge.

So at this point, I've got an agent that can read my emails, browse the web, and chat to me over WhatsApp. Once you've got all that going, it becomes much easier to iterate and add new things.

I use the WhatsApp bridge

That's fair, I'll give a couple of example of things that I've used it for...

I built an app (in codex), that goes through my emails, pulls out any receipts, and adds the purchases to a sqlite database.

I do all my grocery shopping online (through my supermarket), so our weekly shop is stored in the database.

Hermes uses that to predict what items will need restocking (based on how often I buy it, and when I last bought it).

Every week, it logs into the supermarket website on my behalf, books a delivery slot, and adds all the items that need restocking to the shop.

Throughout the week, if I run out of something, I take a photo of the barcode and send it to hermes, it adds it to the shop for me.

I built a similar app for travel; if I have any upcoming travel, hermes will send me updates (delays, platform changes etc...).

I do a lot of train travel for work, if my train is delayed, it will automatically apply for the compensation on my behalf.

It reads my emails, and pulls any actionable items out, and adds it to an Asana list. I can either do the tasks myself, or assign it to Hermes and it will do it for me.

As an example, I had an task to book swimming lessons for my son, hermes monitors the swimming lesson availability and automatically booked it when a suitable one becomes available.

Shopping is a good use case, I have a clock that needed a new battery, it wasn't a standard battery, and required a screwdriver to open the compartment. I took a photo of the clock, sent it to hermes, and it found out what battery took and ordered it of Amazon for me.

I wanted to buy some smart thermostats for our radiators, but wanted to be compatiable with our current smart heating system. Hermes is connected to my home assistant, so it figured out what heating system we have and found a compatiable set of eBay (this is slightly more involved than it sounds, as there is a new version of the heating system with a new set of completely incompatiable accesories).

I run an personal AI agent locally (hermes), and use a combination of local models and whatever model I can get for a reasonable price (usually deepseek). After using it for a couple months, I'm converted. They're so good, the more you use it, the more useful it is; you internalise more of what it could be used for. It runs so many boring parts of my life, leaving me with more time to spend with my family. I liken it to release of the first iPhone; it didn't invent any new technology, but packaged different bits of tech together in a way that was consumer friendly. I think we'll look back at OpenClaw/Hermes/Muse/Instinct in the same way. I do worry about using purely cloud based offerings like Muse, I reckon there is a market for someone to package an consumer AI agent that lives on your home network.

There are so many podcasts where "successful" people (within a narrow definition of success) give advice.

The issue there, is that for every "successful" person, there are thousads who did the exact same things and didn't make it. Arguably a better thing would be to interview the ones who failed, and learn lessons from them.

Secondly, the person themselves can be selective in what they share, they may talk about the risks they took to get where they are, but not about the backstop of a wealthy family to fall back on if it all goes wrong (for example).

When I need to change the channel, I just get up and walk over to the television.

Thanks!

Another question I have, is from a product perspective, how do you decide how much of the internals you show to users (like token usage).

I always thought that seems like (from a non technical users POV), like a implementation detail; and quite hard to map to actual usage. For example, one request might use lots of tokens, and another will barely use any; but it can be difficult for a non technical user to undertand the difference.

It doesn't burn that many tokens, as most of the code is deterministic.

Receipt emails are in a predictable format, so codex built scripts to parse the emails. If there is a receipt email it doesn't recognise, it will use an LLM to build a parser for that kind of email once.

The agent also "self improves", so after asking once to get items to restock, it wrote a skill with the query it used in, so it doesn't burn tokens figuring it out again.

I was impressed that after it logged into my supermarket account, it inspected the network traffic and discovered a GraphQL API, and used that (instead of screen scraping); much more economic.

I've averaged 2.6 cents a day in token costs over the last month.

That’s interesting, so in that scenario, you fixed it for that customer.

How do you then ensure it’s fixed for customers in the future? Do you add instructions to agents to write scripts where possible?

no need to be rude, see my reply to a sibling comment for examples