JohnTheNerd

that's a great idea! I've been looking into that (I'm merely logging all prompts in a JSON file for now, so that I can analyze them later).

skipping the LLM would be tough because there are so many devices in my house, not to mention it would take away from the personality of the assistant.

however, a recommendation algorithm would actually work great since i could augment the LLM prompt with it regardless of the prompt.

If you are scared of messing with electricity like I am, instead of using power monitoring, another viable but less reliable option is to use vibration sensors.

I picked up a simple Zigbee vibration sensor for less than 20$, taped it on top of the washer/dryer, and connected it to HomeAssistant using Zigbee2MQTT. After creating two automations to detect start/stop based on continuous vibrations over a few minutes, I had the notifications working the exact same way.

I think you have a valid point, but the risk of this feels exaggerated.

I already had a few entities I didn't really need it using (not for security reasons, but to shorten the system prompt). I simply excluded them within the Jinja template itself. I can see this being a problem with people who have their ovens or thermostats on HA, but I don't necessarily think it's an unsolvable issue if we implement sensible sanity checks on the output.

hilariously, the model I'm using doesn't even have any RLHF. but I am also not very concerned if GlaDOS decides to turn on the coffee machine. maybe I would be slightly more concerned if I had a smart lock, but I think primitive methods such as "throw big rock at window" would be far easier for a bad person.

when it comes to jailbreak prompts, you need to be able to call the assistant in the first place. if you are authorized to call the HomeAssistant API, why would you bother with the LLM? just call the respective API directly and do whatever evil thing you had in mind. I took an unreasonable number of measures to try to stop this from happening, but I admit that's a risk. however, I don't think that's a risk caused by the LLM, but rather the existence of IoT devices.

"successful" is a very optimistic way of looking at it. it has several downsides but largely works for my needs:

- read access is mostly available for sensors, even if access wasn't granted.

- some integrations (especially custom integrations) don't care about authorization. my fork mentioned in the blog does, because I explicitly added logic to authorize requests. the HomeAssistant authorization documentation is outdated and no longer works. I looked through the codebase to find extensions that implement it for an example. maybe I should submit a PR that fixes the doc...

- each entity needs to be explicitly allowed. this results in a massive JSON file.

- it needs a custom group added to the .storage/auth file. this is very much not officially supported. however, it has survived every update I have received so far (and I always update HomeAssistant)

I will share what I did in detail when I get some time on my hands

I recommend opening the original link if possible, because the archive link is missing the demo video and a few important updates to the jinja templates!

that is correct! the less I rely on external companies and/or servers, the happier I am with my setup.

I actually greatly simplified my infrastructure in the blog... there's a LOT going on behind those network switches. it took quite a bit of effort for me to be able to say "I'm comfortable exposing my servers to the internet".

none of this stuff uses the cloud at all. if johnthenerd.com resolves, everything will work just fine. and in case I lose internet access, I even have split-horizon DNS set up. in theory, everything I host would still be functional without me even noticing I just lost internet!

I do it, but I'm completely insane:

- I actually stay on top of all patches, including HomeAssistant itself

- I run it behind a WAF and IPS. lots of VLANs around. even if you breach a service, you'll probably trip something up in the horrific maze I created

- I use 2-factor authentication, even for the limited accounts

- Those limited accounts? I use undocumented HomeAssistant APIs to lock them down to specific entities

- I have lots of other little things in place as a first line of defense (certain requests and/or responses, if repeated a few times, will get you IP banned from my server)

I would not recommend any sane person expose HomeAssistant to the internet, but I think I locked it down well enough not to worry about a VPN.

it is almost entirely the LLM. I can see this in action by typing a response on my computer instead of using my phone/watch, which bypasses Whisper and Piper entirely.

your approach would work, but I really like the creativity of having the LLM generate the whole thing. it feels much less robotic. 8 seconds is bad, but not quite unusable.

it actually works really well when I use it, but is slow because of the 4060Ti's (~8 seconds) and there is slight overfitting to the examples provided. none of it seemed to affect the actions taken, just the commentary.

I don't have prompts/a video demo on hand, but I might get and post them to the blog when I get a chance.

I didn't intend to make a tech demo, this is meant to help anyone else who might be trying to build something like this (and apparently HomeAssistant itself seems to be planning such a thing!).

thank you for building an amazing product!

I suspect cloning OpenAI's API is done for compatibility reasons. most AI-based software already support the GPT-4 API, and OpenAI's official client allows you to override the base URL very easily. a local LLM API is unlikely to be anywhere near as popular, greatly limiting the use cases of such a setup.

a great example is what I did, which would be much more difficult without the ability to run a replica of OpenAI's API.

I will have to admit, I don't know much about LLM internals (and certainly do not understand the math behind transformers) and probably couldn't say much about your second point.

I really wish HomeAssistant allowed streaming the response to Piper instead of having to have the whole response ready at once. I think this would make LLM integration much more performant, especially on consumer-grade hardware like mine. right now, after I finish talking to Whisper, it takes about 8 seconds before I start hearing GlaDOS and the majority of the time is spent waiting for the language model to respond.

I tried to implement it myself and simply create a pull request, but I realized I am not very familiar with the HomeAssistant codebase and didn't know where to start such an implementation. I'll probably take a better look when I have more time on my hands.

power consumption. I am running multiple GPUs somewhere residential. the 4060Ti only draws 180W at max load (which it almost never reaches). 3090 is about double for 1.5x the VRAM, and it's notorious for briefly consuming much more than its rated wattage.

this isn't just about the power bill. consider that your power supply and electrical wiring can only push so many watts. you really don't want to try to draw more than that. after some calculations given my unique constrains, I decided 4060Ti is the much safer choice.

that is true, but consider two things:

- motherboards and CPUs have a limited number of PCIe lanes available. I went with a second-hand Threadripper 2920x to be able to have 4 GPU's in the future. since you can only fit so many GPUs, your total available VRAM and future upgrade capacity is overall limited. these decisions limit me to PCIe gen 3x8 (motherboard only supports PCIe gen 3, and 4060Ti only supports 8 lanes), but I found that it's still quite workable. during regular inference, mixtral 8x7b at 4-bit GPTQ quant using vLLM can output text faster than I can read (maybe that says something about my reading speed rather than the inference speed, though). I average ~17 tokens/second.

- power consumption is big when you are self-hosting. not only when you get the power bill, but also for safety reasons. you need to make sure you don't trip the breaker (or worse!) during inference. the 4060Ti draws 180W at max load. 3090's are also notorious for (briefly) drawing well over their rated wattage, which scared me away.