mayankchhabra
Joined 6 years ago
Karma
281
Submissions
22
Member since
Jul 27
Recent submissions
Comment · 9 months ago
Comment · 3 years ago
It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.
Comment · 3 years ago
Not sure how powerful their NAS is, but on Umbrel Home (which has an N5105 CPU), it's pretty useable with ~3 tokens generated per second.
Comment · 3 years ago
It's powered by Nous Hermes Llama2 7b. From their docs: "This model stands out for its long responses, lower hallucination rate, and absence of OpenAI censorship mechanisms. [...] The model was trained almost entirely on synthetic GPT-4 outputs. Curating high quality GPT-4 datasets enables incredibly high quality in knowledge, task completion, and style."
Comment · 3 years ago
The main difference is setting everything up yourself manually, downloading the modal, optimizing the parameters for best performance, running an API server and a UI front-end - which is out of reach for most non-technical people. With LlamaGPT, it's just one command: `docker compose up -d` or one click install for umbrelOS home server users.
Comment · 3 years ago
Ah yes, running on GPU isn't supported at the moment. But CUDA (for Nvidia GPUs) and Metal support is on the roadmap!
Comment · 3 years ago
It's an entire app (with a chatbot UI) that takes away the technical legwork to run the model locally. It's a simple one line `docker compose up -d` on any machine, or one click install on umbrelOS home servers.
Comment · 3 years ago
Yes! It can run on any home server or cloud server.
Comment · 3 years ago
Ah unfortunately that's not supported. When running umbrelOS on a Raspberry Pi, it stores everything on the HDD/SSD you connect to the Pi on install to make sure your data isn't susceptible to microSD card's wear and tear. So if you change the HDD/SSD, it will format the drive and use it as the main drive. And connecting two or more drives to the Pi isn't supported because the Pi lacks power to support more than 1 drive at once, causing abnormal behavior.
Comment · 3 years ago
Curious what aspects of Umbrel do you find not satisfactory? I know you said you'd like to see something less crypto focused. Is it the selection of crypto-oriented apps in the app store that you don't like to see? (And I totally get if that's the case, just trying to understand.)
We’ve spent the last 5 years building umbrelOS to make self-hosting accessible. Yesterday, we launched our dream hardware: Umbrel Pro.
Specs: - 4x NVMe SSD slots for storage (tool-less operation) - Intel N300 CPU (8 cores, 3.8GHz) - 16GB LPDDR5 RAM - 64GB onboard eMMC with umbrelOS
The chassis is milled from a single block of aluminum and framed with real American Walnut wood.
Here is a video of the manufacturing process if you want to nerd out on the machining details: https://youtu.be/4IAXfgBnRe8
Also, we built a "FailSafe" mode in umbrelOS, powered by ZFS raidz1. The coolest part is the flexibility: you can start with a single SSD and enable RAID later when you add the second drive (without wiping data), or enable it from day one if you start with multiple drives.
We also really obsessed over the thermal design. The magnetic lid on the bottom has a thermal pad that makes direct contact with all 4 NVMe SSDs, transferring heat into the aluminum. Air is pulled through the side vents on the lid, flows over the SSDs, then the motherboard/CPU, and exits the back. It runs whisper quiet.
Lots more details on our website, but we’ll be hanging out in the comments to answer any questions :)