It’s been through quite a few iterations, but the current setup is a 35U Sysracks (not filled, I just like the space) with:
* 3x Dell R620 running K3os on clustered Proxmox
* 1x X11 Supermicro 2U with ZFS handling the disk array
* 1x X9 Supermicro 2U, also with ZFS, that is a backup target for the other one (WoL to ingest, then shutdown)
* UniFi Dream Machine Pro
* UniFi 24 port PoE 1G switch
* 2x UniFi UAP-AC-PRO APs (not in the cabinet, obvs)
* APC UPS with battery extension
The Dells have Samsung PM863 3.84 TB drives for Ceph, which Proxmox handles. They have some ancient 2.5” enterprise pull SSDs for boot.
The Supermicros have some cheap ADATA NVMe SSDs for boot (the X9 has a modded BIOS to make this possible).
In general I like it, but the current in-progress work is rewriting all of the Ansible playbook that generates QEMU images for Proxmox, and setting up PXE so I can grab IPs on boot for new images. The latter will enable me to shift off of K3os in favor of Talos. Technically I can do so now, but it requires manual input to set the IP addresses.
Beyond that, I want to get a 10G switch so a. RBD-backed devices are faster b. ZFS send/recv is faster c. Just because.
The Supermicros have some cheap ADATA NVMe SSDs for boot (the X9 has a modded BIOS to make this possible).
In general I like it, but the current in-progress work is rewriting all of the Ansible playbook that generates QEMU images for Proxmox, and setting up PXE so I can grab IPs on boot for new images. The latter will enable me to shift off of K3os in favor of Talos. Technically I can do so now, but it requires manual input to set the IP addresses.
Beyond that, I want to get a 10G switch so a. RBD-backed devices are faster b. ZFS send/recv is faster c. Just because.