I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.