First off, I know ultimately I’m the only person who can decide if it’s worth it. But I was hoping for some input from your collective experience.
I have a server I built currently running Ubuntu 22.04. I’m using KVM/qemu to host VMs and have recently started exploring the exciting world of Docker, with a VM dedicated to Portainer. I manage the VMs with a mix of virt-manager via xRDP, cli tools, and (if I’m feeling extra lazy) Cockpit. Disks are spindles currently in software Raid 10 (md), and I use LVM to assign volumes to the KVM VMs. Backups are via a script I wrote to snapshot the LVM volume and back it up to B2 via restic.
It all works. Rather smoothly except when it doesn’t 😀.
I’ve been planning an HD upgrade and was considering using that as an excuse to start over. My thoughts are to either install Debian and continue with my status quo, or to give Proxmox a try. I’ve been reading alot of positive comments about it here and I have longed for one unified web interface to manage my VMs.
My main concerns are:
- Backups. I want to be able to backup to B2 but from what I’ve read I don’t see a way to do that. I don’t mean backup to a local repository and then sync that to B2. I’m talking direct to B2.
- Performance. People rave about ZFS, but I have no experience. Will I get at least equivalent performance out of ZFS and how much RAM will that cost me? Do I even need ZFS or can I just continue to store VMs the way I do today?
Having never used Proxmox to compare I’m really on the fence about this one. I’d appreciate any input. Thanks.
Proxmox wont make backups to B2 easier, but since it is basically a web interface and API for Debian and KVM/QEMU you might be able to use your current backup strategy with very little modification.
As for ZFS, you can expect to use about a GB of RAM for each TB in a ZFS pool. I (only) run 2x 4TB drives in ZFS mirror and it results in about 4-5 GB of RAM overhead.
Another point you might want to consider is automation and the ability to use infrastructure as code. You can use the Proxmox Packer builder and Terraform provider to automate building machine images and cloning virtual machines. If you’re into the learning experience it’s definitely a consideration. I went from backing up entire VM disks to backing up only application data, making it faster and cheaper. It also enabled a lot of automated testing. For a homelab it’s a bit much, the learning experience is the biggest part. It’s an entire rabbit hole.
If you want to see how the automation looks like, check out my example infrastructure repo and the matching tutorial. Also check out my Alpine machine image repo which includes automated tests for image cloning, disk resizing and a CI pipeline.
I found this which leads me to believe I may be able to pipe
zfs sendto restic to replicate my current disk backup strategy. Presumably I could fire up a VM and build a zfs storage pool in it to test that theory out.So if I were to put 4x4TB in a RAID10 equivalent pool I’d be looking at ~ 8GB not 16, whew.
The rabbit hole is where all the fun is. Templating was something I never really got around to in my current setup. I do have an ansible playbook and set of roles that will take a brand new Ubuntu VM and configure it just how I like it.
Thanks for all the info. I’ll be sure to check out your repo.
My zfs cache for 6x 4tb drives in raidz2 is about 10gb of ram.
Replying to myself but I think this is a square peg, round hole, situation.
If I’m starting over with proxmox I likely need to rethink my entire backup strategy.