Kubernetes? docker-compose? How should I organize my container services in 2024? - eviltoast

Currently, I run Unraid and have all of my services’ setup there as docker containers. While this is nice and easy to setup initially, it has some major downsides:

  • It’s fragile. Unraid is prone to bugs/crashes with docker that take down my containers. It’s also not resilient so when things break I have to log in and fiddle.
  • It’s mutable. I can’t use any infrastructure-as-code tools like terraform, and configuration sort of just exist in the UI. I can’t really roll back or recover easily.
  • It’s single-node. Everything is tied to my one big server that runs the NAS, but I’d rather have the NAS as a separate fairly low-power appliance and then have a separate machine to handle things like VMs and containers.

So I’m looking ahead and thinking about what the next iteration of my homelab will look like. While I like unraid for the storage stuff, I’m a little tired of wrangling it into a container orchestrator and hypervisor, and I think this year I’ll split that job out to a dedicated machine. I’m comfortable with, and in fact prefer, IaC over fancy UIs and so would love to be able to use terraform or Pulumi or something like that. I would prefer something multi-node, as I want to be able to tie multiple machines together. And I want something that is fault-tolerant, as I host services for friends and family that currently require a lot of manual intervention to fix when they go down.

So the question is: how do you all do this? Kubernetes, docker-compose, Hashicorp Nomad? Do you run k3s, Harvester, or what? I’d love to get an idea of what people are doing and why, so I can get some ideas as to what I might do.

  • MSgtRedFox@infosec.pubEnglish
    2·
    10 months ago

    I really enjoy these type of conversations, learn a lot.

    Since you’ve gotten lots of good advice on container manager, I’ll encourage your desire for IaC/DevOps CM, etc.

    I believe all the leading CM choices support what you’re wanting to do. I can’t guide you on which one to chose, but just browse through the options or functions your favorite does for the Kx container solution you go with.

    I use SALT because of Security Onion, and open source IDS. I have all my nix systems being babysat by SALT, and can have a new x-arr media server, NGINX, blog, etc running in the amount of time to deploy the template (I use vSphere) and salt applies the desired state. Back up and restore a mount folder, np. IaC is only limited by your imagination. I have salt also specifying all the containers I have running, defining the config files, etc. Basically poor mans/simpleton kub.

    I suspect you already know this, but if there isn’t a module that directly does what you want like running SQL specific functions, you can just have it run programmatic CLI files on the host, or in the container for you.

    I am in the process of moving my IaC code from manager file system to Gitlab. I imagine you’d do this from jump street. Have fun.