Kubernetes? docker-compose? How should I organize my container services in 2024? - eviltoast

Currently, I run Unraid and have all of my services’ setup there as docker containers. While this is nice and easy to setup initially, it has some major downsides:

  • It’s fragile. Unraid is prone to bugs/crashes with docker that take down my containers. It’s also not resilient so when things break I have to log in and fiddle.
  • It’s mutable. I can’t use any infrastructure-as-code tools like terraform, and configuration sort of just exist in the UI. I can’t really roll back or recover easily.
  • It’s single-node. Everything is tied to my one big server that runs the NAS, but I’d rather have the NAS as a separate fairly low-power appliance and then have a separate machine to handle things like VMs and containers.

So I’m looking ahead and thinking about what the next iteration of my homelab will look like. While I like unraid for the storage stuff, I’m a little tired of wrangling it into a container orchestrator and hypervisor, and I think this year I’ll split that job out to a dedicated machine. I’m comfortable with, and in fact prefer, IaC over fancy UIs and so would love to be able to use terraform or Pulumi or something like that. I would prefer something multi-node, as I want to be able to tie multiple machines together. And I want something that is fault-tolerant, as I host services for friends and family that currently require a lot of manual intervention to fix when they go down.

So the question is: how do you all do this? Kubernetes, docker-compose, Hashicorp Nomad? Do you run k3s, Harvester, or what? I’d love to get an idea of what people are doing and why, so I can get some ideas as to what I might do.

  • jkrtn@lemmy.mlEnglish
    1·
    10 months ago

    Could you give a quick example of using NixOS configuration to launch a machine or deploying something remotely? I’m just starting to move beyond a single machine at home. I’d really like to get transition to infra as code.

    • Nico@r.dcotta.euEnglish
      3·
      10 months ago

      I recommend starting with ZeroToNix’s docs and then moving on to nixos.wiki, but here is a minimal, working example that I could deploy to a hetzner VPS that only has nix and ssh installed:

      { config, pkgs, ... }: {
  # generated, this will set up partitions and bootloader in a separate file
  imports = [ ./hardware-configuration.nix ];
  zramSwap.enable = true;
  networking.hostName = "miki";
  # configures SSH daemon with a public key so we can ssh in again
  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [ ''ssh-ed25519 AAAAC3NzaC1lNDI1NTE5AAAAIPJ7FM3wEuWoVuxRkWnh9PNEtG+HOcwcZIt6Qg/Y1jka'' ];
  # creates a timmy user with sudo access and wget installed
  users.users.timmy = {
    isNormalUser = true;
    extraGroups = [ "networkmanager" "wheel" "sudo" ];
    packages = with pkgs; [ wget ];
  };
  # open up SSH port
  networking.firewall.allowedTCPPorts = [ 22 ];
  # start nginx, assumes HTML is present at `/var/www`
  services.nginx = {
    enable = true;
    virtualHosts."default" = {
      forceSSL = true;            # Redirect HTTP clients to an HTTPs connection
      default = true;             # Always use this host, no matter the host name
      root = /var/www;        # Set the web root to ser
    };
  };
  system.stateVersion = "22.11";
}

      This sets up a machine, configures the usual stuff like the ssh daemon, creates a user, and sets up an nginx server. To deploy it you would run nixos-rebuild --target-host root@10.0.0.1 switch. Other tools exist (I use colmena but the idea is the same). Note how easy it was to set up nginx! If I was setting Nomad up, I would just do services.nomad.enable = true.

      As you can see some things you will have to learn (the nix language, what the configs are…) but I think it is worth it.

      • jkrtn@lemmy.mlEnglish
        2·
        10 months ago

        This is such a wealth of information, thank you! I’m really excited to try this out.

      • nopersonalspace@lemmy.worldOPEnglish
        2·
        10 months ago

        This is awesome, ZeroToNix is exactly what I was looking for. I’ve been interested in trying NixOS for a while but I always found the documentation obtuse (extensive, which is great, but not super beginner friendly). I’ll give it a try!

        • Nico@r.dcotta.euEnglish
          2·
          10 months ago

          Good luck on your Nix journey! Happy to help if you have questions.

          Of all the tech I use, I think Nix is the most ‘avant-garde’ in that it is super different from the usual methods (scripting, stateful things), but works very well once past the paradigm shift and the learning curve that entails.