Securing Remote Proxmox for Off-Site Backup - eviltoast

Hey r/homelab,

I’ve got a 3-node Proxmox cluster at home and am gearing up to set up a Proxmox box at a remote site. My main concern is the trustworthiness of the network there, and I want to ensure maximum isolation.

This box will primarily serve as an off-site backup solution and I’ll be utilizing Tailscale, since I have zero access to the firewall or router of the remote site network and I’m also behind CGNAT on both sites.

Specifically, I want to configure the Proxmox box (and the iLO4 on my HP ProLiant ML310e Gen8) so that nobody on the remote network can access them.

Any tips or best practices on securing both Proxmox and iLO4 in this context? I’m especially interested in insights on network isolation and additional security measures.

Thanks a bunch for your help!

  • zedkyuu@alien.topB
    link
    fedilink
    English
    arrow-up
    0
    ·
    1 year ago

    Honestly, I wouldn’t stick any OOB management thing on any network I couldn’t trust. And it sounds like you have no ability to ensure that someone on the remote side can’t just go and change what your box is plugged into arbitrarily.

    With that in mind… I’d probably do Tailscale, bare metal (no virtualization), and set up the machine’s local firewall to drop all incoming connections from the ethernet port. Tailscale would connect out to establish its tunnel and then everything coming in via Tailscale would be fine.

    • sonnybwson@alien.topOPB
      link
      fedilink
      English
      arrow-up
      0
      ·
      1 year ago

      Thanks for your insights! I’m considering your suggestion seriously. To clarify, do you recommend not using iLO4? Are there alternative methods you’d recommend for secure access and control in a scenario where iLO4 isn’t utilized?"

      • zedkyuu@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        You state you want to configure things so that nobody on the remote side “can access them”; I interpret this to mean that they should be firewalled off entirely. In which case, unless you have some way to physically lock anyone on the remote side away from your stuff, there’s no way to ensure this. You’d essentially need to set up your own secure network on the remote end, one that nobody there can tamper with, and you’d want a separate firewall, probably one on which you’d run Tailscale with a local subnet.

        And I think you’d want this anyway if you are looking to do remote management with iLO4. At least in my experience, remote OOB management comes in handy mostly when I screw something up on the server and it won’t boot anymore. If Tailscale is on the server I can’t boot, then I can’t use it to access the OOB management either.

        Now, if you can’t lock things down physically, then things get even simpler. You should assume that someone there can do things like screw with the local console or with the hardware configuration, so you should not put things there in a form that they would be able to do anything with. This implies that your backup there should be encrypted (and it should be anyway; if a drive dies, you can discard it with no worry about data exfiltration). This also implies that it should have no ability to connect back to your home network, and that you should not run any services on it that you would be concerned about anyone breaking into or messing with. If all that is the case, then I think you probably are fine just plugging iLO4 into the same insecure remote network (again subject to the same caveat that it’ll be useless if your machine ever goes down and you have no other way to connect in); if anyone manages to break in, you’ll have controlled what they will be able to do with what they find.

        The bottom line is very simple: don’t trust untrusted environments!

        • sonnybwson@alien.topOPB
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Thanks for the insights! I’ll consider your advice on a separate secure network and encryption for backups. Your points on limiting access and iLO4 usage make a lot of sense. Appreciate your expertise as I figure this out!