How to remotely reboot a Linux host if SSH fails to connect? - eviltoast

Edit2: Thanks all for your responses! I have checked the logs, https://lemmy.nz/comment/6192604, and based on that removed tracker-miner-fs as it’s a search/index tool which I don’t need. No idea why it took over all memory. I’ll also get a WiFi Smartplug as a kill switch. Hopefully that solves it. Thanks again heaps!


I’ve got a HP ProDesk G3 which I’m using as home server, I’ve installed Ubuntu on it. Earlier this week the services I host on it stopped (Immich & Frigate). I tried to SSH, but it just hung after asking for a password. I could ping it, but it was just unresponsive.

I had to force reboot it manually. This is fine, but I’m not always at home.

The chip has Intel vPro as far as I know, which could be an option, but I have no idea how this works. The documentation on the Intel site seems focused on enterprises. I tried to connect with RealVNC which does not work, so I think I’ve got to install/configure something on the server first.

I also asked Bing Chat but it came up with non existing packages & commands. Welcome your thoughts!

/edit: I just found this, which seems to be exactly what I need: https://manpages.ubuntu.com/manpages/focal/en/man7/amt-howto.7.html

  • nightrunner@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    10 months ago

    I’m not in front of my computer atm, but I think I have something that can help you out. I have a 3-node Lenovo Thin client cluster that I manage their KVMs using the Intel vPro. I even went a step further using MeshCentral running on a VM to centralize my KVM access since I have 3 of them, but that’s another story.

    Anyway, I’ll see if I can grab you some URLs in the morning if someone else doesn’t beat me to it or you find it on your own running google queries.

    • sylverstream@lemmy.nzOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      10 months ago

      Thanks mate. It was a bit of a rabbit hole, I found stuff about the watchdog package, and you can configure it to use the iTCO_wdt module, but I also read it was blacklisted, and then I just gave up. I posted somewhere else in the thread what lead up to the hang. And, I think I’ll buy a WiFi smartplug so I can remotely reboot everything; assuming the WiFi still works :D