Lemmy, what are some of your "oh shit" work stories? - eviltoast
  • tetris11@lemmy.ml
    link
    fedilink
    arrow-up
    40
    arrow-down
    2
    ·
    edit-2
    11 months ago

    I have a small PC I use for exposing a private PC to the wider web via nginx proxy. It had two accounts on it: mine, and one I called “remote” with some basic password I set up to forward the proxy connection.

    One day, this machine started making 100% CPU noises, for several hours. Wtf? I check the processes and a Tor node had been setup and was transmitting gigabytes to some Russian IP.

    My brain goes into panic mode, I kill the process, wipe the remote user, and eventually pull the Ethernet plug.

    I wish I hadn’t wiped the user directory as I wanted to know what was being sent and where. Nonetheless the logs showed that several Russian IPs had been attempting an SSH brute force for literally months and one finally guessed “remote” and weak password I set for it.

    I have decades of experience on Unix system, and I cringe having made such a rookie mistake.

    Lesson learned: change the default SSH port to a transient port, have one dedicated SSH user with a non-standard username, and use auth-key entry only.

    I still wonder what was being sent over that Tor node, and why it required all the CPU cores. My best guess is crypto mining, or it was used for a DDOS attack net somewhere.

    • corship@feddit.de
      link
      fedilink
      arrow-up
      10
      ·
      11 months ago

      Obfuscation is not security, changing the port doesn’t increase your security

      • boatswain@infosec.pub
        link
        fedilink
        arrow-up
        19
        arrow-down
        3
        ·
        11 months ago

        I see this claim all the time, and it bugs me every time. Obfuscation is a perfectly reasonable part of a defense in depth solution. That’s why you configure your error messages on production systems to give very generic error messages instead of the dev-centric messages with stack traces on lower environments, for example.

        The problem comes when obscurity is your only defense. It’s not a full remediation on its own, but it has a part in defense in depth.

        • dan@upvote.au
          link
          fedilink
          arrow-up
          6
          ·
          11 months ago

          Changing the port isn’t really much obfuscation though. It doesn’t take long to scan all ports for the entire IPv4 range (see masscan)

          • lud@lemm.ee
            link
            fedilink
            arrow-up
            7
            arrow-down
            1
            ·
            11 months ago

            It helps against stupid automated attacks though.

            If someone has changed the port it’s likely that they have set up a great password or disabled password auth all together.

            It’s worth it for just having cleaner logs and fewer attempts.

            • dan@upvote.au
              link
              fedilink
              arrow-up
              3
              ·
              11 months ago

              It’s worth it for just having cleaner logs

              Those logs are useful to know which IPs to permanently block :)

      • tetris11@lemmy.ml
        link
        fedilink
        arrow-up
        15
        arrow-down
        3
        ·
        11 months ago

        I hear you, but I disagree:

        It buys you enough time to check the journals and see that a group of IPs have attempted various ports giving you enough time to block the IP altogether.

        It also buys you disinterest from the malicious host, since probably there’s a hard limit on how many ports they will test, and they will flag your machine as “too much work” and try another.

        Again, I agree with you that obfuscation is not security, but it sure does help.

        • corship@feddit.de
          link
          fedilink
          arrow-up
          5
          ·
          11 months ago

          From what I understand you obfuscate the port in order to limit the amount of incoming attacks. But then fail2ban would be a much more effective tool.

          The disinterested aspect you described is the actual problem. Because it’s based on the assumption your port won’t be found, but it definitely will, and as soon as that happens you’ll end up in a database such as shodan and the entire effect is GONE.

      • tetris11@lemmy.ml
        link
        fedilink
        arrow-up
        3
        ·
        11 months ago

        I think they were either computing crypto-hashes and passing on the results back home (via Tor), or they were using my machine to send out several ping/fetch requests over Tor to DDOS some unknown target machine.

          • tetris11@lemmy.ml
            link
            fedilink
            arrow-up
            2
            arrow-down
            1
            ·
            edit-2
            11 months ago

            I want to say “yes” but you should still try to change the default ports for any process open to the web. Just because they can’t guess your ssh, doesn’t mean they can’t upload a root php script to your webserver which allows file uploads.

            Just be as invisible as possible. Run nmap on your localhost with the defaults and see if anything is set to open. If so, change that port.