Lemmy.world (and some others) were hacked - eviltoast

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

  • trouser_mouse@lemmy.world
    link
    fedilink
    arrow-up
    92
    arrow-down
    2
    ·
    edit-2
    1 year ago

    First - really good summary and sounds like everyone is working hard.

    Cross posting the below comment.

    Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

    There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

    If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

    Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

    For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

    It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

    • B16_BR0TH3R@lemmy.world
      link
      fedilink
      arrow-up
      23
      ·
      1 year ago

      If a valid browser token gets stolen like in this case, then MFA won’t do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.

      • trouser_mouse@lemmy.world
        link
        fedilink
        arrow-up
        7
        ·
        1 year ago

        That makes sense, thanks so much - there’s a few good explanations here which really help! Would it be right in saying that all affected servers should be logging off all users - some have but not sure if all.

        • Riskable@programming.dev
          link
          fedilink
          arrow-up
          7
          ·
          edit-2
          1 year ago

          The fix is to force the use of a new JWT encryption key which–when set–would immediately invalidate all existing user cookies, forcing all users to relogin.

          Lemmy has a few weaknesses related to their use of JWT in cookies that need to be addressed… The biggest one being that they use the same secret key for all user cookies (every user should have their own unique session key). I’m pretty sure that if they implemented that the scope of this vulnerability would be drastically reduced (but I haven’t looked at the precise mechanism of the vulnerability yet).

          They also need to provide tools in the GUI for admins and users to invalidate all issued sessions (cookies) and a mechanism for regularly rotating session secrets (the cookie currently lasts for a year and even if the session token gets regenerated it’ll still use the same secret).

          They also need to make the expiration times configurable so that security-focused servers can set short expiration times. Related, they need to force the use of unique secrets for every session (even if it’s the same user using different devices/apps).

        • B16_BR0TH3R@lemmy.world
          link
          fedilink
          arrow-up
          6
          ·
          1 year ago

          I guess that would depend on the specific case. If you physically went on my computer to steal my token or infected my computer with a virus to do it then we can assume that no other tokens have been compromised. But if the malicious actor has managed to steal tokens from the actual server (which seems to be the case here) and not the client then yes, as the admin I would certainly require that everyone log in again as a safety measure.

    • hawkwind@lemmy.management
      link
      fedilink
      arrow-up
      19
      ·
      1 year ago

      Out of curiosity, where would the regulators go for a case like this? There’s no “company” running it per. se.

    • VeganPizza69 Ⓥ@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      1 year ago

      Can 2FA be enabled for all users? I don’t see the link to activate it after saving.

      edit

      Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.

      • nelrico@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        edit-2
        1 year ago

        Just reload the settings page after saving and you’ll see the activation link. Just now enabled 2FA for my account.

        • u/lukmly013 💾 (lemmy.sdf.org)@lemmy.sdf.org
          link
          fedilink
          English
          arrow-up
          5
          ·
          1 year ago

          Don’t log out! Open private tab and try logging in to test that it works. Lemmy uses SHA-256 TOTP digest which may not work correctly with some authenticators, only generating useless codes.

          • Muddybulldog@mylemmy.win
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            The interface for TOTP need to be greatly improved as well. I made sure that I had two browsers logged in when I did it because the flow is so hinky. Not having a confirmation process was a bit nerve racking.

          • VeganPizza69 Ⓥ@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.

      • YMS@discuss.tchncs.de
        link
        fedilink
        arrow-up
        5
        ·
        1 year ago

        This is not about turning you in, this is about protecting your users who all possibly just became victims of a crime, and for good reasons it’s not fully upon you to decide whether the possible consequences of this are serious for those users.

      • trouser_mouse@lemmy.world
        link
        fedilink
        arrow-up
        4
        ·
        edit-2
        1 year ago

        It’s more that many people expect those handling their data to be seen to follow the correct procedures and be trusted to handle the data in a fair, transparent, safe and secure way - and in addition to protecting their users, companies are probably encouraged to abide by the regulations because it is very easy for anyone to report where they think action needs to be taken, and regulatory bodies may be more lenient where correct process has been followed.

        If I chance a speeding or parking ticket I can’t be fined nearly 20 million pounds, although I wouldn’t trust some parking companies not to try it! (I’m not saying that would be the case in this instance.)

        https://gdpr.eu/fines/