Major IT outage affecting banks, airlines, media outlets across the world - eviltoast

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

  • Blackmist@feddit.uk
    link
    fedilink
    English
    arrow-up
    22
    arrow-down
    6
    ·
    4 months ago

    I’ve had my PC shut down for updates three times now, while using it as a Jellyfin server from another room. And I’ve only been using it for this purpose for six months or so.

    I can’t imagine running anything critical on it.

    • ccdfa@lemm.ee
      link
      fedilink
      English
      arrow-up
      43
      arrow-down
      2
      ·
      4 months ago

      Windows server, the OS, runs differently from desktop windows. So if you’re using desktop windows and expecting it to run like a server, well, that’s on you. However, I ran windows server 2016 and then 2019 for quite a few years just doing general homelab stuff and it is really a pain compared to Linux which I switched to on my server about a year ago. Server stuff is just way easier on Linux in my experience.

      • conciselyverbose@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        11
        ·
        4 months ago

        It doesn’t have to, though. Linux manages to do both just fine, with relatively minor compromises.

        Expecting an OS to handle keeping software running is not a big ask.

        • sugar_in_your_tea@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          3
          arrow-down
          1
          ·
          4 months ago

          Yup, I use Linux to run a Jellyfin server, as well as a few others things. The only problem is that the CPU I’m using (Ryzen 1st gen) will crash every couple weeks or so (known hardware fault, I never bothered to RMA), but that’s honestly not that bad since I can just walk over and restart it. Before that, it ran happily on an old Phenom II from 2009 for something like 10 years (old PC), and I mostly replaced it because the Ryzen uses a bit less electricity (enough that I used to turn the old PC off at night; this one runs 24/7 as is way more convenient).

          So aside from this hardware issue, Linux has been extremely solid. I have a VPS that tunnels traffic into my Jellyfin and other services from outside, and it pretty much never goes down (I guess the host reboots it once a year or something for hardware maintenance). I run updates when I want to (when I remember, which is about monthly), and it only goes down for like 30 sec to reboot after updates are applied.

          So yeah, Linux FTW, once it’s set up, it just runs.

          • uis@lemm.ee
            link
            fedilink
            English
            arrow-up
            1
            ·
            4 months ago

            not that bad since I can just walk over and restart it.

            You can try to use watchdog to automatically restart on crashes. Or go through RMA.

            • sugar_in_your_tea@sh.itjust.works
              link
              fedilink
              English
              arrow-up
              1
              ·
              4 months ago

              I could, but it’s a pretty rare nuisance. I’d rather just replace the CPU than go through RMA, a newer gen CPU is quite inexpensive, I could probably get by with a <$100 CPU since anything AM4 should work (I have an X370 with support for 5XXX series CPUs).

              I’m personally looking at replacing it with a much lower power chip, like maybe something ARM. I just haven’t found something that would fit well since I need 2-4 SATA (PCIe card could work), 16GB+ RAM, and a relatively strong CPU. I’m hopeful that with ARM Snapdragon chips making their way to laptops and RISC-V getting more available, I’ll find something that’ll fit that niche well. Otherwise, I’ll just upgrade when my wife or I upgrade, which is what I usually do.

              • uis@lemm.ee
                link
                fedilink
                English
                arrow-up
                1
                ·
                edit-2
                4 months ago

                I just haven’t found something that would fit well since I need 2-4 SATA (PCIe card could work), 16GB+ RAM, and a relatively strong CPU.

                4 SATA, 8GB RAM is easy to find. What do you need 16 gigs for? Compiling Gentoo?

                Star64 for ARM and Quartz64 for RV.

                • sugar_in_your_tea@sh.itjust.works
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  edit-2
                  4 months ago

                  Running a few services:

                  • NextCloud + Collabora (or maybe OnlyOffice, haven’t decided)
                  • Jellyfin - terabytes of data, need memory to avoid buffering
                  • Linux package mirror - higher RAM when running updates
                  • compile-farm for my personal Rust projects (e.g. CI/CD)
                  • in the future, HomeAssistant and security camera streaming

                  Star64 for ARM and Quartz64 for RV.

                  You got those backwards.

                  But yeah, I’ve had my eye on those, but I think 8GB will be a bit limiting, and the CPU in both the Star64 and Quartz64 may be a bit weak. I was about to buy one, then decided to upgrade my desktop PC to a SFF PC, which meant I had some decent hardware lying around. It’s mostly up and running now (still working on Collabora/Only Office), so I’ll be monitoring RAM usage to see if I really need 16GB or if I can get away with 8GB instead, but I have a feeling something like either of those two SBCs will be a bit limiting.

                  I don’t need all of those services, but I definitely want the first two, and they wouldn’t need to be used at the same time (e.g. if I’m watching a movie, I’m probably not messing with big docs in NextCloud). The last one is very nice to have, but it could be a separate box (I’m gusesing Quartz64 would be plenty, though I may want to play with object detection).

                  • uis@lemm.ee
                    link
                    fedilink
                    English
                    arrow-up
                    1
                    ·
                    4 months ago
                    • Jellyfin - terabytes of data, need memory to avoid buffering

                    At 8GB it won’t matter going for 16. Even 4 is a lot. It’s not like you watch same video many times.

                    • Linux package mirror - higher RAM when running updates

                    Wierd. Is it not rsync? I ran mirror for my local network on 1GB SBC.

                    • compile-farm for my personal Rust projects (e.g. CI/CD)

                    I was joking about compiling Gentoo. Might need that much RAM here.

                    • in the future, HomeAssistant and security camera streaming

                    So, powerful CPU or hardware encoder.

                    Star64 for ARM and Quartz64 for RV.

                    You got those backwards.

                    My brain got SIGBUS.

                    then decided to upgrade my desktop PC to a SFF PC, which meant I had some decent hardware lying around.

                    Not bad decision. Why buy something new, when something old works too.

                    but I have a feeling something like either of those two SBCs will be a bit limiting.

                    Maybe. I wish I knew PC-style computers on ARM or RV with upgradable memory. Maybe something from SiFive? Dunno.

        • corsicanguppy@lemmy.ca
          link
          fedilink
          English
          arrow-up
          4
          arrow-down
          5
          ·
          4 months ago

          big ask.

          Off the car lot, we say ‘request’. But good on you for changing careers.

      • Blackmist@feddit.uk
        link
        fedilink
        English
        arrow-up
        2
        ·
        4 months ago

        Because I only have one PC (that I need for work), and I can’t be arsed to cock around with dual boot just to watch movies. Especially when Windows will probably break that at some point.

        • uis@lemm.ee
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          2
          ·
          4 months ago

          Can you use Linux as main OS then? What do you need your computer to do?

          • Blackmist@feddit.uk
            link
            fedilink
            English
            arrow-up
            2
            ·
            4 months ago

            I need to run windows software that makes other windows software, that will be run on our customers (who pay us quite well) PCs that also run windows.

            Plus gaming. I’m not switching my primary box to Linux at any point. If I get a mini server, that will probably ruin Linux.

            • uis@lemm.ee
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              1
              ·
              edit-2
              4 months ago

              I need to run windows software that makes other windows software, that will be run on our customers (who pay us quite well) PCs that also run windows.

              Mingw, but whatever. Maybe there is somethong mingw can’t do.

              Plus gaming. I’m not switching my primary box to Linux at any point.

              Unless it is Apex and some other worst offenders or you use GPU from the only company actively hostile to linux, gaming is fine.

      • ji17br@lemmy.ml
        link
        fedilink
        English
        arrow-up
        8
        arrow-down
        1
        ·
        4 months ago

        Wow dude you’re so cool. I bet that made you feel so superior. Everyone on here thinks you are so badass.