[Troubleshooting] Random EOF shutdowns from Klipper. - eviltoast

Hi everyone, a week ago my printer (heavily modified Neptune 3) started randomly shutting down in the middle of prints. I come back to a print with the “Klipper reports: SHUTDOWN / Lost communication with MCU ‘mcu’” error message.

The printer has been “under construction” for the last couple of weeks, but it has been in varying states of “working” for most of the time - working well enough for me to print the parts I needed to get it back to “fully operational”. During this time, the printer never shut down like it is now.

Only once I started making little cosmetic changes did the problem present itself. I was running a known-good print, and I got the above error twice (first time after ~2 hours, second time after ~1 hour) before I got a successful print off of it. This was last week.

After this successful print, I continued other prints with no issues. After a day or two with no problems, an hour long print threw the error at me four consecutive times between 10-45 minutes into the print. This is when I started looking into my klippy log and found some relevant articles citing things like EMF interference, bad power supplies, faulty cables etc. I realized that one of the changes I had made rerouted the printer USB cable right around the Z-stepper, so I rerouted it to how it was originally and immediately managed a successful print. This was 5 days ago.

After moving that cable I had no issues with printing several-hour long prints… until last night. I had been printing all day, then the problem came back. After one print finished, I queued up another print with a plate full of parts, it failed after 1.5 hours. Tried the same print again, failed in 30 minutes. I re-sliced to only a handful of parts to see if I could get those to print before the error occurs, and it’s failing 15 minutes into the print.

The printer power supply is the unit that came with the Neptune, and it isn’t powering anything besides stock hardware (exception being the SKR mini board), so I don’t think it’s that. The pi is on a quality unit. The USB cable has been working for a long time so I also don’t suspect that, but I’m probably going to buy a new one today just to be sure. I adjusted my enclosure setup so that the Pi and SKR are able to get cool air (at one point had a personal fan pointing at the open electronics box, still failed).

Here is a link to my most recent klippy log (abridged to the start of the last failed print). I’m not very familiar with reading through this and finding oddities, but I do think it’s strange that it seemed to load my preheat script in the middle of printing right before the EOF error. (It should be noted that this preheat script was made 1 or 2 failed prints before this most recent one, so it isn’t the source of the error as prints were failing before the script was made). If there’s anything I’m missing or something else I can try, please let me know!

Edit: While typing this post, I was running the same failed print without filament and both heaters turned off. It ran for about 45 minutes (most recent failure occurred at 12 minutes) so I cancelled the print and started it again with heaters turned on, still without filament. It again ran for about 45 minutes, so I again cancelled it and started the print again, this time with filament loaded. It failed in 5 minutes.

Edit 2: A test print with heaters on and no filament failed after 1h8m. So it isn’t an issue with extruding filament.

Edit 3: New cable with the 5v leads taped off per @SzethFriendOfNimi@lemmy.world’s advice. Ran the print without filament until completion. Reloaded the same file with filament, print ran without issue until the 1h14m mark, at which point I tapped my Klipperscreen device to wake up the screen, and as soon as it displayed the status, the printer errored out. This can’t be a coincidence, can it? Whenever the print goes unmonitored for a long time, it fails as soon as I do something (load mainsail, turn on the klipperscreen) to check the status of it.

  • MrQuallzin@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 months ago

    How old is your Pi? I had the same issues on a generic board (Le Potato) but upgrading to a Pi 4 Model B fixed it for me. I have both my Neptune 2S (modded) and Neptune 3 Pro (stock) running on the same Pi with no issues.

    Definitely change your USB cable. If you’ve got some complex files, maybe the cable can’t handle the bandwidth (experienced this back when I was first starting out)

    • papalonian@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      5 months ago

      The pi is an 8gb Pi 4, maybe 3 months old. Usage never goes above ~15% for both cpu and mem during prints. Should be rock solid. This replaced my Pi 3 which was dropping USB connection with 3 webcams and klipperscreen running. (Currently only running one webcam.)

      I’m heading to the store now for a new USB cable, but I’m wondering if it might be something to do with my extruder motor? Since starting the OP, I’ve run the same test print 4 times, first 2 ran for ~45 minutes with no filament, 3rd one with filament errored in 5 minutes, I’m currently running test 4 and it’s gone 25 minutes without filament.

      • MrQuallzin@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        5 months ago

        The only thing I can think of with the extruder is thermal runaway, but Klipper would tell you if that was the case. Have you tried with no camera plugged in? If new USB cables aren’t working, it might be worth starting from scratch and rebuilding your Klipper instances. If a clean Klipper install (with no webcams or anything) is breaking, then it’s likely going to be a hardware issue.

        • papalonian@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          5 months ago

          I haven’t tried with the camera unplugged, but I have tried with it disabled in crowsnest, no change. It did fail not too long ago with no filament loaded, so that excludes an extruder issue.

          The Klipper install is relatively new so I don’t want to go that route just yet. I bought a new cable, but when I went to install it, realized that I had a nice Anker cable being used for my klipperscreen device and a random Chinese cable for the printer board (the Anker cable was longer so I blindly used it for the klipperscreen device). I swapped the Anker cable over to the printer, and changed it to a USB 3.0 port instead of the 2.0 it was on, and have the test print running now. We’ll see how this goes, I guess!

            • papalonian@lemmy.worldOP
              link
              fedilink
              English
              arrow-up
              1
              ·
              edit-2
              5 months ago

              It did not :( the print ran for 53 minutes before the error kicked in.

              Interestingly, and this has happened several times now, the print ran perfectly fine until I checked on it. I know it sounds crazy, but I’d say that 75% of the errors occur right after checking the status of the print. Knowing this, I started the print via Klipperscreen, then turned the Klipperscreen device off. I verified the print started via mainsail on my main computer, then closed the tab and let the print run. After ~50 minutes, I used my phone to load mainsail and verify the print was still running; it was. 2 minutes later, I refresh the mainsail screen on my phone and the print has failed.

              I will say that prints have failed both with a mainsail page constantly loaded, as well as with no mainsail or klipper interface loaded, so it can’t be caused entirely by that. However, since the problem started happening more consistently, it’s a pretty good chance that when it does fail, it’s within a minute of loading one of those interfaces.

              • MrQuallzin@lemmy.world
                link
                fedilink
                English
                arrow-up
                1
                ·
                5 months ago

                Do you have another printer to test with? Maybe it’s something in the SKR Mini that’s not working.

                • papalonian@lemmy.worldOP
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  5 months ago

                  No other printer to test with, no. Coincidentally, the parts I am trying to print are for a voron 2.4, because I’m absolutely tired of dealing with all the issues I’ve had with this one (see post history).

  • SzethFriendOfNimi@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 months ago

    Another thing to consider is making a data only USB ( I use a sliver of electrical tape over the 5v+ line ) to stop any power draw between the usb port and the printer.

    And, of course, check that you have a good power brick since shoddy ones always cause issues with pis.

    To be honest sounds like you’ve got a gremlin jumping from one piece of equipment to another so maybe prayer, voodoo, holy water and or appeals to the deity of universal fundamental force of your choosing could help too.

    • papalonian@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      5 months ago

      I’d never heard of the blocking the 5v trick before. If the current test fails I might give that a go.

      Power brick for the pi is good. It isn’t the official one, but a highly reviewed one, and I never get voltage warnings like I did on the old one.

      To be honest sounds like you’ve got a gremlin jumping from one piece of equipment to another

      You must have seen my other posts.

  • morbidcactus@lemmy.ca
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    5 months ago

    How are you powering the pi itself? Wondering if the wakeup draw from your screen is enough to make it unstable, 4 I believe having higher power drawn than the 3 if you’re using the same power supply. Pi isn’t oced either?

    Quick Edit: I run my v2.4 on a lepotato with oodles of usb attachments (camera, multiple mcus, wifi and a screen. Replacing the mcu connections with a usb-canbus bridge when the heatwave ends) but with it connected directly to a meanwell 5v5a PSU into the gpio header, have never had communication issues to the octopus pro I use. Skr mini on the other printer is connected to a laptop host so it definitely has enough power, did have some odd issues with the skr mini and having an accelerometer connected to the spi header on boot and separately the 24v supply becoming loose that I fixed by crimping ferules onto the supply wires when I added a molex connector to make taking the printer out of its enclosure easier.

    • papalonian@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      5 months ago

      Pi is not OC’ed, but I did just realize that it’s running on a power supply that came in a cheap starter kit and not the nicer one I’d ordered, so maybe that’s an issue. You would think the Pi would report under voltage if that were the case though, no? When Klipper errors out, the pi doesn’t shut off or report any issues.

      I will say that I’ve completed two 4 hour long prints with all external USB devices disconnected, and when I tried a duplicate of one of the completed prints with the klipperscreen device (on a new nice cable) and my webcam (connected but not enabled via crowsnest) it failed after 3 hours.

      • morbidcactus@lemmy.ca
        link
        fedilink
        English
        arrow-up
        1
        ·
        5 months ago

        You’d think right, anecdotally when I was meaaign with undervolting on my desktop, you’d get instability or crashes at the limits without useful errors. Could think maybe the processor isn’t getting enough power and that’s causing instability? Found a klipper thread that about losing connection with the mcu which totally can show up as an EOF error.

        • papalonian@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          2
          ·
          5 months ago

          I suppose it could be a power issue based on that. That’s the same article I linked in the OP; my klippy.log shows that it loses connection with the MCU, then a few lines later I get the EOF error.

          Printer is finishing the third 4 hour print with klipperscreen device unplugged. Really leaning towards this route, I’ll try swapping my nice power supply in when I get home and see if that helps.

          • morbidcactus@lemmy.ca
            link
            fedilink
            English
            arrow-up
            1
            ·
            5 months ago

            Fingers crossed. There’s hats you can get to power the pi off of printer’s 24v supply, considering doing it on my klippered mk3s and switch that over to using a SBC instead of the laptop.

            If you can swing it, definitely worth printing a spare set for the stealthburner + cw2, just in case you crack anything during assembly.

            • papalonian@lemmy.worldOP
              link
              fedilink
              English
              arrow-up
              1
              ·
              5 months ago

              I found this instruction of assembly pretty entertaining. “Break it and try again without breaking it”.

              Actually have the SB already on the current printer. It was having extrusion issues for the better part of 3 months and I built it as a last ditch effort to save the printer from the closet. After building that I realized how much I loved the voron designs and once the printer started acting up again I made the decision to just start piece mailing a 2.4

              • morbidcactus@lemmy.ca
                link
                fedilink
                English
                arrow-up
                1
                ·
                5 months ago

                Totally get you, I swapped my prusa’s hotend for a stealthburner as well, they’re super nice to service, especially with the 2 part PCB, but having the entire assembly mounted from the front is fantastic.

                Stealthburner ended up fixing your clog issues entirely?

                • papalonian@lemmy.worldOP
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  5 months ago

                  Stealthburner ended up fixing your clog issues entirely?

                  More or less. You can check my post history, made a lot of fuss about it here trying to get it squared away, in part it was caused by some magically reoccurring issue where my printer boards were reporting incorrect hotend temps (sometimes off by as much as 70°C). But even after fixing that side of things my old setup kept clogging; kept diagnosing back and forth between extruder and hotend, decided why not build a stealthburner and replace both, if that doesn’t fix the problem then I’m part way done with building a voron.

                  Now I’m building the voron anyways, haha