Fiber provider is absolutely shitting the bed today [Services Degraded] - eviltoast

This is the second time today I’ve had to cut over to the backup WAN (which is much slower). Fiber provider is royally screwing the pooch today, and I currently have no ETA. The only silver lining is it’s a widespread issue, so hopefully they’re inclined to address the problem in a timely manner.

To relieve congestion on the backup connection, I’ve temporarily disabled external access to pictrs. All requests (GET or POST) to /pictrs will return a 404 response until the main connection is back online.

Sorry for the inconvenience. Please direct all rage at the shit fiber provider I foolishly “upgraded” to.

Status as of 5:17PM:

Service Status Notes
Lemmy ActivityPub Up (Degraded) Running on backup WAN
Lemmy API Up (Degraded) Backup WAN
Pict-rs Down Intentionally scuttled to save bandwidth
Tesseract Media Proxy Down Intentionally scuttled
Matrix Server Up (Degraded) Backup WAN
Matrix Web Client Up (Degraded) Backup WAN
DNS Up (Degraded) Operationally up, but no backup servers available
SMTP Up (Degraded) Backup WAN

Update: 4:30 PM: Tried to call and re-activate my account on my old provider since I own the equipment, and it’s all still hooked up. I can even still access my old account on the web portal. Should be easy, right? Fucking wrong! They said they couldn’t activate it until they sent me equipment, I returned it, and then called back to say I want to use my own. What the actual fuck?? I’m literally waving money at you, you’re still charging me a $100 “install” fee (where you have to do nothing), and all you have to do is take it (and my equipment serial number). Fuck my life (and fuck those idiots, too).

Update 5:17 PM: We remain on backup WAN. ETA for fiber restoration is within 24 hours, but I’m not holding my breath. Like, how do you even fuck up this badly? (the whole goddamned state is affected including business customers)

Update 7:39 PM: Primary connection is back online (for now?). Services have successfully failed back over.

Service Status Notes
Lemmy ActivityPub Up -
Lemmy API Up -
Pict-rs Up -
Tesseract Media Proxy Up -
Matrix Server Up -
Matrix Web Client Up -
DNS Up -
SMTP Up -