About the outages (July 1st and 2nd 2024) - eviltoast

Happy Canada Day everyone.

Related to the outage that happened last night, we rebooted the Lemmy services but we’re still trying to figure out the root cause, which seems to point to an out of memory issue in the logs. However it’s not what we see in our monitoring console.

In the meantime, we will monitor the service more closely until we are confident the issue is resolved, and we will improve our tools to detect such a problem faster.

EDIT: Also happened at night on July 2nd, still trying to find the root cause…

Apologies for the extended downtime.