hey there

I had not seen https://lemmy-meter.info/ yet, but I just reached out to see if they’re still maintaining it and can add us.

Another public grafana I’ve found useful in the past is this one:

• https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=eviltoast.org
• https://grafana.lem.rocks/d/edf3gjrxxnocgd/federation-health-activities-behind?orgId=1&var-instance=All&var-remote_instance=eviltoast.org&var-remote_software=All
• https://grafana.lem.rocks/dashboards

For the most part I’ve just been relying on a local uptime-kuma instance + regular server metrics if there is an issue.

lol I suppose it being a cat isn’t the worst thing.

Cats do like toast (probably)

I’m not sure if it was the migration or the update that triggered it, but I saw it for the first time too and its a bit… big.

If its a one time per user thing, its probably okay. I get that they would like to get paid. If its a Wikipedia style of yelling for money every page load, I’ll have more of an issue with it

All appears to be well, but let me know if not!

Also leaving the CDN off for now. It might affect speeds some, but we’ll see.

Pictrs should be working as well:

Live on the new environment. CDN is also disabled at least temporarily. Doing some additional testing.

This is going to be starting soon. Just doing a few last minute checks and then we’ll stop lemmy on the old servers, promote the new db server, start lemmy on the new servers again.

Orange cats are cute when they’re not causing trouble

This was a dumb issue but I think I might’ve solved it. Let me know if you keep noticing issues!

(I didn’t update my list of allowed ips from my cdn, so rate limiting was all wonky)

Hmm have you seen more of these errors by any chance? I’ve gotten a few alerts about 502s but didn’t see anything obvious at the same time

testing big pictures again

Thanks for the heads-up. pictrs should hopefully be better now. I upgraded it, gave it more ram, and migrated the database from the file-based sled repo to postgres.

I might still migrate the storage itself at some point, but I don’t think it’s a bottleneck at the moment.

Let me know if you keep noticing issues with it!

not 4k:

edit: pasting 4k image still fails

Haha well at least you helped figure out why pictrs crashes occasionally :)

I think I have a temporary fix (more memory!) for now, but I’ll work on it some more. A few months ago I mentioned migrating to new hardware - I haven’t actually done that yet, but do plan on doing that hopefully in the next month. It’ll give me more resources to dedicate to lemmy and co.

Looks like it was an issue with our pictrs instance. When I cleared my cache, I noticed avatars/etc weren’t loading either.

Should be fixed now!

Do you recall if they were all around the same time? I wonder if you opened it right after the bot posted a bunch of things all at once or something like that? I haven’t paid much attention to it, but I wouldn’t be surprised if it just runs on a cron or something.

Also, I would be curious on people’s opinions for de-federating from them. I’m not interested in seeing the reddit reposts, but I also don’t want to restrict what people can/can’t see too much.

We weren’t ever defederated from lemmit.online, but it’s possible someone recently subscribed to some communities there. When you’re not logged in, what is the home page sorting by for you?

I don’t actually see much from lemmit.online on the main feed right now even when not logged in, but it looks like the bot is marked as a bot account: https://eviltoast.org/u/bot@lemmit.online

If you haven’t already, you can also go to your account settings and disable “show bot accounts” to hide posts from bots as long as they mark themselves as a bot. It only works when you’re logged in though.

There’s actually a view votes link in lemmy-ui, but it’s only exposed to admins and mods:

It’d be nice if it was available to all since I think that’s how some of the other fediverse software does it. Lemvotes still looks pretty convenient though because it can get all of the vote info into one place.

If it’s for personal use, you could probably set up a small cloud vm at hetzner or wherever and just use it to reverse proxy back to the server in your house through a vpn.

Actually installing/setting it up isn’t too bad though. There’s a few matrix channels if you run into issues, but it seems like the ansible route is pretty popular. It at least has a few configs in the repo you can use as a reference when needed.

Essentially it is:

• postgres
• pictrs
• lemmy api
• lemmy-ui
• nginx / caddy / etc

I think you can get away with not having pictrs, but I don’t remember offhand. If you do set up pictrs, I’d recommend turning off the option in lemmy that proxies all images through your local pictrs instance. It’ll eat up disk space really fast if it’s on.