Looks like it was an issue with our pictrs instance. When I cleared my cache, I noticed avatars/etc weren’t loading either.

Should be fixed now!

Do you recall if they were all around the same time? I wonder if you opened it right after the bot posted a bunch of things all at once or something like that? I haven’t paid much attention to it, but I wouldn’t be surprised if it just runs on a cron or something.

Also, I would be curious on people’s opinions for de-federating from them. I’m not interested in seeing the reddit reposts, but I also don’t want to restrict what people can/can’t see too much.

We weren’t ever defederated from lemmit.online, but it’s possible someone recently subscribed to some communities there. When you’re not logged in, what is the home page sorting by for you?

I don’t actually see much from lemmit.online on the main feed right now even when not logged in, but it looks like the bot is marked as a bot account: https://eviltoast.org/u/bot@lemmit.online

If you haven’t already, you can also go to your account settings and disable “show bot accounts” to hide posts from bots as long as they mark themselves as a bot. It only works when you’re logged in though.

There’s actually a view votes link in lemmy-ui, but it’s only exposed to admins and mods:

It’d be nice if it was available to all since I think that’s how some of the other fediverse software does it. Lemvotes still looks pretty convenient though because it can get all of the vote info into one place.

If it’s for personal use, you could probably set up a small cloud vm at hetzner or wherever and just use it to reverse proxy back to the server in your house through a vpn.

Actually installing/setting it up isn’t too bad though. There’s a few matrix channels if you run into issues, but it seems like the ansible route is pretty popular. It at least has a few configs in the repo you can use as a reference when needed.

Essentially it is:

• postgres
• pictrs
• lemmy api
• lemmy-ui
• nginx / caddy / etc

I think you can get away with not having pictrs, but I don’t remember offhand. If you do set up pictrs, I’d recommend turning off the option in lemmy that proxies all images through your local pictrs instance. It’ll eat up disk space really fast if it’s on.

Very cool, thanks for sharing! We’re set up on https://lemmy-federate.com/ so hopefully that is part of what triggers it to pick up new communities faster. There’s basically a bot user that will subscribe to new communities automatically, as long as they’re from another community set up there.

I’ll look into Jerboa. I tried Voyager on my phone, and didn’t see a way to change my home instance from lem.ee to eviltoast. I should probably go read the directions.

In case you haven’t figured it out yet, Voyager supports multiple accounts so I think all you need to do is click the middle button on the bottom row (probably says lem.ee if that’s’ the current home instance. Mine says eviltoast.org). That should pop up the accounts dialog where you can click edit and add another. At least on Android, now sure about ios.

Thanks! I’ve settled on the new hardware, but it’ll be a few weeks before it gets set up and I can actually migrate to it.

I should write about how everything is hosted now, maybe I will once I get the new stuff set up. Other than hardware, I’m planning on migrating away from hashicorp nomad to kubernetes along with a more normal stack.

Expect me to dump more in the selfhosting and cats communities. I’ve got a hell of a homelab going, and I need to brag.

Combining some of the best hobbies (cats and homelabbing) :).

You’re right about duplicated efforts. One thing I’d like to see improved on lemmy is something like federated/decentralized communities. There are multiple selfhosted communities for example, it’d be nice if they could all be combined into one place so that you don’t have to look for all of them when subscribing.

This post, right? https://eviltoast.org/post/13704694

Can you try it again? I found a couple other posts like that that were always returning 502 errors. They should hopefully be fixed.

Thanks for letting me know!

I have run into the session issue. It seems worse than before, but I haven’t figured out what exactly is causing it. I use Traefik as the main reverse proxy and I suspect it’s something different there vs the normal nginx config lemmy recommends.

I restarted things for now, but I have a couple things to try tomorrow. Thanks for the timestamp btw, I was able to find an issue around that time and my monitoring triggered briefly around the same time too.

Until I migrate to new hardware (and away from the current setup that uses traefik), I decided to enable a CDN temporarily (bunny.net). It’ll retry requests transparently, so it won’t fix the issue but should make it less noticeable. Let me know if you notice either way.

Poor cat looks like he had a brainfreeze

Thanks for the kind words!

I don’t see any mention of how you fund this other than it shares space with something else you host. It wouldn’t kill me to kick out $20 every now and then to help out. Particularly if it helps pay for bandwidth.

Everything’s funded out of pocket by me currently, but it’s not in risk of going away. I am planning on migrating to newer hardware in the near future though, and considering accepting donations since it will be more expensive to run.

ai bots

I’ve blocked the “abusive” ones for the most part, but do probably need to come up with something better. I saw some creative methods recently too that could be fun. Some are harder to block the normal ways because they are coming from residential IPs and using fake user agents.

I did find one consistent error related to postgres not having enough memory. That should be fixed now, but it looked like it was triggered by some internal queries from lemmy.

Hey @dbtng@eviltoast.org , thanks for the report.

There has been a lot more traffic than normally lately, mostly due to ai bot scraping/crawling. I think the “too many requests” errors you saw are probably from that. Hitting an overall rate limit in the webserver vs an individual limit.

Looking at the logs, there have also been more errors lately, but it’s hard to tell how many are real errors. For example, a “couldnt_find_post” error can show up as a 500 internal server error instead of a 404. I’ll dig into it some more

I normally use one of the apps (Jerboa or Voyager) and haven’t noticed any issues, but they might be retrying requests transparently.

It’s infra that was already hosting other things. Lemmy got added to it because there were spare resources in the cluster.

Oh man, I feel like eviltoast.org is on the expensive side of things now. There’s less than 20 mau, but I pay close to $400/mo. Not including object storage or hardware running at home.

Realistically this is hosted on hardware I was already renting for other reasons, so actual cost is zero for lemmy itself. I could probably migrate to a small vm for $20/mo and only lose redundancy.

I could be totally wrong but my take on it is that they’re just wanting the community to handle it so they can work on other parts of lemmy. They at least don’t seem like they’re against better mod tools.

day-to-day mod and admin actions on this instance are pretty low. Bigger instances like LW probably would have more reason to complain :D

IMO I don’t think it is something the lemmy devs are focused on, but there is a growing community creating helpful tools around moderation/federation/maintenance. There were some core features added like mods being able to see downvotes to detect mass downvoting, and open issues for things like an invite system.

LemmyAutomod is an example of a community effort. I saw some other similar bots posted too.

It does look like the issue in the previous thread is what caused a few instances to defederate from us. The spam issue was solved, so hopefully we’ll be able to get refederated again to those instances.