`WARN: Error encountered while processing the incoming HTTP request` Spam (from lemmy.world & lemmy.ml) - eviltoast

Hi, I run the Lemmy instance over at https://compuverse.uk I seem to be getting spammed with literally thousands of incoming activity requests (primarily coming in from lemmy.world and lemmy.ml). (I measured about 1 new request ever 0.01 -> 0.02 seconds)

The error message in the Lemmy logs is very unhelpful, giving 0 detail about the actual error itself.

The UUIDs in the activities seem to be all unique, rather than duplicates. Does anybody have any idea what’s going on and why there is such a large torrent of requests coming in?

This was so bad it essentially caused a DoS attack on my instance, and brought it down. I have had to move the PostgreSQL server onto a more powerful machine in order to keep up with the request rate!

Any help or advice is greatly appreciated :)

Thanks,

Cameron

  • RoundSparrow@lemmy.ml
    3·
    1 year ago

    I looked at your instance, All, sorted by New. And I do see you have a lot of communities being fed from lemmy.ml and lemmy.world. Each vote, each comment, is federated as a single connection. Just the votes on posts and comments can be a huge number coming out of lemmy.world servers.

    The error messages don’t really include the JSON content that causes them, and it seems pretty typical to get a lot of errors with 0.18.3 in routine operation. I’ve had to edit the Rust code to add logging earlier in the process, or even capture it out of the Nginx to Rust proxyiing to find out exactly what activity was causing the error. It might also be in the PostgreSQL tables for activity before it reaches this point, but not sure.

    The UUIDs in the activities seem to be all uniqu

    I haven’t found any way to trace those back to something useful from the peer server. Getting into the JSON content I find the ap_id is far more useful… but it doesn’t seem to come out in these logs.