I’ve run my own mailserver for about 20 years. I don’t know if I’d recommend others do the same, but I wouldn’t recommend against it either.
Once it’s up and running, it’s surprisingly low-friction. I have a VPS with a provider I trust, and it’s running nothing else. Other than keeping everything updated, it requires very little ongoing maintenance. Mostly making sure you keep up with dmarc, TLS, etc best practices before the big providers call them requirements, instead of after.
I think the real difficulty is starting fresh, greenfield. Not only can one misconfiguration ruin your day, but if it’s an issue that other providers notice, the smell hangs around for a long time. Most the big providers (gmail, microsoft, yahoo) will do absolutely nothing to work with you, so if they take a dislike to you - well you’re screwed. There’s no way to get in touch with them, no way to ask them to look again, etc. The juggernauts will usually give the impression they don’t actually have anyone working for them at all.
You’ll also learn a lot more about DNS. Whether you like it or not :)
Things that aren’t so fun … OS updates are always the terrifying one. My provider is really good about letting you spin up a new instance while keeping the old one around for a month so you can switchover when you’re ready. I use that for most things - but for my mailserver, I don’t want to because I don’t want a new IP. I like that it’s my ball and I can pick it up and go play somewhere else if I want, but the amount of reputation that the big providers pin to IP, makes this a lot more difficult than it sounds.
The other fun sticking point is monitoring. I get emails if my mailserver (or DNS) go down … but because my mailserver is down, I don’t receive them until it’s back. That’s not ideal, but I never seem to get around to doing anything about it. (because when it’s working, I want to leave it alone. When it’s not working, it’s too late.)
I think the main thing to keep in mind is that it’s difficult to “lab” outbound mail. There’s very little “just trying something”, very little experimentation, etc. Getting things wrong has too many long-term effects. You wanted to try a new MTA and now Google think you’re a spammer? Putting the old one back does not fix your reputation. Putting the old config back does not fix your reputation. Doesn’t matter how much you clean, that smell is going to take a long time to go away.
That’s a lot more difficult to put into words than I thought it’d be.
I think the big thing is that they’re not in the race to the bottom. Their customers choose them for their level of services, not because they were the cheapest host in a list. So spammers don’t want to use them because they’re not the cheapest, and they don’t want to host spammers because that ruins their value proposition to their regular customers.
What else … small enough that they’re not faceless. and I’m not nobody to them either. They’ve been at this at least as long as I have, so it doesn’t feel like they’re going to disappear tomorrow. And they’re fairly active with their community through a good old-fashioned mailing list. Which also helps to get to know them and what level they’re working on. It’s nice knowing that when I mail them, I don’t get through to an AI, or an L1 on a script, I’m gonna get Andy.
It’s a tough one though, because trust is earnt, not researched. But I do prioritise putting a mailserver on a provider that keeps a clean house - because you don’t want to find yourself getting blocked because your neighbours misbehave.
tl;dr; everything AWS ain’t.