Found it. The TCP window size was too small.

Raising the maximum TCP window size on the NAS and changing the TCP window size on iperf fixed the problem. The bandwidth changed from 23 Mbit/s to 190 Mbit/s over VPN.

I believe the root cause is a combination of high latency between the endpoints and low maximum TCP window size. The latency between the two routers was 179ms, and the maximum TCP window size on the NAS was set to 128KB. Because of the high latency, the TCP ACK takes a long time to arrive, and because of the small window size the sender is waiting for the ACK instead of sending more data.

• Between the two routers the bandwidth was fine because the maximum window size was large (16MB) and the size scaled up
• Between the NAS and its local router the latency was small so the TCP window size didn’t matter.
• When sending data from the NAS to the remote router, the NAS kept waiting for TCP ACKs to arrive, and underused the available bandwidth.

I changed the max TCP window size on the NAS to match the routers’ 16MB:

sudo sysctl -w net.core.rmem_max=16777216  
sudo sysctl -w net.core.wmem_max=16777216

And changed iperf command to use the -w 8M flag to send data with a 8MB TCP window size. I found the 8M value with trial-and-error, I’m sure it could be optimized further but 190Mbit/s is good enough for this connection.

They are duplex.

Sry for the giant wall of text. There is a single-sentence tldr and a picture though, I edited the post to make the tldr more visible.