I’ve found a fix! So your hint about packet corruption kept bugging me. I didn’t have a spare router on hand so I couldn’t test the theory, but today out of the blue, I suddenly had the idea to try flashing LEDE onto my router instead of using the stock firmware.
I am happy to report that this completely got rid of the issue!
I am not sure why this fixes it. Maybe the stock firmware has a bug or it can’t handle that high throughput?