I try it and have no delay before download starts. It took around 1 second before download of 800Mb file starts, I interrupt it then.
My setup is similar to yours, except HW: it is cubeitruck with tiny CPU.
Check your debugging page in browser, what is happening down there.
Also you can check logs in real time to see what happens, e.g.
sudo tail -f /var/log/apache2/access.log
sudo tail -f /var/log/apache2/error.log
sudo tail -f /var/log/php7.3-fpm.log
May be you have something strange in Apache2 config?