I found the fix for me.
Since my docker-instance is running on plain http and running on port 80(within the docker container) and the reverse-proxy is handling the SSL, I had to specify the following 2 config options for the reverse proxy:
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
The full apache2 virtual host section is:
<VirtualHost *:443>
ServerName server.domain.tld
SSLCertificateFile /etc/letsencrypt/live/server.domain.tld/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/server.domain.tld/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/server.domain.tld/chain.pem
ErrorLog ${APACHE_LOG_DIR}/onlyoffice_error.log
CustomLog ${APACHE_LOG_DIR}/onlyoffice_access.log combined
ServerAdmin webmaster@localhost
DocumentRoot /var/www/empty
<Location />
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
ProxyPass http://localhost:10080/
ProxyPassReverse http://localhost:10080
</Location>
</VirtualHost>