So after reading various issues here and there I was able to get onlyoffice work with the onlyoffice document_server app (instead of using the official onlyoffice document_server running in a dedicated container): AFAIU the problem is that the document server app prepared by Nextcloud team doesn’t support the libc provided by alpine (musl), it requires a glibc. It looks like they copied some binaries (at least x2t) from the official onlyoffice document server and those binaries need a glibc.
And as I’m following the example from the nextcloud/docker repository I was using the alpine variant…
I’ve now switched to non-alpine variants and was able to open documents 
This comment was the key thing:
To summarize I did this:
## Stop using alpine-version for nextcloud containers
vim docker-compose.yml
:%s/-alpine//gc
:wq
docker-compose pull && docker-compose build --pull
docker-compose up -d
# be sure to pick the correct container
docker exec mynextcloud_app_1 chown -R www-data:root /var/www
docker-compose down && docker-compose up -d
Before this, while debugging, I also uninstalled the onlyoffice and document_server apps, restarted my deployment, cleaned the previous onlyoffice configuration options set in /var/lib/docker/volumes/mynextcloud_nextcloud/_data/config/config.php, installed first the onlyoffice app then the document_server app and kept the default values.
My config.php also contains so that the proper values have been discovered in the office app:
'overwrite.cli.url' => 'https://my.host.fqdn/',
'overwriteprotocol' => 'https',
The onlyoffice settings are thus:
- Document Editing Service address: https://my.host.fqdn/apps/documentserver_community/
- Document Editing Service address for internal requests from the server: https://my.host.fqdn/apps/documentserver_community/
- Server address for internal requests from the Document Editing Service: https://my.host.fqdn/
Thanks for all the work on Nextcloud, really appreciate being able to deploy such a platform!