I finally know the root cause of this issue and how to fix it.
When answering the questions from @jtr about the underlying filesystem it came to my mind that the issue might have something to do with the virtiofs which I am using to import the /var/www directory in the nextcloud VM. And that it is, indeed.
The “Too many open files” error is not coming from the nextcloud VM. Thats the reason why it did not make any difference when I changed ulimit values in the nextcloud VM.
The “Too many open files” error is coming from the host. The host is reporting this error to the VM and the VM is killing the upgrade process accordingly with the proper message.
When running the upgrade process I see the list of open files on the host growing well above 1 million before it errors out. The reason is the virtiofsd on the host which is caching all the file requests from the VM by default.
The solution is, to turn off caching for the shared folders. This is easily done in the XML configuration of the VM by adding <cache mode="none"/> to the XML configuration. Like this:
<binary path="/usr/lib/virtiofsd">
<cache mode="none"/>
</binary>
That is it. When I did this, the upgrade process was running fine until the end, including backup - although the “backup” and “delete old files” steps took roughly 90 min each to complete.
I hope other people with nextcloud on kvm find this info helpful.