tough stuff… I have no idea how to spot the issue easily. maybe checking tcpdump of the systems one by one gives you an idea e.g. if the traffic leaves you network and comes back. luckily you can easily test with curl so you only need to analyze short network traces.
I made a drawing to better understand the architecture - let me know if this is not right.
according to you all the way from client to both services work.
but the two connections from Nextcloud and Collabora are broken. As we know DNS is right the problem must be routing or firewall. You can check the routing into the internet if you do curl www.google.com or any other external resource - if this works you know routing and firewall from your local system to the internet works. if internet access works the most promising failure spot is the VPS - here the request coming from internal #wireguard tunnel/interface must leave towards the internet and the come back and flow all the way back to the systems… this might be an issue where the logic could fail in many places - the system might fail connecting to itself or route the traffic from wireguard → internet → internet → wireguard.
PS: I have long lived plans to check if docker internal dnsalias works to shortcut the traffic one the reverse proxy. once I have time to test I will update. This is definitely the most effective and performant solution for such setups.