Nextcloud takes about 6 seconds until it loads a doc in collabora

Hey,
My setup
Ubuntu 20.04.4 LTS server, 16GB RAM, Intel(R) Core™ i5-3470 CPU @ 3.20GHz
I run my nextcloud behind swag reverse proxy
I use nextcloud official container along with mariadb, redis, and collabora containers. Images:

ghcr.io/linuxserver/swag
redis
mariadb:latest
nextcloud:apache
collabora/code

My custom_apps and config dirs, as well as user data resides on a ZFS HDD mount and everything else is on an ssd (including mariadb sql). I tried moving the config and custom_apps dir to the ssd but it had no effect on the latency.

nextcloud version is 23.0.2
My issue is that loading files to edit with collabora can take up to 10 seconds and usually takes about 6 seconds.
I had another VM with similar setup (All the data is on vnme ssd in the VM) where files are loaded fast. I tried to set the nextcloud to use the collabora instance of the VM but it still loaded slow.

Some logs from the moment I try to open a file:
Nextcloud

<IP> - - [13/Mar/2022:07:34:06 +0000] "PROPFIND /remote.php/dav/files/manager/cc.xlsx HTTP/1.1" 207 1337 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
<IP> - - [13/Mar/2022:07:34:06 +0000] "GET /index.php/apps/richdocuments/index?fileId=17031&requesttoken=LqnE%2BjJDjCH6couYBemoGg5sNMnPn8p1NRK5AneRErc%3D%3AXPO1nEgZxWWVRLzTZIODQ2NYZYekz7leeXXzbBjJWM4%3D&path=%2Fcc.xlsx HTTP/1.1" 200 8150 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
<IP> - - [13/Mar/2022:07:34:06 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 763 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
<IP> - - [13/Mar/2022:07:34:08 +0000] "GET /index.php/apps/photos/service-worker.js HTTP/1.1" 200 6366 "https://<SERVER-DOMAIN>/index.php/apps/photos/service-worker.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
<IP> - - [13/Mar/2022:07:34:11 +0000] "GET /index.php/apps/richdocuments/wopi/files/17031_oc6skyzf515d?access_token=I9YTNmvdjT3jZEB1doox5fMDQUIDx6tq&access_token_ttl=0 HTTP/1.1" 200 2048 "-" "COOLWSD HTTP Agent 21.11.2.4"
<IP> - - [13/Mar/2022:07:34:15 +0000] "GET /index.php/apps/richdocuments/wopi/files/17031_oc6skyzf515d/contents?access_token=I9YTNmvdjT3jZEB1doox5fMDQUIDx6tq&access_token_ttl=0 HTTP/1.1" 200 6572 "-" "COOLWSD HTTP Agent 21.11.2.4"

Collabora

wsd-00001-00275 2022-03-13 07:34:11.026283 +0000 [ docbroker_00a ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:722
wsd-00001-00275 2022-03-13 07:34:15.120159 +0000 [ docbroker_00a ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:722
wsd-00001-00048 2022-03-13 07:34:15.579429 +0000 [ websrv_poll ] ERR  FileServerRequestHandler: File not found: Invalid URI request: [/browser/e8ac597/nextcloud/;%7D].| wsd/FileServer.cpp:719

So you can see it takes about 5 seconds until the request even reaches collabora.

subdomain configuration
Nextcloud

server {                                                                                                                                                      
                                                                                                                                                              
    listen 443 ssl http2;                                                                                                                                     
    listen [::]:443 ssl http2;                                                                                                                                
    include /config/nginx/ssl.conf;                                                                                                                           
    server_name nextcloud.<DOMAIN>;                                                                                                                        
    
    include /config/nginx/authelia-server.conf;                                                                                                               
    
    # Specfic server headers                                                                                                                                  
        client_max_body_size 0;                                                                                                                               
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains";                                                                           
                                                                                                                                                              
    add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";                                                                                        
        location / {                                                                                                                                          
        include /config/nginx/proxy.conf;                                                                                                                     
        include /config/nginx/resolver.conf;                                                                                                                  
                                                                                                                                                              
                include /config/nginx/authelia-location.conf;                                                                                                 
                                                                                                                                                              
        # Specfic location headers                                                                                                                            
                proxy_max_temp_file_size 2048m;                                                                                                               
                
                set $upstream_app nextcloud;                                                                                                                  
                set $upstream_port 80;                                                                                                                        
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;                                                                                            
    }                                                                                                                                                         
    
    location /.well-known/carddav {                                                                                                                           
        return 301 $scheme://$host/remote.php/dav;                                                                                                            
    } 
    
    location /.well-known/caldav {                                                                                                                            
        return 301 $scheme://$host/remote.php/dav;
    }                                                                                                                                                         
}

Collabora

server {

    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    include /config/nginx/ssl.conf;
    server_name collabora.<DOMAIN>;

    include /config/nginx/authelia-server.conf;

    # Specfic server headers
        client_max_body_size 0;

    add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
    location / {
        include /config/nginx/proxy.conf;                                                                                                                     
        include /config/nginx/resolver.conf;                                                                                                                  
                                                                                                                                                              
                include /config/nginx/authelia-location.conf;                                                                                                 
                                                                                                                                                              
        # Specfic location headers                                                                                                                            
                
                set $upstream_app collabora;
                set $upstream_port 9980;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }   
    
    # static files
    location ^~ /browser {
      proxy_set_header Host $http_host;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }                                                                                                                                                         

    # WOPI discovery URL
    location ^~ /hosting/discovery {
      proxy_set_header Host $http_host;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }

    # Capabilities
    location ^~ /hosting/capabilities {
      proxy_set_header Host $http_host;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }

    # main websocket
    location ~ ^/cool/(.*)/ws$ {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $http_host;
      proxy_read_timeout 36000s;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }

    # download, presentation and image upload
    location ~ ^/(c|l)ool {
      proxy_set_header Host $http_host;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }

    # Admin Console websocket
    location ^~ /cool/adminws {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $http_host;
      proxy_read_timeout 36000s;
      set $upstream_app collabora;
    set $upstream_port 9980;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }
}

I tried disabling Authelia but it had no effect (Currently I bypass those subdomains in Authelia).

I run the same scenario again with debug logs:
Nextcloud

{"reqId":"RyJJqPev8k28VszFghEJ","level":0,"time":"2022-03-13T20:25:25+00:00","remoteAddr":"<CLIENT_IP>","user":"<USER>","app":"scss_cacher","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=17031&requesttoken=A8RNFUdnCFsMLdlLKZcbSBgOcEzA8Jz6ywOGkD5U12A%3D%3AZJM0QQgpOxM%2BTLgjTdJYIjMhIBmNuK3L8zHtqWdlmCY%3D&path=%2Fcc.xlsx","message":"SCSSCacher::process ordinary check follows","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36","version":"23.0.2.1"}
{"reqId":"RyJJqPev8k28VszFghEJ","level":0,"time":"2022-03-13T20:25:25+00:00","remoteAddr":"<CLIENT_IP>","user":"<USER>","app":"scss_cacher","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=17031&requesttoken=A8RNFUdnCFsMLdlLKZcbSBgOcEzA8Jz6ywOGkD5U12A%3D%3AZJM0QQgpOxM%2BTLgjTdJYIjMhIBmNuK3L8zHtqWdlmCY%3D&path=%2Fcc.xlsx","message":"SCSSCacher::process ordinary check follows","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36","version":"23.0.2.1"}
{"reqId":"ZY1os9hbtra8O7F3NoqP","level":0,"time":"2022-03-13T20:25:30+00:00","remoteAddr":"<CLIENT_IP>","user":"--","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/wopi/files/17031_oc6skyzf515d?access_token=aUuslAdWwD3ejS3G0VT6NC4mCpSAONeU&access_token_ttl=0","message":"Loaded WOPI Token record: {\"id\":\"72\",\"owner_uid\":\"<USER>\",\"editor_uid\":\"<USER>\",\"guest_displayname\":null,\"fileid\":\"17031\",\"version\":\"0\",\"canwrite\":\"1\",\"server_host\":\"https://nextcloud.<DOMAIN>/\",\"token\":\"aUuslAdWwD3ejS3G0VT6NC4mCpSAONeU\",\"expiry\":\"1647204925\",\"template_destination\":\"0\",\"template_id\":\"0\",\"hide_download\":\"0\",\"direct\":\"0\",\"remote_server\":\"\",\"remote_server_token\":\"\",\"share\":null,\"token_type\":\"0\"}.","userAgent":"COOLWSD HTTP Agent 21.11.2.4","version":"23.0.2.1"}
{"reqId":"iqOdRUv0J1RNkpw5VDpa","level":0,"time":"2022-03-13T20:25:34+00:00","remoteAddr":"<CLIENT_IP>","user":"--","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/wopi/files/17031_oc6skyzf515d/contents?access_token=aUuslAdWwD3ejS3G0VT6NC4mCpSAONeU&access_token_ttl=0","message":"Loaded WOPI Token record: {\"id\":\"72\",\"owner_uid\":\"<USER>\",\"editor_uid\":\"<USER>\",\"guest_displayname\":null,\"fileid\":\"17031\",\"version\":\"0\",\"canwrite\":\"1\",\"server_host\":\"https://nextcloud.<DOMAIN>/\",\"token\":\"aUuslAdWwD3ejS3G0VT6NC4mCpSAONeU\",\"expiry\":\"1647204925\",\"template_destination\":\"0\",\"template_id\":\"0\",\"hide_download\":\"0\",\"direct\":\"0\",\"remote_server\":\"\",\"remote_server_token\":\"\",\"share\":null,\"token_type\":\"0\"}.","userAgent":"COOLWSD HTTP Agent 21.11.2.4","version":"23.0.2.1"}

Collabora

wsd-00001-00474 2022-03-13 20:25:30.432454 +0000 [ docbroker_012 ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:722
wsd-00001-00474 2022-03-13 20:25:34.524648 +0000 [ docbroker_012 ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:722
wsd-00001-00048 2022-03-13 20:25:34.978198 +0000 [ websrv_poll ] ERR  FileServerRequestHandler: File not found: Invalid URI request: [/browser/e8ac597/nextcloud/;%7D].| wsd/FileServer.cpp:719
wsd-00001-00474 2022-03-13 20:25:35.499869 +0000 [ docbroker_012 ] WRN  Dropping empty tile response: tile: nviewid=0 part=0 width=256 height=256 tileposx=0 tileposy=0 tilewidth=3840 tileheight=3840 oldwid=2 wid=2 ver=69| wsd/DocumentBroker.cpp:2852

So if I understand correctly, WOPI token load took about 5 seconds, is the normal?

Also, I tried to move the appdata directory to the ssd, it had no effect.
To me it looks like an issue with nextcloud and not with collabora. Any ideas?

Update:
I tried using demo servers instead of the collabora container and I saw some improvement in the response time. I don’t fully understand why that is since the same setup on a virtual machine on a stronger PC works perfectly. It could be due to old HW

I have a same problem