Nextcloud Office opening simple documents extremely slow

I am fighting with Nextcloud Office - maybe it is only big misunderstanding.

I have Nextcloud installed on my webserver which is almost dedicated for it

Gentoo Linux, apache-2.4.54, mysql-8.0.27
CPU i3-12100 @ 3.3GHz
32GiB RAM
NVME SSDs
PHP 8.0.24 with memory limit: 512 MB (should be more?)
Nextcloud version: Nextcloud Hub 3 25.0.2
latest Firefox as client webbrowser

Everything is reasonable fast except Nextcloud Office. Whenever i am tryin open any ODS or ODT document it takes minutes.

I do not want use Docker and i cannot dedicate another server only for Collabora Office.

I would like have possibility open ODT/ODS only for occasional use in webbrowser. Nothing more.

Question is - how can i diagnose where could be problem? Htop doesn’t show any significant CPU load (less than 2% on some cores), network traffic is very minimal, almost nothing is read or written on SSD. Even browser on client PC doesn’t generate significant CPU load.

It seems to me - Nexcloud Office is waiting for something. But i have no idea why and for what.

In processes i can see this:

1338 ?        S      0:00 bash -c ( /home/eventarenanextcloud/www/apps/richdocumentscode/collabora/Collabora_Online.AppImage --pidfile=/tmp/coolwsd.pid || /home/eventarenanextcloud/www/apps/richdocumentscode/collabora/Collabora_Online.AppImage --appimage-extract-and-run --pidfile=/tmp/coolwsd.pid) >/dev/null & disown
 1339 ?        S      0:00  \_ /bin/bash /tmp/.mount_CollabrYmtzq/AppRun --pidfile=/tmp/coolwsd.pid
 1350 ?        Sl   162:05      \_ coolwsd --config-file=/tmp/.mount_CollabrYmtzq/etc/coolwsd/coolwsd.xml --disable-cool-user-checking --port=9983 --lo-template-path=/tmp/.mount_CollabrYmtzq/opt/collaboraoffice --o:sys_template_path=/tmp/.mount_CollabrYmtzq/ --o:security.capabilities=false --o:security.seccomp=false --o:child_root_path=/tmp/coolwsd.mEaMN1nbWc/jails --o:file_server_root_path=/tmp/.mount_CollabrYmtzq/usr/share/coolwsd --o:ssl.enable=false --o:net.proxy_prefix=true --o:memproportion=50 --o:logging.file[@enable]=true --o:logging.file.property[0][@name]=path --o:logging.file.property[0]=/tmp/coolwsd.mEaMN1nbWc/coolwsd.log --o:welcome.enable=true --o:mount_jail_tree=false --o:user_interface.mode=default --o:allowed_languages=de_DE el en_GB en_US es_ES fr_FR hu it nl pt_BR pt_PT ru --pidfile=/tmp/coolwsd.pid
 1352 ?        S      0:20          \_ /tmp/.mount_CollabrYmtzq/usr/bin/coolforkit --systemplate=/tmp/.mount_CollabrYmtzq/ --lotemplate=/tmp/.mount_CollabrYmtzq/opt/collaboraoffice --childroot=/tmp/coolwsd.mEaMN1nbWc/jails/ --clientport=9983 --masterport=coolwsd-ajajT62p --rlimits=limit_virt_mem_mb:0;limit_stack_mem_kb:8000;limit_file_size_mb:0;limit_num_open_files:0 --version --nocaps --noseccomp --ui=default --disable-cool-user-checking
26798 ?        Sl     0:00              \_ /tmp/.mount_CollabrYmtzq/usr/bin/coolforkit --systemplate=/tmp/.mount_CollabrYmtzq/ --lotemplate=/tmp/.mount_CollabrYmtzq/opt/collaboraoffice --childroot=/tmp/coolwsd.mEaMN1nbWc/jails/ --clientport=9983 --masterport=coolwsd-ajajT62p --rlimits=limit_virt_mem_mb:0;limit_stack_mem_kb:8000;limit_file_size_mb:0;limit_num_open_files:0 --version --nocaps --noseccomp --ui=default --disable-cool-user-checking
28269 ?        Sl     0:01              \_ /tmp/.mount_CollabrYmtzq/usr/bin/coolforkit --systemplate=/tmp/.mount_CollabrYmtzq/ --lotemplate=/tmp/.mount_CollabrYmtzq/opt/collaboraoffice --childroot=/tmp/coolwsd.mEaMN1nbWc/jails/ --clientport=9983 --masterport=coolwsd-ajajT62p --rlimits=limit_virt_mem_mb:0;limit_stack_mem_kb:8000;limit_file_size_mb:0;limit_num_open_files:0 --version --nocaps --noseccomp --ui=default --disable-cool-user-checking
28851 ?        S      0:00              \_ /tmp/.mount_CollabrYmtzq/usr/bin/coolforkit --systemplate=/tmp/.mount_CollabrYmtzq/ --lotemplate=/tmp/.mount_CollabrYmtzq/opt/collaboraoffice --childroot=/tmp/coolwsd.mEaMN1nbWc/jails/ --clientport=9983 --masterport=coolwsd-ajajT62p --rlimits=limit_virt_mem_mb:0;limit_stack_mem_kb:8000;limit_file_size_mb:0;limit_num_open_files:0 --version --nocaps --noseccomp --ui=default --disable-cool-user-checking
 1344 ?        Ssl    0:05 /home/eventarenanextcloud/www/apps/richdocumentscode/collabora/Collabora_Online.AppImage --pidfile=/tmp/coolwsd.pid

Several times Nextcloud frozen so only option was restart of Apache.

I even found folder /tmp/coolwsd.mEaMN1nbWc where is subfolder jails and 6.5GiB big coolwsd.log

At the end of it are many messages like this:

wsd-01350-28377 2023-01-16 17:30:29.934910 +0100 [ docbroker_02a ] WRN  ToClient-344226: Tracker tileID 0:34560:15360:3840:3840:0 was dropped because of time out (9823ms). Tileprocessed mess
age did not arrive in time.| wsd/ClientSession.cpp:2037

All ODT/ODS documents, which i am experimenting with, are small and very simple. For example one page of text without any pictures or anything else. After restart of Apache document can be opened, it is still slow, mostly it tooks several seconds. But often i cannot open same document second time after it was closed. I am not sure, but maybe it is worse when i am trying open more documents in more browser tabs.

I just found something very wierd.

Nextcloud, after first user is logged in, creates in /tmp some folders and files, but one of the files is ?.mount_CollabVhIuno (last 6 characters are random). This file cannot be deleted, doesn’t dissapear even if Apache2 is stopped, cannot be unlocked…

I am very sure it is not bad storage (no bad badblocks, no io error, everything else is running fine), but it looks more like compatibility issue with filesystem. After reboot is /tmp purged, but this file will be created by Nextcloud again, and exactly with same problem.

I am using BTRFS in RAID mode (only mirror). Could it be problem?

(for now it is running with Linux kernel Linux version 5.15.67… i will update it.)

I don’t know if it have any connection to the problem with Nextcloud Office, but it is definitely bad behaviour.

I just added another hard drive, /tmp is now on EXT4 fs. And Nexcloud behaves exactly same - still creates wierd file with questionmark…

But! ODS and ODT files are now opening much faster, even if hard drive mounted to /tmp is several times slower rotating magnetic drive, not SSD.

For comparison:

hdparm -tT /dev/nvme0n1

/dev/nvme0n1:
Timing cached reads: 34792 MB in 2.00 seconds = 17419.64 MB/sec
Timing buffered disk reads: 9172 MB in 3.00 seconds = 3056.71 MB/sec

(on this drive is space for linux, nextcloud files and application)

hdparm -tT /dev/sdd

/dev/sdd:
Timing cached reads: 34612 MB in 2.00 seconds = 17329.09 MB/sec
Timing buffered disk reads: 388 MB in 3.00 seconds = 129.24 MB/sec

(on this drive is /tmp)

So Nexcloud behaves differently with different filesystems on /tmp. For some reason have problem with much modern BTRFS in comparison with EXT4.

Why?

I definitely recommend more…
unfortunately nothing useful else to add: my (all docker) installation with 2GB PHP memory limit runs Collabora very well on much older hardware…