I guess you may need better CPU.
I am no expert here, but I have similar issue, though in my case, my CPU is much older and weaker but fundamental behavior of this issue is similar.
I have more users than your setup and rarely I am using more than 1 GB of system ram, even when manually setting value to allow higher RAM allocation for php memory or such.
Mainly it comes down to CPU
Here is my issue → High CPU Usage NextCloud 22.2.0 (snap) on Ubuntu 20 LTS
By the way, you didn’t mention the NextCloud version of your setup. Someone in below post mentions that 22.2.3 is out and this latest version is suppose to solve this high CPU load issue. → NC 22.2.1 performance warning!
I haven’t received the update yet, waiting for snap version to come up with the update.
Lastly, I am no expert, so please wait for other expert users to reply.
Thanks.
Update :- My Bad, I didn’t notice, yes, I have received the said 22.2.3 update
Although the web interface performance is now much better and overall responsiveness has gone up. Snap by default didn’t have GZip compression, I have turned that on.
However, my Intel Pentium N3530 still feels inadequate with multiple simultaneous active users. CPU usage still shoots up with php-fm and mysql when multiple active users and simultaneously transferring large files.
Thanks.