Dedicated collabora server slow spreadsheets with nextcloud

I have a nextcloud vm with the following specs:
Ubuntu 20.04 - NC version 22.2.5 - 4vcpus 4gb ram
I have a separate dedicated Collabora VM:
Debian 11 - Coolwsd version 21.11.2 - 2vcpus 2gb ram

Both VMs are located behind an NGINX reverse proxy.
The Nextcloud instance runs very smoothly and is very responsive.

When opening an empty spreadsheet, scrolling is very laggy. It takes almost a full second for the spreadsheet to move after scrolling.

CPU utilization on the collabora vm only peaks at 40% usage when opening the empty spreadsheet (seen through top).
Looking at the admin console statistics, the cpu usage seems to spike to 100% when opening an empty spreadsheet, and stay there when using a large spreadsheet.
Nextcloud cpu utilziation is barely a couple percent per process.

I have tried increasing php memory on nextcloud to a gb, but that didnt do anything.

Why is an empty spreadsheet lagging so much?
What configs can I change to hopefully increase performance?

Collabora logs some errors when the spreadsheet is open:
Error while handling poll at 1 in websrv_poll: #24 SSL BIO error: socket closed unexpectedly (-1)| net/Socket.cpp:465

That log is repeated a lot.
No error logs for nextcloud.