Preview generation brings server to a halt (extreme CPU and RAM usage) [Docker]

Nextcloud version: official docker nextcloud:apache image (24.0.5)
Operating system and version: Docker 20.10.17 on Debian 11
Apache or nginx version: the one included in the official image
PHP version: the one included in the official image

The issue you are facing:
Image preview generation brings Nextcloud to a halt, maxing out CPU and RAM allowances, swapping to disk and getting killed by the kernel OOM killer.

I am hosting a private Nextcloud instance on a VPS with 4 vCPUs (AMD EPYC), 8GB RAM and SSD storage using Docker (separate containers using official images for: Nextcloud, PostgreSQL, Redis, OnlyOffice, nginx as reverse proxy in front of Nextcloud) and with no other resource intensive applications.

I am using Nextcloud to upload my photos and share them with my friends.
The first time I uploaded around 2.5GBs of ~500 JPEGs straight from my camera and everything went smoothly until I shared the folder to two friends of mine: for about half a minute the server just froze.
That half a minute was probably when they first accessed the shared folder and previews had to be generated on demand.

I had netdata monitoring on and it reported hundreds of httpd processes inside the Nextcloud container using up all the available CPU and RAM.
So I went in my compose file and set up CPU and RAM limits to avoid my whole server crashing because of Nextcloud, but that didn’t fix the root cause: httpd still keeps spamming hundreds of processes, pinning the CPU at 100% of its limit and the RAM too, which means that it also starts swapping to disk thus invoking the OOM killer inside the container and just returning “504 gateway timeout”.

Additionally, I can see in the Nextcloud logs that it exceeded the max number of allowed connections by PostgreSQL.

I also tried setting up Imaginary in a separate container, but that just worsens the problem, because now I have both Nextcloud and Imaginary containers using 100% of their allowed CPU and RAM…

Is there some way to limit the number of concurrent preview generation requests?
I want to have previews but Nextcloud needs to be limited and not spam processes like it’s running on a supercomputer…

Is this the first time you’ve seen this error?
No, it happens every time I load a bunch of photos and share them.

Steps to replicate it:

  1. load a bunch of photos on a new Nextcloud instance hosted on a sufficiently beefy VPS
  2. open the Photos app
  3. look at your server load spike up into double digits

The output of your Nextcloud log in Admin > Logging:
Little snippet of the only thing that is logged when this freezing happens

{"reqId":"pOpjAxVrk6Jqm47V8ea3","level":3,"time":"2022-09-22T16:31:12+00:00","remoteAddr":"151.68.147.156","user":"--","app":"core","method":"GET","url":"/apps/files_sharing/publicpreview/Hn3MEtAaRwxfQip?fileId=8880&file=%2FDSC00867.JPG&c=34709eb61e78236e4541847cc958d861&x=750&y=750","message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","userAgent":"Mozilla/5.0 (Linux; Android 12; CPH2207) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Mobile Safari/537.36","version":"24.0.5.1","exception":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","Code":7,"Trace":[{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1519,"function":"connect","class":"OC\\DB\\Connection","type":"->","args":[]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1041,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->","args":[]},{"file":"/var/www/html/lib/private/DB/Connection.php","line":261,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[],null]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":345,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[]]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":411,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":176,"function":"loadConfigValues","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":367,"function":"getApps","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/legacy/OC_App.php","line":983,"function":"getValues","class":"OC\\AppConfig","type":"->","args":[false,"installed_version"]},{"file":"/var/www/html/lib/private/Server.php","line":713,"function":"getAppVersions","class":"OC_App","type":"::","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ICacheFactory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/Server.php","line":1105,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\Lock\\ILockingProvider"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\Lock\\ILockingProvider",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\Lock\\ILockingProvider"]},{"file":"/var/www/html/lib/private/Server.php","line":2082,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\Lock\\ILockingProvider"]},{"file":"/var/www/html/lib/private/Files/View.php","line":122,"function":"getLockingProvider","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/html/lib/private/Server.php","line":454,"function":"__construct","class":"OC\\Files\\View","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Files\\Node\\HookConnector"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Files\\Node\\HookConnector",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Files\\Node\\HookConnector"]},{"file":"/var/www/html/lib/private/Server.php","line":1445,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Files\\Node\\HookConnector"]},{"file":"/var/www/html/lib/base.php","line":602,"function":"boot","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/html/lib/base.php","line":1111,"function":"init","class":"OC","type":"::","args":[]},{"file":"/var/www/html/index.php","line":34,"args":["/var/www/html/lib/base.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/DB/Connection.php","Line":139,"CustomMessage":"--"}}
{"reqId":"pOpjAxVrk6Jqm47V8ea3","level":3,"time":"2022-09-22T16:31:12+00:00","remoteAddr":"151.68.147.156","user":"--","app":"core","method":"GET","url":"/apps/files_sharing/publicpreview/Hn3MEtAaRwxfQip?fileId=8880&file=%2FDSC00867.JPG&c=34709eb61e78236e4541847cc958d861&x=750&y=750","message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","userAgent":"Mozilla/5.0 (Linux; Android 12; CPH2207) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Mobile Safari/537.36","version":"24.0.5.1","exception":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","Code":7,"Trace":[{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1519,"function":"connect","class":"OC\\DB\\Connection","type":"->","args":[]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1041,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->","args":[]},{"file":"/var/www/html/lib/private/DB/Connection.php","line":261,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[],null]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":345,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[]]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":411,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":176,"function":"loadConfigValues","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":367,"function":"getApps","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/legacy/OC_App.php","line":983,"function":"getValues","class":"OC\\AppConfig","type":"->","args":[false,"installed_version"]},{"file":"/var/www/html/lib/private/Server.php","line":713,"function":"getAppVersions","class":"OC_App","type":"::","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ICacheFactory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":89,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ICacheFactory",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":108,"function":"array_map","args":[{"__class__":"Closure"},[{"__class__":"ReflectionParameter","name":"config"},{"__class__":"ReflectionParameter","name":"oldDispatcher"},"*** sensitive parameters replaced ***",{"__class__":"ReflectionParameter","name":"eventDispatcher"}]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":116,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[{"__class__":"ReflectionClass","name":"OC\\User\\Manager"}]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":133,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":161,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/Server.php","line":540,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/Server.php","line":1392,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/Server.php","line":1275,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":89,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":108,"function":"array_map","args":[{"__class__":"Closure"},[{"__class__":"ReflectionParameter","name":"tokenGenerator"},"*** sensitive parameters replaced ***"]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":116,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[{"__class__":"ReflectionClass","name":"OC\\Security\\CSRF\\CsrfTokenManager"}]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":133,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":161,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/Server.php","line":1055,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/Server.php","line":1562,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/legacy/OC_Template.php","line":323,"function":"getRequest","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/html/index.php","line":74,"function":"printExceptionErrorPage","class":"OC_Template","type":"::","args":[{"__class__":"Doctrine\\DBAL\\Exception"},500]}],"File":"/var/www/html/lib/private/DB/Connection.php","Line":139,"CustomMessage":"--"}}
{"reqId":"eWJphVYZBToRtRRkohr3","level":3,"time":"2022-09-22T16:31:12+00:00","remoteAddr":"151.68.147.156","user":"--","app":"core","method":"GET","url":"/apps/files_sharing/publicpreview/Hn3MEtAaRwxfQip?fileId=8790&file=%2FDSC00858.JPG&c=fe58c14472b4d54c592450a5921260e4&x=750&y=750","message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","userAgent":"Mozilla/5.0 (Linux; Android 12; CPH2207) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Mobile Safari/537.36","version":"24.0.5.1","exception":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08006] [7] FATAL:  remaining connection slots are reserved for non-replication superuser connections","Code":7,"Trace":[{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1519,"function":"connect","class":"OC\\DB\\Connection","type":"->","args":[]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Connection.php","line":1041,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->","args":[]},{"file":"/var/www/html/lib/private/DB/Connection.php","line":261,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[],null]},{"file":"/var/www/html/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":345,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->","args":["SELECT * FROM \"oc_appconfig\"",[],[]]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":411,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":176,"function":"loadConfigValues","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppConfig.php","line":367,"function":"getApps","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/html/lib/private/legacy/OC_App.php","line":983,"function":"getValues","class":"OC\\AppConfig","type":"->","args":[false,"installed_version"]},{"file":"/var/www/html/lib/private/Server.php","line":713,"function":"getAppVersions","class":"OC_App","type":"::","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ICacheFactory"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ICacheFactory",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":89,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ICacheFactory",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":108,"function":"array_map","args":[{"__class__":"Closure"},[{"__class__":"ReflectionParameter","name":"config"},{"__class__":"ReflectionParameter","name":"oldDispatcher"},"*** sensitive parameters replaced ***",{"__class__":"ReflectionParameter","name":"eventDispatcher"}]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":116,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[{"__class__":"ReflectionClass","name":"OC\\User\\Manager"}]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":133,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":161,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Manager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/Server.php","line":540,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":184,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\User\\Session"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/Server.php","line":1392,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IUserSession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/Server.php","line":1275,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\ISession"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":89,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\TokenStorage\\SessionStorage",true]},{"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":108,"function":"array_map","args":[{"__class__":"Closure"},[{"__class__":"ReflectionParameter","name":"tokenGenerator"},"*** sensitive parameters replaced ***"]]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":116,"function":"buildClass","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":[{"__class__":"ReflectionClass","name":"OC\\Security\\CSRF\\CsrfTokenManager"}]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":133,"function":"resolve","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":161,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager",true]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/Server.php","line":1055,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Security\\CSRF\\CsrfTokenManager"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":162,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php","line":122,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":129,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/ServerContainer.php","line":136,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest",false]},{"file":"/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php","line":57,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/Server.php","line":1562,"function":"get","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\IRequest"]},{"file":"/var/www/html/lib/private/legacy/OC_Template.php","line":323,"function":"getRequest","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/html/index.php","line":74,"function":"printExceptionErrorPage","class":"OC_Template","type":"::","args":[{"__class__":"Doctrine\\DBAL\\Exception"},500]}],"File":"/var/www/html/lib/private/DB/Connection.php","Line":139,"CustomMessage":"--"}}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
Pastebin

The output of your Apache/nginx/system log in /var/log/____:
Text file

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.
See above

1 Like

I have been doing some more testing, changing resource limits and forcing preview generation to go through Imaginary only but it makes no real difference: nextcloud/httpd still spams processes that fill up its RAM to the limit and starts swapping, then once imaginary reaches its RAM limit (4GB) it also starts swapping to disk, the CPU usage drops to zero because all the load is just iowait and the OOM-killer starts killing processes.

I even disabled swapping for the imaginary container and what happens is that once the RAM limit is reached, OOM-killer is invoked instantly and imaginary just returns to idle (while no previews were generated).

I don’t understand what’s wrong with Imaginary, their docs say that 2GB of RAM is enough for a “high-load HTTP service processing potentially large images or exposed to an eventual high concurrency”, but my instance has 4GB dedicated and exceeds that by just opening the Photos app once after uploading new photos.
Imaginary is configured with -concurrency 5, 4 times lower than suggested in their docs (20 requests/s).

The obvious solution would be to rate limit preview generation, but how?
I wonder what solution is given to paying customers that complain about this kind of performance degradation?

I investigated some more about Imaginary… turns out this project has been unmaintained for around two years.
Yet, I recall seeing at least one GitHub issue which presented a valid issue with the preview generation just get dismissed because Imaginary is just supposed to automagically fix it, which it doesn’t…

Alright so I ended up installing the preview generator app, configuring it as per this blog post and scheduling it every 10 minutes.
After spending 3 minutes regenerating all previews for my 3.5GB of photos, the preview folder is a mere 210MB and my server was happy at 40% CPU usage during the process (all user load, no iowait, with Imaginary too).

Can’t believe this issue has been known for so many years, yet developers have never bothered addressing it as the major issue it is, and worse yet just moved the blame to an unmaintained project, while also including the default Photos app that will just bring your server down!

2 Likes