Nextcloud Office always maxes out PHP processes

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 26.0.1
Operating system and version (eg, Ubuntu 20.04): Ununtu 20.04.05 LTS
Apache or nginx version (eg, Apache 2.4.25): Apache2 2.2.41
PHP version (eg, 7.4): 8.1.18

The issue you are facing:
With Nextcloud Office and Collabora online - Built-in CODE server active as soon as Files is access PHP starts to spawn new processes, which is good. But they never stop, they just sit there as active untill max_children is reached at which point everything else stops. Each action within with Files, such as open a directory, open a file etc causes more processes to spawn.
Other apps don’t cause a problem.
If Nextcloud Office and CODE are disabled then browsing folders and opening .md files causes no problems, and office docs are downloaded so again no problems. It seems Office/CODE is the issue - and I’m seeing this on two separate systems - the Test one I created for a community group and the Dev box I created so I can poke around at this without disrupting their exploration of other features.

PHP real-time status reports all of the stuck processes have Request URI of one of two forms:
/index.php?fileId=[num]&x=250&y=250
/index.php?file=[num]&c=[32 hex digit number]&x=313&y=313%forceIcon=0&a=1

On current test run processes have served between 2-130 requests before getting stuck. Leaving it to run overnight made no difference - the processes don’t time out.

Host has 2GB RAM, less than half in use, swap not touched. max_children currently set to 32, I’ve had issues even with it set to 60 - which is just stupidly high. Using ondemand makes no difference as the processes are seen as active so never get reaped.

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. Enable Office & Code
  2. Open Files folders and office and .md docs
  3. Watch the active process count increase

The output of your Nextcloud log in Admin > Logging:
same as nextcloud.log below

[PHP] Error: Trying to access array offset on value of type null at /srv/nextcloud/www/apps/richdocuments/lib/WOPI/DiscoveryManager.php#132

GET /index.php/apps/richdocuments/index?fileId=10&requesttoken=2EqhkVw5Z%2BoJqkNRY7YQw36C7NPlfzFsk7EbXMGaI%2BY%3D%3AjAzExA4AI7M5nnIdGdBkiAnIhIHdNnYa4clWP5TcRLU%3D&path=%2FDocuments%2FWelcome%20to%20Nextcloud%20Hub.docx
from 192.168.42.64 by docadmin at 2023-05-14T19:02:24+00:00

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '2258',
  'passwordsalt' => 'andvinegar',
  'secret' => 'squirrel',
  'trusted_domains' => 
  array (
    0 => 'nextcloud-doc.public.example.com',
    ),
  'datadirectory' => '/srv/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '26.0.1.1',
  'overwrite.cli.url' => 'http://nextcloud-doc.public.example.com',
  'dbname' => 'dbdbname',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'dbusername',
  'dbpassword' => 'dbuserpasswd',
  'installed' => true,
  'default_phone_region' => 'GB',
  'memcache.local' => '\OC\Memcache\APCu',
);

The output of your Apache/nginx/system log in /var/log/____:

200 responses from phpstatus....

PASTE HERE


Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

{“reqId”:“xGe83Pw10udnIGZIaECZ”,“level”:3,“time”:“2023-05-14T19:02:24+00:00”,“remoteAddr”:“192.168.42.64”,“user”:“docadmin”,“app”:“PHP”,“method”:“GET”,“url”:“/index.php/apps/richdocuments/index?fileId=10&requesttoken=2EqhkVw5Z%2BoJqkNRY7YQw36C7NPlfzFsk7EbXMGaI%2BY%3D%3AjAzExA4AI7M5nnIdGdBkiAnIhIHdNnYa4clWP5TcRLU%3D&path=%2FDocuments%2FWelcome%20to%20Nextcloud%20Hub.docx”,“message”:“Trying to access array offset on value of type null at /srv/nextcloud/www/apps/richdocuments/lib/WOPI/DiscoveryManager.php#130”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”,“version”:“26.0.1.1”,“data”:{“app”:“PHP”}}
{“reqId”:“xGe83Pw10udnIGZIaECZ”,“level”:3,“time”:“2023-05-14T19:02:24+00:00”,“remoteAddr”:“192.168.42.64”,“user”:“docadmin”,“app”:“PHP”,“method”:“GET”,“url”:“/index.php/apps/richdocuments/index?fileId=10&requesttoken=2EqhkVw5Z%2BoJqkNRY7YQw36C7NPlfzFsk7EbXMGaI%2BY%3D%3AjAzExA4AI7M5nnIdGdBkiAnIhIHdNnYa4clWP5TcRLU%3D&path=%2FDocuments%2FWelcome%20to%20Nextcloud%20Hub.docx”,“message”:“Trying to access array offset on value of type null at /srv/nextcloud/www/apps/richdocuments/lib/WOPI/DiscoveryManager.php#131”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”,“version”:“26.0.1.1”,“data”:{“app”:“PHP”}}
{“reqId”:“xGe83Pw10udnIGZIaECZ”,“level”:3,“time”:“2023-05-14T19:02:24+00:00”,“remoteAddr”:“192.168.42.64”,“user”:“docadmin”,“app”:“PHP”,“method”:“GET”,“url”:“/index.php/apps/richdocuments/index?fileId=10&requesttoken=2EqhkVw5Z%2BoJqkNRY7YQw36C7NPlfzFsk7EbXMGaI%2BY%3D%3AjAzExA4AI7M5nnIdGdBkiAnIhIHdNnYa4clWP5TcRLU%3D&path=%2FDocuments%2FWelcome%20to%20Nextcloud%20Hub.docx”,“message”:“Trying to access array offset on value of type null at /srv/nextcloud/www/apps/richdocuments/lib/WOPI/DiscoveryManager.php#132”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”,“version”:“26.0.1.1”,“data”:{“app”:“PHP”}}

Here are some other topics about this on the forum. Should be many on github as well and on web searches.

https://help.nextcloud.com/search?context=topic&context_id=162121&q=Error%3A%20Trying%20to%20access%20array&skip_context=true

Still not able to get a consistent failure scenario. Mostly there’s no issues opening documents from the Dashboard recommended files list, but not always. Mostly the Files app starts causing permanently active processes the moment it’s access, but occaisionally it’s fine immediately after a restart.

I did find errors in nextcloud.log regarding locks, and the locks table had a lot of entries in. Empyting the table and disabling locking made no difference.

In the apache log I have seen references to “GET /index.php/core/preview?fileId=10&x=250&y=250 HTTP/1.1” 504 792 “-”

The parameters match the Request URI on the stuck processes. Disabling previews completely appears to have bypassed the problem (at least for the last 10 minutes). While I’m not that bothered about icons for documents, this is not exactly brilliant for the Photo’s app so it’s not a solution maybe a hint of where the issue might be?

Top shows c. 278MB free, 700 used, 1024 buff/cache and swap is barely touched so it doesn’t look like RAM shortage, and I’ve not seen anything in other logs to suggest there’s other problems with resource as mentioned in the previews doco.

Ideas most welcome!

-R

Are you running the recently released 8.0.1 of the richdocuments app or still on the older one?

Can you please provide the lock related logs you’re
seeing?

Also what are your currently enabled apps?

Also, are you using a self-signed certificate by chance?

This topic was created before that release, so definitely was reported for the older version.

Hi jtr,
It’s an Internal CA, the CA root cert is trusted by browser and is also on the host too. That’s only recently been added to the dev host though. There is a public cert on a proxy, but dev host isn’t using that and both dev and test show same symptoms.

It really didn’t like me trying TLS termination on the proxy and using plain HTTP on Nextcloud.

Nextcloud, PHP & Apache versions haven’t changed - still 8.0.1 / 8.1.18 / 2.2.41
Apps shows:
CODE 22.5.1301
Nextcloud Office 8.02
Group Folders 14.0.2

These are dev / test hosts so keeping up to date with only the freshest and juiciest set of bugs :slight_smile:

Full list from OCC

Enabled:

  • activity: 2.18.0
  • circles: 26.0.0
  • cloud_federation_api: 1.9.0
  • comments: 1.16.0
  • contactsinteraction: 1.7.0
  • dashboard: 7.6.0
  • dav: 1.25.0
  • federatedfilesharing: 1.16.0
  • federation: 1.16.0
  • files: 1.21.1
  • files_pdfviewer: 2.7.0
  • files_rightclick: 1.5.0
  • files_sharing: 1.18.0
  • files_trashbin: 1.16.0
  • files_versions: 1.19.1
  • firstrunwizard: 2.15.0
  • logreader: 2.11.0
  • lookup_server_connector: 1.14.0
  • nextcloud_announcements: 1.15.0
  • notifications: 2.14.0
  • oauth2: 1.14.0
  • password_policy: 1.16.0
  • photos: 2.2.0
  • privacy: 1.10.0
  • provisioning_api: 1.16.0
  • recommendations: 1.5.0
  • related_resources: 1.1.0-alpha1
  • richdocuments: 8.0.2
  • richdocumentscode: 22.5.1301
  • serverinfo: 1.16.0
  • settings: 1.8.0
  • sharebymail: 1.16.0
  • support: 1.9.0
  • survey_client: 1.14.0
  • systemtags: 1.16.0
  • text: 3.7.2
  • theming: 2.1.1
  • twofactor_backupcodes: 1.15.0
  • updatenotification: 1.16.0
  • user_status: 1.6.0
  • viewer: 1.10.0
  • weather_status: 1.6.0
  • workflowengine: 2.8.0
    Disabled:
  • admin_audit: 1.16.0
  • bruteforcesettings: 2.6.0
  • encryption: 2.14.0
  • files_external: 1.18.0
  • fileslibreofficeedit: 1.1.0 (installed 1.1.0)
  • suspicious_login: 4.4.0
  • twofactor_totp: 8.0.0-alpha.0
  • user_ldap: 1.16.0

50 pages of this in the nextcloud.log, all look to be same pattern of trace with different date stamps and files.
Lock entries were being created in the database even for viewing docs, yet in most cases it was possible to edit, at least untill it ran out of PHP processes.

{“reqId”:“puC25x8vvDX8zxvCjWMJ”,“level”:3,“time”:“2023-05-23T20:31:17+00:00”,“remoteAddr”:“192.168.13.10”,“user”:“docadmin”,“app”:“richdocuments”,“method”:“POST”,“url”:“/index.php/apps/richdocuments/wopi/files/169_oci2vbbianjn/contents?access_token=jGk1zWeDVnWD2tAACXPlUT98PZtdYJF4&access_token_ttl=1684909868000%2Fws%3FWOPISrc%3Dhttps%3A%2F%2Fnextcloud-doc.public.example.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F169_oci2vbbianjn&compat=”,“message":“"TestDoc.odt" is locked”,“userAgent”:“COOLWSD HTTP Agent 22.05.13.1”,“version”:“26.0.1.1”,“exception”:{“Exception”:“OCP\Lock\LockedException”,“Message”:“"TestDoc.odt" is locked”,“Code”:0,“Trace”:[{“file”:”/srv/nextcloud/www/lib/private/Files/View.php",“line”:674,“function”:“changeLock”,“class”:“OC\Files\View”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Node/File.php”,“line”:71,“function”:“file_put_contents”,“class”:“OC\Files\View”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:504,“function”:“putContent”,“class”:“OC\Files\Node\File”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:786,“function”:“OCA\Richdocuments\Controller\{closure}”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”,“args”:[“*** sensitive parameters replaced “]},{“file”:”/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php",“line”:763,“function”:“retryOperation”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Lock/LockManager.php”,“line”:29,“function”:“OCA\Richdocuments\Controller\{closure}”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php",“line”:766,“function”:“runInScope”,“class”:“OC\Files\Lock\LockManager”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:505,“function”:“wrappedFilesystemOperation”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/Http/Dispatcher.php”,“line”:230,“function”:“putFile”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/Http/Dispatcher.php”,“line”:137,“function”:“executeController”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/App.php”,“line”:183,“function”:“dispatch”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Route/Router.php”,“line”:315,“function”:“main”,“class”:“OC\AppFramework\App”,“type”:“::”},{“file”:“/srv/nextcloud/www/lib/base.php”,“line”:1056,“function”:“match”,“class”:“OC\Route\Router”,“type”:“->”},{“file”:“/srv/nextcloud/www/index.php”,“line”:36,“function”:“handleRequest”,“class”:“OC”,“type”:“::”}],“File”:“/srv/nextcloud/www/lib/private/Files/View.php”,“Line”:2018,“Previous”:{“Exception”:“OCP\Lock\LockedException”,“Message”:“"files/8dec4aacc0254a55cd4ac5ed9bc2ae0d" is locked”,“Code”:0,“Trace”:[{“file”:“/srv/nextcloud/www/lib/private/Files/Storage/Common.php”,“line”:826,“function”:“changeLock”,“class”:“OC\Lock\DBLockingProvider”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Storage/Wrapper/Wrapper.php”,“line”:629,“function”:“changeLock”,“class”:“OC\Files\Storage\Common”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Storage/Wrapper/Wrapper.php”,“line”:629,“function”:“changeLock”,“class”:“OC\Files\Storage\Wrapper\Wrapper”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/View.php”,“line”:2012,“function”:“changeLock”,“class”:“OC\Files\Storage\Wrapper\Wrapper”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/View.php”,“line”:674,“function”:“changeLock”,“class”:“OC\Files\View”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Node/File.php”,“line”:71,“function”:“file_put_contents”,“class”:“OC\Files\View”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:504,“function”:“putContent”,“class”:“OC\Files\Node\File”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:786,“function”:“OCA\Richdocuments\Controller\{closure}”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php",“line”:763,“function”:“retryOperation”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Files/Lock/LockManager.php”,“line”:29,“function”:“OCA\Richdocuments\Controller\{closure}”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”,“args”:[" sensitive parameters replaced ***”]},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:766,“function”:“runInScope”,“class”:“OC\Files\Lock\LockManager”,“type”:“->”},{“file”:“/srv/nextcloud/www/apps/richdocuments/lib/Controller/WopiController.php”,“line”:505,“function”:“wrappedFilesystemOperation”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/Http/Dispatcher.php”,“line”:230,“function”:“putFile”,“class”:“OCA\Richdocuments\Controller\WopiController”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/Http/Dispatcher.php”,“line”:137,“function”:“executeController”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/AppFramework/App.php”,“line”:183,“function”:“dispatch”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:“->”},{“file”:“/srv/nextcloud/www/lib/private/Route/Router.php”,“line”:315,“function”:“main”,“class”:“OC\AppFramework\App”,“type”:“::”},{“file”:“/srv/nextcloud/www/lib/base.php”,“line”:1056,“function”:“match”,“class”:“OC\Route\Router”,“type”:“->”},{“file”:“/srv/nextcloud/www/index.php”,“line”:36,“function”:“handleRequest”,“class”:“OC”,“type”:“::”}],“File”:“/srv/nextcloud/www/lib/private/Lock/DBLockingProvider.php”,“Line”:220},“CustomMessage”:“–”}}

I call Col. Mustard to the dining room with the lead piping… I have a suspect.

This appears to be some issue between preview generation and Nextcloud Office / CODE.

Disabling previews completely and the problem goes away. And also images in Photos app.
config.php:
‘enable_previews’ => false,

Excluding OpenDocument from the list of previews (and of course re-enabling previews) and the problem goes away
config.php:
‘enabledPreviewProviders’ => [
‘OC\Preview\PNG’,
‘OC\Preview\JPEG’,
‘OC\Preview\GIF’,
‘OC\Preview\BMP’,
‘OC\Preview\TXT’,
‘OC\Preview\MarkDown’,
]

Adding ‘OC\Preview\OpenDocument’ into the list and the problem comes right back, and even after removing it again the issue continues until PHP is restarted.

Theory / wild guess
When first user logs in after a restart Dashboard shows documents which triggers preview generation. But CODE is not launched untill the first user opens a document - and then takes a few seconds to get going. Therefore preview cannot connect to code server - hence 504 errors in log - but the process keeps trying.

Somehow on the test instance, some Office docs do have previews generated, so it may be a specific document type or content that’s causing the issue.

Personally I’d rather have an icon for a document rather than a thumbnail of the first page, I’ll probably take markdown and text off the list. I guess I’m just old fashioned.

As to why this is happening, I’ve no idea. Maybe a gentleman detective of superior intellect will be able to deduce what is going on and ask some elementary questions.

Oh and one last thing, I’ve also re-enabled file locking and 57 entries came back into the oc_file_locks table. But since it doesn’t seem to be actively breaking anything I’ve not got back into the logs to see if it’s complaining.