Some requests with user provisioning API are very slow

We are running a large Nextcloud instance with about 5000 total users. On this instance, the following API call takes around 50 seconds to execute.

curl -X PUT https://admin:password@server/ocs/v1.php/cloud/users/myuser -d key="quota" -d value="400MB" -H "OCS-APIRequest: true"

That is obviously way too slow to be acceptable. Running the same command via occ takes just one second.

sudo docker-compose exec -T --user www-data nextcloud php /var/www/html/occ user:setting myuser "" quota 400MB

Interestingly, repeating the http call a second time (so the setting doesnt actually change) is very fast:

time curl -X PUT https://admin:password@ecloud.global/ocs/v1.php/cloud/users/myuser -d key="quota" -d value="200MB" -H "OCS-APIRequest: true"
# 52 seconds
time curl -X PUT https://admin:password@ecloud.global/ocs/v1.php/cloud/users/myuser -d key="quota" -d value="200MB" -H "OCS-APIRequest: true"
# 0.6 seconds

I also tested the same API call on another server which uses the same setup and same software versions (but less users). There the API call is very fast as expected.

Nextcloud version: 15.0.12 (via Docker)
Operating system and version: Ubuntu 18.04
Apache or nginx version: nginx 1.17 (via Docker)
PHP version: PHP 7.3.10

As this is a large instance, it is hard to find out which logging lines are related to the problem. But grepping for the affected usernamein data/nextcloud.log only show these lines:

{"reqId":"oHQdsEcVv1GaV2om9MXI","level":0,"time":"2019-10-11T12:31:07+00:00","remoteAddr":"51.75.xx","user":"ncadmin","app":"no app in context","method":"PUT","url":"\/ocs\/v1.php\/cloud\/users\/myuser","message":"No cache entry found for \/appdata_ocgvy93waet9\/avatar\/myuser\/avatar.jpg (storage: local::\/var\/www\/html\/data\/, internalPath: appdata_ocgvy93waet9\/avatar\/myuser\/avatar.jpg)","userAgent":"curl\/7.58.0","version":"15.0.12.1"}
{"reqId":"oHQdsEcVv1GaV2om9MXI","level":0,"time":"2019-10-11T12:31:33+00:00","remoteAddr":"51.75.xx","user":"ncadmin","app":"no app in context","method":"PUT","url":"\/ocs\/v1.php\/cloud\/users\/myuser","message":"No cache entry found for \/appdata_ocgvy93waet9\/dav-photocache\/945ec84bf5288ce00587c1cbde534f73 (storage: local::\/var\/www\/html\/data\/, internalPath: appdata_ocgvy93waet9\/dav-photocache\/945ec84bf5288ce00587c1cbde534f73)","userAgent":"curl\/7.58.0","version":"15.0.12.1"}

Other logs dont show anything interesting. Do you have any idea how this problem could be debugged or fixed?

1 Like

I have the same problem. But we have ~50 users.

I have the same problem.

I startet testing the API via curl and recognized that the first API calls ran fast. After a while it got slow. I tested for example creating and deleting of a user. Everytime with the same credentials.

We have <10 users!

So i thought it’s kind of a database cleanup problem but occ files:cleanup and occ versions:cleanup didn’t improve the long time.

We have a similar stack like @nutomic:

  • Nextcloud version 18.0.4 (<10 users)
  • OS: debian 10.3
  • webserver nginx 1.14.2
  • PHP version: PHP 7.3.17

UPDATE:

I took a look under the preferences --> overview and there was a warning of missing indices.
So i made a occ db:add-missing-indices and after that it was a way much faster! Hope it helps.