Slow performance generally with NC 12

Sorry to hear you’re facing problems :slight_frown:

If you’re urgently looking for support to keep your business running, consider checking out our paid support options.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the following as you can (you can cut and paste this into your topic):


Nextcloud version (eg, 10.0.2): 12.0.3
Operating system and version (eg, Ubuntu 16.04): 16.04
Apache or nginx version (eg, Apache 2.4.25): Nginx 1.12.1
PHP version (eg, 5.6): 7.0.22
Is this the first time you’ve seen this error?: yes

Can you reliably replicate it? (If so, please outline steps):

The issue you are facing:

General performance is slow. On mobile apps (both Android/iOS) thumbnails do not load quickly and sometimes on iOS a lot of thumbnails are not loaded until I tap on it. When I try to upload video via iOS app, it does not work. First, there is a delay of about 10-15 sec, second I get ‘Bad request’ error. On Android video uploading works fine.

I thought it is due to enabled encryption but after turning it off, removing all the files and re-uploading again I did not notice any difference.

I did another test: uploaded a video file (about 750Mb) to my nextcloud and to another cloud storage provider in Germany. Here is the result:

My nextcloud - 13 mins
Germany provider - 3 mins

My NC is installed on VPS with the following config:

6GB RAM
4 vCPU Cores
150GB Storage
1Gbps uplink
OpenVZ

From my current location ping is about 40-50ms. I will ask my friend to test it from his location.

Any advice would be highly appreciated because I am thinking to give up now and go for another service :frowning:

Here is my nginx conf:

upstream php-handler {
    #server 127.0.0.1:9000;
    server unix:/run/php/php7.0-fpm.sock;
}

server {
    listen 80;
    server_name mydomain;
    # enforce https
    return 301 https://$server_name$request_uri;

    # Redirect non-https traffic to https
     if ($scheme != "https") {
         return 301 https://$host$request_uri;
     } 

}

server {
    listen 443 ssl http2;
    server_name mydomain;

    ssl_certificate /etc/ssl/mydomain/fullchain.pem;
    ssl_certificate_key /etc/ssl/mydomain/privkey.pem;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;
    ssl_dhparam		/etc/nginx/dhparams.pem;
    # Add headers to serve security related headers
    # Before enabling Strict-Transport-Security headers please read into this
    # topic first.
     add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    #
    # WARNING: Only add the preload option once you read about
    # the consequences in https://hstspreload.org/. This option
    # will add the domain to a hardcoded list that is shipped
    # in all major browsers and getting removed from this list
    # could take several months.
    
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # Path to the root of your installation
    root /var/www/nextcloud/;

    proxy_buffering off;
    proxy_max_temp_file_size 0;

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # The following 2 rules are only needed for the user_webfinger app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
    #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
    # last;

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }
    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # set max upload size
    client_max_body_size 2050M;
    fastcgi_buffers 4 256K;
    fastcgi_busy_buffers_size 256k; 
    fastcgi_temp_file_write_size 256k;

    # Enable gzip but do not remove ETag headers
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    # Uncomment if your server is build with the ngx_pagespeed module
    # This module is currently not supported.
    #pagespeed off;

    location / {
        rewrite ^ /index.php$uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        deny all;
    }
    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
        deny all;
    }

    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param HTTPS on;
        #Avoid sending the security headers twice
        fastcgi_param modHeadersAvailable true;
        fastcgi_param front_controller_active true;
        fastcgi_pass php-handler;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
	fastcgi_read_timeout 600;
    }

    location ~ ^/(?:updater|ocs-provider)(?:$|/) {
        try_files $uri/ =404;
        index index.php;
    }

    # Adding the cache control header for js and css files
    # Make sure it is BELOW the PHP block
    location ~ \.(?:css|js|woff|svg|gif)$ {
        try_files $uri /index.php$uri$is_args$args;
        add_header Cache-Control "public, max-age=15778463";
        # Add headers to serve security related headers (It is intended to
        # have those duplicated to the ones above)
        # Before enabling Strict-Transport-Security headers please read into
        # this topic first.
        # add_header Strict-Transport-Security "max-age=15768000;
        #  includeSubDomains; preload;";
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Robots-Tag none;
        add_header X-Download-Options noopen;
        add_header X-Permitted-Cross-Domain-Policies none;
        # Optional: Don't log access to assets
        access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
        try_files $uri /index.php$uri$is_args$args;
        # Optional: Don't log access to other assets
        access_log off;
    }

} 

The output of your Nextcloud log in Admin > Logging:No server logs Everything is working fine

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'auth.bruteforce.protection.enabled' => true,
  'instanceid' => 'private_info',
  'passwordsalt' => 'private_info',
  'secret' => 'private_info',
  'trusted_domains' => 
  array (
    0 => 'mydomain',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'overwrite.cli.url' => 'mydomain',
  'dbtype' => 'mysql',
  'version' => '12.0.3.3',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_admin',
  'dbpassword' => 'private_info',
  'installed' => true,
  'logfile' => '/var/log/nextcloud.log',
  'loglevel' => 2,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'filelocking.enabled' => 'true',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mail_from_address' => 'admin',
  'mail_smtpmode' => 'php',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'mydomain',
  'maintenance' => false,
);

The output of your Apache/nginx/system log in /var/log/____:

error.log:

2017/09/23 09:10:35 [error] 29774#29774: *1 access forbidden by rule, client: IP , server: mydomain, request: “GET /data/.ocdata?t=1506154390057 HTTP/2.0”, host: "mydomain"
2017/09/23 09:17:24 [warn] 29774#29774: *1173 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/1/00/0000000001 while reading upstream, client: IP, server: mydomain, request: “GET /remote.php/webdav/VID_20170807_000316.mp4 HTTP/2.0”, upstream: “fastcgi://unix:/run/php/php7.0-fpm.sock:”, host: "mydomain"
2017/09/23 09:19:04 [error] 29774#29774: *1 access forbidden by rule, client: IP, server: mydomain.com, request: “GET /data/.ocdata?t=1506154899615 HTTP/2.0”, host: “mydomain”

php7.0-fpm.log:

[22-Sep-2017 23:27:39] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[23-Sep-2017 00:40:33] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[23-Sep-2017 06:59:10] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[23-Sep-2017 07:06:51] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[23-Sep-2017 07:15:22] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[23-Sep-2017 08:00:55] NOTICE: Terminating …
[23-Sep-2017 08:00:55] NOTICE: exiting, bye-bye!


Remember, this information may be requested if it isn’t supplied; for fastest response please provide as much as you can :heart: Feel free to use a pastebin service, otherwise log files can be indented with 4 spaces on each line to present them in a friendlier way on the forum.

You can’t use more connections. You can try to increase the limits (if you have enough resources left on your system).

Thanks! I changed my settings as per Adjusting child processes for PHP-FPM (Nginx) :: myshell.co.uk but cannot say I noticed a big improvement.

Good news is that I was able to upload 50Mb video from iOS

my install runs with debian and apache, so i can’t help you with ngnix.
with the default configuration of nextcloud (and its dependent components) i was (very) disappointed regarding its performance, also, especially since one of my users shares a lot of images (90GB) and i had intened it as a kind of replacement solution for gallery2.
the server-hardware is not too bad (14GB RAM, quadcore-xeon), but users experienced wait-times of several minutes just looking at gallery topdirs etc.
i configured php-opcache, which did not help noticeably.
what really boosted performance and and solved the problem was giving huge amounts of memory to mariadb and installing and running the preview app.
mariadb: (Server version: 10.1.26-MariaDB-0+deb9u1 Debian 9.1)
server.cnf (these values are used for ALL dbs running)
i doubled the values of the section “Fine Tuning” so that it now contains:
key_buffer_size = 32M
max_allowed_packet = 64M
thread_stack = 384K
thread_cache_size = 128

and changed/added some values in the sectioon Query Cache Configuration:
# Query Cache Configuration
query_cache_limit = 64M
query_cache_size = 48M
join_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M

and added some extra-vars for innodb:
# EXTRA-vars
innodb_buffer_pool_size = 2G
#innodb_buffer_pool_instances = 2
innodb_buffer_pool_instance = 2
innodb_log_buffer_size = 32M
innodb_log_file_size = 512M
innodb_max_dirty_pages_pct = 95
# innodb_flush_log_at_trx_commit = 1
# !!! 0 kills ACID-compliance
innodb_flush_method = O_DIRECT

if you use mariadb/mysql adjusting those settings might help; (please adjust them to your system)

there is some info here: server and here innodb

the preview app is …well, kind of interesting ,… but since it has generated all the previews the gallery really is impressingly fast. that’s a serious pro for me.
the drawbacks:
-about 95GB of (image)files created 82GB of previews
-preview-generation took ~40hours
-broken/unreadable imagefiles caused the process to quit without an errormessage. i could then see in which virtual folder the process had failed, look at all the pix with the web-interface and delete the one(s) that had caused the failure. luckily, the storage is quite structured.

GOOD LUCK!

3 Likes

@pete.dawgg Thanks! It feels like the performance has increased! However, I am having troubles uploading BIG video files via iOS native client. It freezes for 10-15 sec and then gives me ‘bad request’ error. Sometimes it uploads a file but with 0 size…any idea how to fix it?