Upload file & synchronization performance problem

Relative github issue: https://github.com/nextcloud/docker/issues/908

My running environments:
Physical OS: Windows server 2019 Datecenter (v1809, 17763.805)
Docker: Docker for windows 2.1.0.4 (Engine: 19.03.4, Compose: 1.24.1, Hyper-V with 4 vCPUs and 4G vRAM)
MySQL: MySQL for winx64 8.0.18 Community Server
Nginx: Nginx for windows 1.16.1
Nextcloud: 17.0.0 (nextcloud:latest)

Additional Containers:
Onlyoffice-document-server: https://hub.docker.com/r/onlyoffice/documentserver, tag:latest
redis: https://hub.docker.com/_/redis, tag:latest
My docker-compose yml:
version: '3'

networks:
  nextcloud:

services:

  redis:
    image: redis
    container_name: redis
    hostname: redis
    restart: always
    networks:
      - nextcloud
    expose:
      - 6379

  nextcloud:
    image: wyxls/nextcloud:full
    container_name: nextcloud
    restart: always
    depends_on:
      - redis
    environment:
      - NEXTCLOUD_TABLE_PREFIX=oc_
    volumes:
      - C:/Docker/nextcloud:/var/www/html
      - D:/Docker/nextcloud/data:/var/www/html/data
    ports:
      - 10000:80
    networks:
      - nextcloud

  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    stdin_open: true
    tty: true
    restart: always
    depends_on:
      - nextcloud
    volumes:
      - C:/Docker/onlyoffice/document_data:/var/www/onlyoffice/Data
      - C:/Docker/onlyoffice/document_log:/var/log/onlyoffice
      - C:/Docker/onlyoffice/document_fonts:/usr/share/fonts/truetype/custom
      - C:/Docker/onlyoffice/document_forgotten:/var/lib/onlyoffice/documentserver/App_Data/cache/files/forgotten
    ports:
      - 10005:443
    networks:
      - nextcloud
My nginx.conf and nextcloud.conf:
#user nobody;
worker_processes auto;
error_log  logs/error.log;
pid        logs/nginx.pid;


events {
        multi_accept on;
        worker_connections 1024;
}


http {
      include       mime.types;
      #include       naxsi_core.rules;
      default_type  application/octet-stream;

      access_log     off;
      charset        utf-8;
      #aio           threads;
      #directio      1000m;
      sendfile       on;
      tcp_nopush     on;
      tcp_nodelay    on;
      server_tokens  off;
      types_hash_max_size 2048;

      keepalive_timeout 65s;
      keepalive_requests 100;
      client_body_timeout 60s;
      client_header_timeout 60s;
      send_timeout 60s;
      reset_timedout_connection on;
      client_header_buffer_size 4k;
      large_client_header_buffers 4 16k;
      client_max_body_size 32m;
	  client_body_buffer_size 32m ;

      open_file_cache max=100000 inactive=20s;
      open_file_cache_valid    30s;
      open_file_cache_min_uses 2;
      open_file_cache_errors   on;

      fastcgi_connect_timeout 600s;
      fastcgi_send_timeout 600s;
      fastcgi_read_timeout 600s;

      gzip on;
      gzip_vary on;
      gzip_min_length 1k;
      gzip_buffers 8 32k;
      gzip_comp_level 4;
      gzip_types text/plain text/css text/xml text/x-component application/json application/javascript application/rss+xml application/xhtml+xml application/atom+xml image/svg+xml image/x-ms-bmp image/x-icon;
      gzip_disable "MSIE [1-6]\.(?!.*SV1)";

      include C:/nginx/appconf/*.conf;
}
server {
    listen      10002 ssl;
    server_name example localhost 127.0.0.1;
    root        C:/Docker/nextcloud;
    index       index.php;

    ssl_certificate       C:/SSL-Certificates/cer.cer;
    ssl_certificate_key   C:/SSL-Certificates/key.key;
    ssl_protocols SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
    ssl_prefer_server_ciphers on;

    proxy_hide_header Strict-Transport-Security;
    proxy_hide_header X-Content-Type-Options;
    proxy_hide_header X-Robots-Tag;
    proxy_hide_header X-Frame-Options;
    proxy_hide_header X-Download-Options;
    proxy_hide_header X-Permitted-Cross-Domain-Policies;
    proxy_hide_header Referrer-Policy;
    proxy_hide_header X-XSS-Protection;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-Robots-Tag "none";
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Download-Options "noopen";
    add_header X-Permitted-Cross-Domain-Policies "none";
    add_header Referrer-Policy "no-referrer";
    add_header X-XSS-Protection "1; mode=block";

    client_max_body_size 10G;
    fastcgi_buffers 64 4K;
    fastcgi_hide_header X-Powered-By;
    
    location / {
        proxy_pass http://127.0.0.1:10000/;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_buffer_size 64k;
        proxy_buffers 8 64k;
        proxy_busy_buffers_size 64k;
        proxy_temp_file_write_size 64k;
    }

	location = /.well-known/carddav {
        return 301 $scheme://$http_host/remote.php/dav;
    }

	location = /.well-known/caldav {
        return 301 $scheme://$http_host/remote.php/dav;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
        deny all;
    }

    location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
        deny all;
    }

}
My MySQL my.ini
[mysqld]
port=3306
basedir=C:\mysql-winx64
datadir=C:\mysql-winx64\Data
max_connect_errors=10
character-set-server=utf8mb4
collation-server = utf8mb4_general_ci
default-storage-engine=INNODB
default_authentication_plugin=mysql_native_password

key_buffer_size = 128M
max_allowed_packet = 32M
bulk_insert_buffer_size = 32M
thread_stack = 256K
thread_cache_size = 16
table_open_cache = 1024
innodb_buffer_pool_size = 512M
innodb_log_buffer_size = 32M

[mysql]
default-character-set=utf8

[client]
port=3306

I run Nextcloud windows client to synchronize large mount of small file (like xlsx, docs, png, jpg, etc). After I added sync folder in client , the client was noticing “check for external changes”. The progress became stuck every several folders while the client was noticing "check for external changes ‘folder name’ ".

Similar things happen while I uploading big file to Nextcloud in web. Uploding at maximum speed 20+MB/s and slowing down to zero every several seconds.

If it stuck for a long time >60s, sync and upload will fail.

In Nginx’s error.log, it shows :

upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while reading response header from upstream, client: x.x.x.x, server: example.com, request: "PROPFIND /remote.php/dav/files/wyxls/filename.xlsx HTTP/1.1", upstream: " http://127.0.0.1:10000/remote.php/dav/files/wyxls/filename.xlsx", host: "example.com:10002"

Nextcloud sync client shows:

Error transferring https://example.com:10002/remote.php/dav/files/filename.xlsx - server replied: 504 Gateway time-out

At the same time, Nextcloud’s web seems to be frozen and cannot be access. It’s like the Nextcloud backend server doesn’t respond to Nginx nor client’s HTTP requests.

I have enlarged lots of variables:

MySQL's buffer things
Nginx's proxy buffer things
Nextcloud's memory limit
.....etc

There’s no relative error in nextcloud.log nor relative error in “docker logs -f nextcloud” while those “gateway time out” problems happening.

It seems like some parts of the Nextcloud Container reach bottleneck and never release until I restart the Hyper-V Docker VM.

I run “docker stats” to see if the hardware resource is totally being drained but CPU and Memory are only being consumed about 20%

Nothing work though. The most weird thing is that everything else works well including redis and onlyoffice-document-server.

Now I’m going nuts and don’t know what I can do to solve this problem.

Anyone helps?

I think the problem is really about the nextcloud container. Everytime I stop&rm&up the nextcloud via compose.yml, all of the functions work extremely fine in a short time at the beginning (about 4-20m)

After a while, that Symptom begins to happen again

You’re more or less running the most unsupported installation here.

Helpful advice:

  1. Nginx on windows has issues. Especially since it can only use one of the workers it spawns.

  2. You have no php.ini listed and the php version is absent.

  3. Your nginx.conf is not the recommended configuration. The recommended configuration is in the official documentation.

1.I have another webserver (a personal blog) running on totally the same environments and it works fine. That’s why I said the problem was somewhere else inside the Nextcloud Container.

2.The php is combined to the Nextcloud image.Sorry about missing the version info.
PHP 7.3.10 (cli) (built: Oct 17 2019 15:02:18) ( NTS )
Zend Engine v3.3.10, Copyright Š 1998-2018 Zend Technologies
with Zend OPcache v7.3.10, Copyright Š 1999-2018, by Zend Technologies

3.Of course it’s not the recommended configuration because there’s no official reverse proxy .conf can be referred. I searched lots of stuffs to make this one.
As my host system is server 2019, I don’t know how to run nginx correctly within docker (nginx image container) to handle external https requests.

Unless you provide the php.ini it can’t be ruled out.

Nor can the reverse proxy or your actual web server… or windows.

The software (nginx in particular) is not designed with windows in mind. If you remove it from the equation what happens?

What do the logs say in windows/nginx/php/mysql? That would help narrow it down. You said nextcloud had no errors, but what about the other software?

It takes me some time to find the logs and php.ini.

PHP:

There are many of ini in /usr/local/etc/php. No php.ini in it but they are named php.ini-development and php.ini-production

In /usr/local/etc/php/conf.d folder, it seems have many extensions ini. Here are some of them.

memory.ini: (it’s matched with the memory limit showed at Nextcloud website-settings-system)
memory_limit=512M
opcache-recommended.ini:
opcache.enable=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1

The others are like:

extension=zip.so

Nginx’s error.log:

There are some patterns of errors while sync client failed as “Gateway time-out”.

2019/10/29 13:49:24 [error] 14740#14948: *71900 connect() failed (10061: No connection could be made because the target machine actively refused it) while connecting to upstream, client: 192.168.9.1, server: example.com, request: "PROPFIND /remote.php/dav/files/Dilong/ HTTP/1.1", upstream: "http://127.0.0.1:10000/remote.php/dav/files/Dilong/", host: "example.com:10002"

2019/10/29 13:49:30 [error] 14740#14948: *71900 upstream prematurely closed connection while reading response header from upstream, client: 192.168.9.1, server: example.com, request: "PROPFIND /remote.php/dav/files/Dilong/ HTTP/1.1", upstream: "http://127.0.0.1:10000/remote.php/dav/files/Dilong/", host: "example.com:10002"

2019/10/29 13:57:09 [error] 14740#14948: *72275 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while reading response header from upstream, client: 192.168.9.1, server: example.com, request: "PROPFIND /remote.php/dav/files/Dilong/%E5%A4%87%E7%94%A8/%E7%81%AF%E9%A5%B0%E8%B4%A7%E6%AC%BE%E8%A1%A8/%E8%B4%A7%E6%AC%BE%E8%A1%A8/2002/5%E6%9C%88%E4%BB%BD%E8%B4%A7%E6%AC%BE%E8%A1%A8 HTTP/1.1", upstream: "http://127.0.0.1:10000/remote.php/dav/files/Dilong/%E5%A4%87%E7%94%A8/%E7%81%AF%E9%A5%B0%E8%B4%A7%E6%AC%BE%E8%A1%A8/%E8%B4%A7%E6%AC%BE%E8%A1%A8/2002/5%E6%9C%88%E4%BB%BD%E8%B4%A7%E6%AC%BE%E8%A1%A8", host: "example.com:10002"
Nextcloud:

I can’t see any relative error while the issue happens. Only some unknown errors with autoloader.php#137 show in the nextcloud.log

{"reqId":"Qc5GUYqW2UkxzzweFUkR","level":3,"time":"2019-11-01T03:24:05+00:00","remoteAddr":"14.20.169.15","user":"--","app":"PHP","method":"PROPFIND","url":"\/remote.php\/dav\/files\/Dilong\/","message":"OCP\\AutoloadNotAllowedException: Autoload path not allowed: \/var\/www\/html\/apps\/dav\/lib\/connector\/sabre\/exceptionloggerplugin.php at \/var\/www\/html\/lib\/autoloader.php#137","userAgent":"Mozilla\/5.0 (Windows) mirall\/2.6.0stable-Win64 (build 20190927) (Nextcloud)","version":"17.0.0.9"}
{"reqId":"yQQ8yxbhsFKm5jteQpoh","level":3,"time":"2019-11-01T04:58:03+00:00","remoteAddr":"172.26.0.1","user":"Zakikun","app":"jsresourceloader","method":"GET","url":"\/apps\/music\/","message":"Could not find resource core\/js\/placeholder.js to load","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.70 Safari\/537.36","version":"17.0.0.9"}
{"reqId":"Q52QVwp1Z9miztsaWoaC","level":3,"time":"2019-11-01T05:00:17+00:00","remoteAddr":"172.26.0.1","user":"Zakikun","app":"jsresourceloader","method":"GET","url":"\/apps\/music\/","message":"Could not find resource core\/js\/placeholder.js to load","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/78.0.3904.70 Safari\/537.36","version":"17.0.0.9"}
{"reqId":"4FsHuaPiN55AEQn30w4I","level":3,"time":"2019-11-01T07:11:19+00:00","remoteAddr":"172.26.0.1","user":"--","app":"PHP","method":"GET","url":"\/ocs\/v1.php\/cloud\/user?format=json","message":"Doctrine\\DBAL\\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] Connection timed out at \/var\/www\/html\/lib\/private\/DB\/Connection.php#64","userAgent":"Mozilla\/5.0 (Windows) mirall\/2.6.0stable-Win64 (build 20190927) (Nextcloud)","version":"17.0.0.9"}
Nextcloud’s config.php:
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\OC\Memcache\Redis',
  'memcache.distributed' => '\OC\Memcache\Redis',
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' => 
  array (
    'host' => 'redis',
    'port' => 6379,
  ),
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'instanceid',
  'passwordsalt' => 'passwordsalt',
  'secret' => 'secret',
  'trusted_domains' => 
  array (
    0 => '*',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '17.0.0.9',
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'http://127.0.0.1:10000',
  'dbname' => 'nextcloud',
  'dbhost' => '192.168.9.13',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'nextcloud',
  'installed' => true,
  'check_data_directory_permissions' => false,
  'default_language' => 'zh_CN',
  'default_locale' => 'zh_Hans_CN',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

I tried a lot to locate where the narrow point is.
1.All parts run in the Docker (MySQL, Nginx). They’re all running on Linux OS. The issue occured.
2.Transferred MySQL from docker to windows. Same issue.
3.Nginx is migrated to windows too. The same.
4.Now I remove ‘overwriteprotocol’ => ‘https’ in nextcloud’s config.php and connect sync client to Nextcloud via LAN network (192.168.9.13:10000) and the problems still occured.

So I’m sure it’s not about MySQL or Nginx. I really know that deploying all softwares on centOS will be much more better. Since my families are used to Windows and demand for Gaming, the server must run windows.

To be honest, I have an x86 platform Router which is flashed into LEDE and runs nextcloud pretty well. I just wondering if it’s possible to deploy Nextcloud on Windows.

Never mind.

At last, It’s like some kind of resource in Nextcloud container is exhausted, which leads to hanging for sometime and the time-out of syncing.

Looking forward to your reply.

Going back to just your nginx config, why are you deviating so much from the official recommended configuration? Ignoring the proxy parts entirely.

I would really need your php.ini, but since you don’t seem to know where it is, that’s too bad.

Your nginx error log shows connections being refused. This is not a nextcloud issue.

The nextcloud log shows your database losing connection. This is not a nextcloud issue.

I’m not sure why you have your setup the way you do. I don’t know why you’re not using a hyper-v container to host linux and run everything on bare metal (no docker/snap). That’s your prerogative but it’s far outside of the support this community can provide, especially since it’s not a nextcloud issue.

If you don’t want to change your setup you should hire a knowledgeable system administrator to help you.

Cheers!

My nginx config begins with very simple proxy_pass and expands to present because the nextcloud selfcheck indicates it demands those I-dont-know-why headers and redirects.
I do my best to “get close” to the official configuration but there’s no nginx reverse proxy document can be referred to.

In my understanding, those “location” parts are only tuning nextcloud. I copied them into my nginx conf but it caused css style, js, png&jpg showing problems. The nextcloud mainpage became unusable. (http://expamle.com/apps/files/)

So please let me know which conf I should refer to.

official example:

https://docs.nextcloud.com/server/17/admin_manual/installation/nginx.html (no reverse proxy relative configuration)
https://docs.nextcloud.com/server/17/admin_manual/configuration_server/reverse_proxy_configuration.html (only about config.php and no nginx configuration)


I have no choice. I have tried to run MySQL (like mariadb:10.1) and Nginx in Docker, and have tried many other plans.

But Mariadb container can’t read/write files in the folder “D:\mysql\db”. It’s like a file system properties problem.

I don’t know how to configure Nginx exactly correct due to the complex NAT network ( Windows—HyperV vEthernet—Docker Host—containers). Failure requesting makes me dumb.

It just works fine when I put them on my windows OS except those issues being mentioned at topic.


I did a lot of troubleshooting and tests to locate the problem of nextcloud itself. Especially the other blog webserver (docker version) has no problem in the same environment.

What nginx error log shows “upstream time out” means the nextcloud container doesn’t respond to nginx—client request. This is not nginx issue.

What nextcloud log shows is totally not relative with the time-out issue. This is nextcloud common issue and can be found a lot on Google.
Even on my x86 LEDE router, autoloader.php#137 often appears in log. But it does nothing about the performance problem.


Yes, it’s my prerogative. I do recognized running nextcloud on windows is a foolish job. I wonder if it’s possible to set all things properly on windows sever 2019 and make the best use of its hardware which is much more better than my poor router’s.

Guess what, I’m almost giving up now. Nextcloud is slow on my LEDE router, but it’s also available and stable.

How is this a nextcloud issue? What error do you see that indicates it’s a nextcloud issue?

Just because your other website works fine… doesn’t mean it’s a nextcloud issue.

I, personally, have experienced none of the issues you’ve posted about including the connection timeouts.

Why isn’t nextcloud’s issue?
The nginx log shows the "upstream: “http://127.0.0.1:10000/remote.php/dav/files/Dilong/filename” has failed to respond

Otherwise, everytime I up the nextcloud container, no issue happens. It occurs after running a period of random time.

“has failed to respond” means the proxy can’t connect to the backend. That… doesn’t make it specifically a nextcloud issue. Maybe an issue with your container, the web server, the mysql database, the file system, etc. But that’s not a nextcloud error.

So what’s your suggestion?
it’s a meaningless discussion about which parts have problems?
it’s not helpful to solve at all
And I sure that nextcloud container causing performance slowdown.
All you said can’t explain why my personal wordpress blog runs well compared to nextcloud.
Stop arguing this. I’m dumb with docker but not a stupid of troubleshooting.

Without an error there is no suggestion because there is no way to know where the problem is.

I also don’t believe this to be a nextcloud issue based on what’s been provided.

1 Like

IMHO @Paradox55 gives fair advice and seeks to lend a hand.
:+1:

However, @wyxls please mind this is an open ‘home user’ forum.
:neutral_face:

Furthermore, one could consult the Nextcloud Community Guidelines


Be pragmatic

Nextcloud is a pragmatic community. We value tangible results over having the last word in a discussion. We defend our core values like freedom and respectful collaboration, but we don’t let arguments about minor issues get in the way of achieving more important results. We are open to suggestions and welcome solutions regardless of their origin. When in doubt support a solution which helps getting things done over one which has theoretical merits, but isn’t being worked on. Use the tools and methods which help getting the job done. Let decisions be taken by those who do the work.


Maybe a more friendly attitude and a more collaborative approach could help?
:innocent:

Happy hacking.
:sunflower:

1 Like

I have changed to use seafile in the same system environments (MySQL for windows, nginx for windows, docker seafile, etc)

It runs much more efficient while I uploading tons of small files (such as 20000+ and all files <1MB) and large files >10GB.No performance issue after testing a day time.

Don’t know why you think it isn’t nextcloud efficiency problem.

I’m also a “home user” and just need a private sync software for my work files.

Nextcloud is a good product for multiple usage such as IM talk, onlyoffice document server, audio plugins.But its basic upload function doesn’t meet my demands.

So Forget about this thread and let there be peace in this forum.

Do you or do you not have nextcloud related errors other then the proxy backend not connecting to your web server?

If you feel this is a nextcloud issue run php and nextcloud in debug mode and find out.

If you find an error, post here. We can help you then.

Hello, guys. I’m back and answer to my own question.

I have solved the problem by chance. It’s some kind of datebase running issue on Windows as I was using MySQL for Windows.

When I come back to use MariaDB on Linux ( within Windows Docker ), everything goes well without any performance problem. I can still keep the datebase files mounted to Windows NTFS volume for permanent store and backup purpose