Sharing Big Directories are very slow and crash the server

Hihi,

I have big directories and want to share it with groups. It works with smaller directories but needs a long time. If I do this with bigger directories (20GB) it takes a long time and sometimes the server is not reachable. I think it is a memory problem but which?


Nextcloud version (eg, 10.0.2): 11.0.1.2
Operating system and version (eg, Ubuntu 16.04): macOS Sierra
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.23 (Unix)
PHP version (eg, 5.6): 7.0.13
Is this the first time you’ve seen this error and can you replicate it?: YES

The issue you are facing: long time of sharing and sometimes crashing the server

The output of your Nextcloud log in Admin > Logging:

{
“id”: “ocoe0r4f14j2”,
“items”: [
[
“server”,
“version”,
“11.0.1.2”
],
[
“server”,
“code”,
“other”
],
[
“server”,
“enable_avatars”,
“yes”
],
[
“server”,
“enable_previews”,
“yes”
],
[
“server”,
“memcache.local”,
"\OC\Memcache\APCu"
],
[
“server”,
“memcache.distributed”,
“none”
],
[
“server”,
“asset-pipeline.enabled”,
“no”
],
[
“server”,
“filelocking.enabled”,
“yes”
],
[
“server”,
“memcache.locking”,
“none”
],
[
“server”,
“debug”,
“no”
],
[
“server”,
“cron”,
“ajax”
],
[
“php”,
“version”,
“7.0.13”
],
[
“php”,
“memory_limit”,
536870912
],
[
“php”,
“max_execution_time”,
3600
],
[
“php”,
“upload_max_filesize”,
17179869184
],
[
“database”,
“type”,
“mysql”
],
[
“database”,
“version”,
“5.6.34”
],
[
“database”,
“size”,
204963840
],
[
“apps”,
“activity”,
“2.4.1”
],
[
“apps”,
“comments”,
“1.1.0”
],
[
“apps”,
“dav”,
“1.1.1”
],
[
“apps”,
“encryption”,
“1.4.1”
],
[
“apps”,
“federatedfilesharing”,
“1.1.1”
],
[
“apps”,
“federation”,
“1.1.1”
],
[
“apps”,
“files”,
“1.6.1”
],
[
“apps”,
“files_accesscontrol”,
“1.1.2”
],
[
“apps”,
“files_pdfviewer”,
“1.0.1”
],
[
“apps”,
“files_reader”,
“0.8.1”
],
[
“apps”,
“files_sharing”,
“1.1.1”
],
[
“apps”,
“files_texteditor”,
“2.2”
],
[
“apps”,
“files_trashbin”,
“1.1.0”
],
[
“apps”,
“files_versions”,
“1.4.0”
],
[
“apps”,
“files_videoplayer”,
“1.0.0”
],
[
“apps”,
“firstrunwizard”,
“2.0”
],
[
“apps”,
“gallery”,
“16.0.0”
],
[
“apps”,
“logreader”,
“2.0.0”
],
[
“apps”,
“lookup_server_connector”,
“1.0.0”
],
[
“apps”,
“nextcloud_announcements”,
“1.0”
],
[
“apps”,
“notifications”,
“1.0.1”
],
[
“apps”,
“password_policy”,
“1.1.0”
],
[
“apps”,
“provisioning_api”,
“1.1.0”
],
[
“apps”,
“richdocuments”,
“disabled”
],
[
“apps”,
“serverinfo”,
“1.1.1”
],
[
“apps”,
“sharebymail”,
“1.0.1”
],
[
“apps”,
“survey_client”,
“0.1.5”
],
[
“apps”,
“systemtags”,
“1.1.3”
],
[
“apps”,
“theming”,
“1.1.1”
],
[
“apps”,
“twofactor_backupcodes”,
“1.0.0”
],
[
“apps”,
“updatenotification”,
“1.1.1”
],
[
“apps”,
“workflowengine”,
“1.1.1”
],
[
“stats”,
“num_files”,
239169
],
[
“stats”,
“num_users”,
2
],
[
“stats”,
“num_storages”,
5
],
[
“stats”,
“num_storages_local”,
1
],
[
“stats”,
“num_storages_home”,
2
],
[
“stats”,
“num_storages_other”,
2
],
[
“stats”,
“num_comments”,
0
],
[
“stats”,
“num_comment_markers”,
0
],
[
“stats”,
“num_systemtags”,
0
],
[
“stats”,
“num_systemtags_mappings”,
0
],
[
“files_sharing”,
“num_shares”,
5
],
[
“files_sharing”,
“num_shares_user”,
0
],
[
“files_sharing”,
“num_shares_groups”,
5
],
[
“files_sharing”,
“num_shares_link”,
0
],
[
“files_sharing”,
“num_shares_link_no_password”,
0
],
[
“files_sharing”,
“num_fed_shares_sent”,
0
],
[
“files_sharing”,
“num_fed_shares_received”,
0
],
[
“files_sharing”,
“permissions_1_31”,
“5”
],
[
“encryption”,
“enabled”,
“yes”
]
]
}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php $CONFIG = array ( 'instanceid' => 'ocoe0r4f14j2', 'passwordsalt' => 'xxxx', 'secret' => 'xxxx', 'trusted_domains' => array ( 0 => 'nextcloud:8888', ), 'datadirectory' => '/Users/deu/Desktop/www/nextcloud/data', 'overwrite.cli.url' => 'http://nextcloud:8888', 'dbtype' => 'mysql', 'version' => '11.0.1.2', 'dbname' => 'nextcloud', 'dbhost' => 'localhost', 'dbport' => '', 'dbtableprefix' => 'oc_', 'dbuser' => 'xxxx', 'dbpassword' => 'xxxx', 'logtimezone' => 'UTC', 'installed' => true, 'memcache.local' => '\\OC\\Memcache\\APCu', ); The output of your Apache/nginx/system log in `/var/log/____`: [Wed Jan 18 12:08:06 2017] [notice] Digest: generating secret for digest authentication ... [Wed Jan 18 12:08:06 2017] [notice] Digest: done [Wed Jan 18 12:08:06 2017] [notice] Apache/2.2.31 (Unix) mod_wsgi/3.5 Python/2.7.12 PHP/7.0.13 mod_ssl/2.2.31 OpenSSL/1.0.2j DAV/2 mod_fastcgi/2.4.6 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations [Wed Jan 18 12:08:06 2017] [notice] FastCGI: process manager initialized (pid 22933) [Wed Jan 18 12:09:16 2017] [notice] Graceful restart requested, doing restart [Wed Jan 18 12:09:16 2017] [notice] Digest: generating secret for digest authentication ... [Wed Jan 18 12:09:16 2017] [notice] Digest: done [Wed Jan 18 12:09:16 2017] [notice] FastCGI: process manager initialized (pid 23776) [Wed Jan 18 12:09:16 2017] [notice] Apache/2.2.31 (Unix) mod_wsgi/3.5 Python/2.7.12 PHP/7.0.13 mod_ssl/2.2.31 OpenSSL/1.0.2j DAV/2 mod_fastcgi/2.4.6 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations [Wed Jan 18 12:09:58 2017] [notice] Graceful restart requested, doing restart [Wed Jan 18 12:09:59 2017] [notice] Digest: generating secret for digest authentication ... [Wed Jan 18 12:09:59 2017] [notice] Digest: done [Wed Jan 18 12:09:59 2017] [notice] Apache/2.2.31 (Unix) mod_wsgi/3.5 Python/2.7.12 PHP/7.0.13 mod_ssl/2.2.31 OpenSSL/1.0.2j DAV/2 mod_fastcgi/2.4.6 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations [Wed Jan 18 12:09:59 2017] [notice] FastCGI: process manager initialized (pid 24337) [Wed Jan 18 12:11:28 2017] [error] [client ::1] client denied by server configuration: /Users/deu/Desktop/www/nextcloud/data/.ocdata [Wed Jan 18 12:11:35 2017] [error] [client ::1] client denied by server configuration: /Users/deu/Desktop/www/nextcloud/data/.ocdata [Wed Jan 18 12:53:03 2017] [notice] Graceful restart requested, doing restart [Wed Jan 18 12:53:03 2017] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0) [Wed Jan 18 12:53:03 2017] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye [Wed Jan 18 12:53:04 2017] [notice] Digest: generating secret for digest authentication ... [Wed Jan 18 12:53:04 2017] [notice] Digest: done [Wed Jan 18 12:53:04 2017] [notice] Apache/2.2.31 (Unix) mod_wsgi/3.5 Python/2.7.12 PHP/7.0.13 mod_ssl/2.2.31 OpenSSL/1.0.2j DAV/2 mod_fastcgi/2.4.6 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations [Wed Jan 18 12:53:04 2017] [notice] FastCGI: process manager initialized (pid 25831) [Wed Jan 18 12:56:08 2017] [notice] caught SIGTERM, shutting down [Wed Jan 18 12:56:20 2017] [notice] Digest: generating secret for digest authentication ... [Wed Jan 18 12:56:20 2017] [notice] Digest: done [Wed Jan 18 12:56:20 2017] [notice] FastCGI: process manager initialized (pid 26870) [Wed Jan 18 12:56:20 2017] [notice] Apache/2.2.31 (Unix) mod_wsgi/3.5 Python/2.7.12 PHP/7.0.13 mod_ssl/2.2.31 OpenSSL/1.0.2j DAV/2 mod_fastcgi/2.4.6 mod_perl/2.0.9 Perl/v5.24.0 configured -- resuming normal operations libpng warning: iCCP: known incorrect sRGB profile libpng warning: iCCP: extra compressed data --- Remember, this information may be requested if it isn't supplied; for fastest response please provide as much as you can :heart: Feel free to use a pastebin service, otherwise log files can be indented with 4 spaces on each line to present them in a friendlier way on the forum.

How long does it take to share a directory in your server? an example

Also what is your Nextcloud server specs? RAM, CPU, etc…

How many clients are you syncing with.

When you say big directories are you talking about a few large files, or many small files?

I had this problem back in the old Owncloud v8 or 9 days and I was running on a system with only 512mb of ram. When I upgraded to 1GB it fixed my problem. I was the only user with about 5 clients.

Can you share the output of your NC installation log? It only needs to be error related.

And good lord don’t publish your passwords on here. Edited your post.

1 Like

its a mac mini with 16 GB RAM. I think it is ok, isn´t it?

Where can I find the NC Installation log. Sorry.

Yeah 16GB should be plenty

but what can I do?

If you plan to handle a lot of files, I would recommend to use a redis file locking cache. That reduces the load on your database. It’s also worth to review your database configuration, the default config is not really adapted for NC. There are some scripts that help you to analyze the different parameters (tuning-primer, mysqltuner). You will find some topics here or on the owncloud forum.

Please note that Mac is not a supported OS. There are some problems; some were related to utf-8 encoding.