Hm, interesting.
Maybe one of your installed Nextcloud apps is causing this then.
You could disable one for one and check if that makes it work (or you disable all non-default apps and restart the containers to test this faster. Afterwards enable one for one again)…
Thank you very much @szaimen
I just tried this: I stopped and disabled all the containers optional and then started Nextcloud again. Instead of 30+ minutes, it took 15 minutes.
It’s a big improvement but when I was testing with a a new OS reinstall and a clean Nextcloud AIO instance (before restoring my borg backup) it took less than 5 minutes to start all the containers (including the optional ones).
Is this normal?
2024-09-04T15:25:48.539238546Z Enabling Imagick...
2024-09-04T15:25:50.105556443Z WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/main: No such file or directory
2024-09-04T15:25:50.105790053Z WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/community: No such file or directory
2024-09-04T15:25:50.147087190Z Connection to nextcloud-aio-redis (172.18.0.5) 6379 port [tcp/redis] succeeded!
Yes
Got it.
Thank you.
In case this might be useful, here are the logs that can be found at https://Local-IP:8080/api/docker/logs?id=nextcloud-aio-nextcloud
This is what was shown when I disabled all the optional containers and it took ~15 minutes to start.
2024-09-04T14:20:16.765800513Z Connection to nextcloud-aio-database (172.18.0.2) 5432 port [tcp/postgresql] succeeded!
2024-09-04T14:20:18.802343993Z now
2024-09-04T14:20:18.802366238Z -------------------------------
2024-09-04T14:20:18.802371560Z 2024-09-04 10:20:18.801546-04
2024-09-04T14:20:18.802375853Z (1 row)
2024-09-04T14:20:18.802380034Z
2024-09-04T14:20:18.809449458Z + '[' -f /dev-dri-group-was-added ']'
2024-09-04T14:20:18.810013617Z ++ find /dev -maxdepth 1 -mindepth 1 -name dri
2024-09-04T14:20:18.810653087Z + '[' -n '' ']'
2024-09-04T14:20:18.810666794Z + set +x
2024-09-04T14:20:18.822567914Z Enabling Imagick...
2024-09-04T14:20:20.001270975Z WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/main: No such file or directory
2024-09-04T14:20:20.001294279Z WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/community: No such file or directory
2024-09-04T14:20:20.025352076Z Connection to nextcloud-aio-redis (172.18.0.3) 6379 port [tcp/redis] succeeded!
2024-09-04T14:21:21.203235781Z Applying one-click-instance settings...
2024-09-04T14:21:51.637322417Z System config value one-click-instance set to boolean true
2024-09-04T14:22:22.066859009Z System config value one-click-instance.user-limit set to integer 100
2024-09-04T14:22:52.494031712Z System config value one-click-instance.link set to string https://nextcloud.com/all-in-one/
2024-09-04T14:23:22.921088433Z support already enabled
2024-09-04T14:23:22.935583656Z Adjusting log files...
2024-09-04T14:23:53.343650063Z System config value upgrade.cli-upgrade-link set to string https://github.com/nextcloud/all-in-one/discussions/2726
2024-09-04T14:24:23.769512422Z System config value logfile set to string /var/www/html/data/nextcloud.log
2024-09-04T14:24:54.195325538Z Config value were not updated
2024-09-04T14:25:24.634546874Z System config value updatedirectory set to string /nc-updater
2024-09-04T14:25:55.072875670Z System config value maintenance_window_start set to integer 100
2024-09-04T14:25:55.087168124Z Applying network settings...
2024-09-04T14:26:25.521975372Z System config value allow_local_remote_servers set to boolean true
2024-09-04T14:26:55.969710181Z System config value davstorage.request_timeout set to integer 3600
2024-09-04T14:27:26.415935631Z System config value trusted_domains => 1 set to string secret.supersecurenc.org
2024-09-04T14:27:56.865110878Z System config value overwrite.cli.url set to string https://secret.supersecurenc.org/
2024-09-04T14:28:27.314353720Z System config value htaccess.RewriteBase set to string /
2024-09-04T14:28:57.760169624Z .htaccess has been updated
2024-09-04T14:29:28.207668112Z System config value dbpersistent set to boolean false
2024-09-04T14:29:58.658580743Z System config value auth.bruteforce.protection.enabled set to boolean true
2024-09-04T14:30:29.106472652Z System config value ratelimit.protection.enabled set to boolean true
2024-09-04T14:30:59.552247305Z System config value files_external_allow_create_new_local set to boolean false
2024-09-04T14:32:30.926394011Z notify_push is up-to-date or no updates could be found
2024-09-04T14:33:01.372228561Z System config value trusted_proxies => 0 set to string 127.0.0.1
2024-09-04T14:33:31.820075931Z System config value trusted_proxies => 1 set to string ::1
2024-09-04T14:34:02.293169031Z System config value trusted_proxies => 10 set to string 172.18.0.0/16
2024-09-04T14:34:32.740960256Z Config value were not updated
2024-09-04T14:35:03.194232017Z spreed disabled
2024-09-04T14:35:03.270940188Z spreed 19.0.8 removed
2024-09-04T14:35:03.690852813Z files_antivirus disabled
2024-09-04T14:35:03.705141728Z files_antivirus 5.5.7 removed
2024-09-04T14:35:04.531646168Z app_api disabled
2024-09-04T14:35:04.554126992Z app_api 3.1.0 removed
2024-09-04T14:35:04.590038774Z + '[' true = true ']'
2024-09-04T14:35:04.590066802Z + '[' 443 = 443 ']'
2024-09-04T14:35:04.590722351Z ++ dig nextcloud-aio-apache A +short +search
2024-09-04T14:35:04.591487268Z ++ grep '^[0-9.]\+$'
2024-09-04T14:35:04.591501229Z ++ sort
2024-09-04T14:35:04.591506599Z ++ head -n1
2024-09-04T14:35:04.611964152Z + IPv4_ADDRESS_APACHE=172.18.0.6
2024-09-04T14:35:04.612542037Z ++ dig nextcloud-aio-apache AAAA +short +search
2024-09-04T14:35:04.613557093Z ++ grep '^[0-9a-f:]\+$'
2024-09-04T14:35:04.613573851Z ++ sort
2024-09-04T14:35:04.613579164Z ++ head -n1
2024-09-04T14:35:04.633659699Z + IPv6_ADDRESS_APACHE=
2024-09-04T14:35:04.634239743Z ++ dig nextcloud-aio-mastercontainer A +short +search
2024-09-04T14:35:04.634253873Z ++ grep '^[0-9.]\+$'
2024-09-04T14:35:04.634388757Z ++ sort
2024-09-04T14:35:04.634559419Z ++ head -n1
2024-09-04T14:35:04.654825109Z + IPv4_ADDRESS_MASTERCONTAINER=172.18.0.14
2024-09-04T14:35:04.655388850Z ++ dig nextcloud-aio-mastercontainer AAAA +short +search
2024-09-04T14:35:04.655402361Z ++ grep '^[0-9a-f:]\+$'
2024-09-04T14:35:04.655535782Z ++ sort
2024-09-04T14:35:04.655640240Z ++ head -n1
2024-09-04T14:35:04.676997840Z + IPv6_ADDRESS_MASTERCONTAINER=
2024-09-04T14:35:04.677024398Z + sed -i 's|^;listen.allowed_clients|listen.allowed_clients|' /usr/local/etc/php-fpm.d/www.conf
2024-09-04T14:35:04.678817406Z + sed -i 's|listen.allowed_clients.*|listen.allowed_clients = 127.0.0.1,::1,172.18.0.6,,172.18.0.14,|' /usr/local/etc/php-fpm.d/www.conf
2024-09-04T14:35:04.680864980Z + sed -i '/^listen.allowed_clients/s/,,/,/g' /usr/local/etc/php-fpm.d/www.conf
2024-09-04T14:35:04.683401835Z + sed -i '/^listen.allowed_clients/s/,$//' /usr/local/etc/php-fpm.d/www.conf
2024-09-04T14:35:04.685008053Z + grep listen.allowed_clients /usr/local/etc/php-fpm.d/www.conf
2024-09-04T14:35:04.685959643Z listen.allowed_clients = 127.0.0.1,::1,172.18.0.6,172.18.0.14
2024-09-04T14:35:04.686118626Z + set +x
2024-09-04T14:35:06.026459339Z [04-Sep-2024 10:35:06] NOTICE: fpm is running, pid 680
2024-09-04T14:35:06.026488978Z [04-Sep-2024 10:35:06] NOTICE: ready to handle connections
About these logs, is there any way to get more detailed logs? Like putting this in debug mode or something like that to see all the steps? Because I think something in here is causing the issue because when I stop and then start all the containers and I check the logs for each one, all the containers are either waiting for the Nextcloud container to start or are already started.
each of the 4 steps “System config value” steps runs for 30 sec… the 5th step takes 1:30 (likely 3x30sec) not talking about the fact it usually takes on fragment of a second - 30 sec sounds for me like some timeout… is there some networking layer in between or some strange storage config which takes longer to confirm the write?
likely the problem remains once the system is up and running and you can reproduce the issue running some occ command… this gives you a chance to dig deeper into the problem once the system runs…
@szaimen - “System config value” performs just a simple config manipulation or is there more logic like networking operations included?
Hello there @wwe
Thank you for commenting on my post.
Yeah, I noticed that every single line in the log takes around 30 seconds each to complete. The networking has not change at all. I’ve been self hosting Nextcloud for a long time and just recently it started doing this.
Any occ command you want me to run to try to test if it take 30 seconds?
I would change some meaningless value… one of the mentioned is fine as well e.g. switch bruteforce protection off an on again… maybe the problem occurs already when you read the config… try to isolate the smallest possible operation where the delay is visible - later we can try to break down…
I tested an occ command and it worked right away. It didn’t take 30 seconds.
this a real mystery… I have no explanation why your interactive command runs fast while the one from AiO is slow… there seems to be nothing else - exactly the same command is executed in the entrypoint.sh:
do you have any resource limiting on your installation? which could exceed during massive operations and relax later during normal operations?
further look at you log shows not all operation are delayed in similar way…
until here it runs fast… the problem seems to start here:
do you see anything strange like high resource usage on redis container during this time? try running docker ps, docker compose stats or maybe redis-cli (e.g. docker compose exec -ti redis redis-cli MONITOR) to analyze if there is some excessive activity…
if not try to setup another AiO instance with another FQDN, fresh storage and minimum of functionality to verify the problem occurs there as well.
As I mentioned earlier, no. This is a standard AIO installation. I followed the instructions that can be found here.
Nothing is limiting Nextcloud since Nextcloud is the only thing installed in this VM.
If you read my previous comments you’ll see that I’ve already tried testing with a clean installation and even a clean OS install and everything works fine as it should. But when I restore the borg backup, everything slows down again.
In this case the problem is in the backup… there is not many things which could cause this problem - most likely the config. It should be not hard to work through the config setting by setting and find the difference.
@wwe Here is my configuration file.
<?php
$CONFIG = array (
'one-click-instance' => true,
'one-click-instance.user-limit' => 100,
'memcache.local' => '\\OC\\Memcache\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
'appsallowlist' => false,
'check_data_directory_permissions' => false,
'memcache.distributed' => '\\OC\\Memcache\\Redis',
'memcache.locking' => '\\OC\\Memcache\\Redis',
'redis' =>
array (
'host' => 'nextcloud-aio-redis',
'password' => 'ThisPasswordIsVerySecure',
'port' => 6379,
),
'overwritehost' => 'secret.supersecurenc.org',
'overwriteprotocol' => 'https',
'passwordsalt' => 'SomeSaltHash',
'secret' => 'IWillTellYouThisSecret',
'trusted_domains' =>
array (
0 => 'localhost',
1 => 'secret.supersecurenc.org',
),
'datadirectory' => '/mnt/ncdata',
'dbtype' => 'pgsql',
'version' => '29.0.4.1',
'overwrite.cli.url' => 'https://secret.supersecurenc.org/',
'dbname' => 'nextcloud_database',
'dbhost' => 'nextcloud-aio-database',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => 'oc_nextcloud',
'dbpassword' => 'WellThisIsTheSuperSecureDBPassword',
'installed' => true,
'instanceid' => 'AllThemIdsEverywhere',
'maintenance' => false,
'loglevel' => 2,
'log_type' => 'file',
'logfile' => '/var/www/html/data/nextcloud.log',
'log_rotate_size' => '10485760',
'log.condition' =>
array (
'apps' =>
array (
0 => 'admin_audit',
),
),
'preview_max_x' => 2048,
'preview_max_y' => 2048,
'jpeg_quality' => 60,
'enabledPreviewProviders' =>
array (
1 => 'OC\\Preview\\Image',
2 => 'OC\\Preview\\MarkDown',
3 => 'OC\\Preview\\MP3',
4 => 'OC\\Preview\\TXT',
5 => 'OC\\Preview\\OpenDocument',
6 => 'OC\\Preview\\Movie',
7 => 'OC\\Preview\\Krita',
0 => 'OC\\Preview\\Imaginary',
),
'enable_previews' => true,
'upgrade.disable-web' => true,
'mail_smtpmode' => 'smtp',
'trashbin_retention_obligation' => 'auto, 30',
'versions_retention_obligation' => 'auto, 30',
'activity_expire_days' => '30',
'simpleSignUpLink.shown' => false,
'share_folder' => '/Shared',
'one-click-instance.link' => 'https://nextcloud.com/all-in-one/',
'upgrade.cli-upgrade-link' => 'https://github.com/nextcloud/all-in-one/discussions/2726',
'updatedirectory' => '/nc-updater',
'maintenance_window_start' => 100,
'allow_local_remote_servers' => true,
'davstorage.request_timeout' => 3600,
'htaccess.RewriteBase' => '/',
'dbpersistent' => false,
'files_external_allow_create_new_local' => false,
'trusted_proxies' =>
array (
0 => '127.0.0.1',
1 => '::1',
10 => '172.18.0.0/16',
),
'preview_imaginary_key' => 'HashForImaginaryKey',
'default_phone_region' => 'US',
'mail_smtpsecure' => 'ssl',
'mail_sendmailmode' => 'smtp',
'mail_from_address' => 'mailer',
'mail_smtphost' => 'RandomSmtpHost.com',
'mail_domain' => 'super-securenc.org',
'mail_smtpport' => '123',
'mail_smtpauth' => 1,
'mail_smtpname' => 'mailtome@supersecurenc.org',
'mail_smtppassword' => '12345qwertyNotMyRealPasswordTho',
'memories.db.triggers.fcu' => true,
'memories.exiftool' => '/var/www/html/custom_apps/memories/bin-ext/exiftool-amd64-musl',
'memories.vod.path' => '/var/www/html/custom_apps/memories/bin-ext/go-vod-amd64',
'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
'memories.vod.ffprobe' => '/usr/bin/ffprobe',
'auth.bruteforce.protection.enabled' => true,
'ratelimit.protection.enabled' => true,
'data-fingerprint' => 'FingerprintHash',
'preview_imaginary_url' => 'http://nextcloud-aio-imaginary:9000',
);
Did you find anything in my configuration file that could be causing the delays?