Nextcloud AIO v7.11.2 non-responsive / High CPU every 13 days (network locationon host for ncdata)

sunnyd24 · January 11, 2024, 8:36pm

Details
Nextcloud version (eg, 20.0.5): 27.1.5
Operating system and version (eg, Ubuntu 20.04): Windows 11 HyperV with Ubuntu 22.04.1
Apache or nginx version (eg, Apache 2.4.25): n/a
PHP version (eg, 7.4): 8.1.27

The issue you are facing:

No Nextcloud service available on any clients due to webservice Apache “stopped”.

Nextcloud seems to be using high CPU (50-70%) and high memory usage (27.5GB) in HyperV, see screenshot below:

Slow responses from Windows host, HyperV VM Ubuntu OS, master container website, HyperV console, and SSH console.

See below for statuses during Nextcloud error:

nextcloud-aio-apache [Stopped] - see screenshot below

All others docker services running apart from:

nextcloud-aio-nextcloud [Starting] - "Waiting for nextcloud-aio-apache to start..."
nextcloud-aio-notify-push [Starting] - "Waiting for Nextcloud to start..."
nextcloud-aio-docker-socket-proxy - [Starting] - "Waiting for Nextcloud to start..."

Further investigation on Ubuntu VM shows:
‘top’ command shows “janus” using significant VM CPU, see below screenshot.

Suspect Janus is causing the high CPU usage. Is this a WebRTC service used for Talk server related to this running process?
Not sure how this High CPU usage of Janus process stops Apache & Nextcloud from running…or how AIO gets stuck in this way?

MasterContainer log:

I saw there were container updates available on master container webpage, I have clicked on “stop containers”, it got stuck with the waiting blue circle.

‘sudo docker ps’ shows unhealthy statuses from nextcloud containers:

So forced the HyperV VM to turn off, and turned back on.
Then checking master container status, Apache was still showing stopped so I pressed stop but got stuck with waiting blue circle.
Checking ‘sudo docker ps’ showed nextcloud-aio-nextcloud and nextcloud-aio-mastercontainer was ‘up’.
Ran ‘sudo docker stop nextcloud-aio-mastercontainer’, but nextcloud-aio-nextcloud was still ‘up’.
Ran ‘sudo docker stop nextcloud-aio-nextcloud’, no more running nextcloud docker processes.
Ran ‘sudo docker rm nextcloud-aio-mastercontainer’.
Ran my ‘sudo docker run’ config for master container. Finally all containers were at stopped status.
Click 'Start and update containers", Hooray, finally all containers started successfully with CPU usage ar 0% and memory usage at 11GB.

Is this the first time you’ve seen this error? No, it happened 13 days ago too.

Steps to replicate it:

Leave it running for 10+ days, run backup every night at 3am.
Use Nextcloud services as normal for client file access from day to day using webDAV and Nextcloud client on windows PC and android phone.
Mainly sync photos/videos from phone overnight using FolderSync.
See above steps to get Nextcloud AIO operational again.

The output of your Nextcloud error log over the last 3 days:

Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T20:09:56+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T19:05:15+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T18:00:09+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T16:55:05+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T15:51:05+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T02:43:17+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T01:38:28+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-11T00:33:16+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T23:33:13+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T22:28:23+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T21:23:24+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T20:18:19+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T19:13:17+0000
Error	mail	Horde_Imap_Client_Exception: Mail server denied authentication.		2024-01-10T18:28:44+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T18:08:33+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T17:08:26+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T16:08:16+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T15:03:16+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T13:58:24+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T12:53:15+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T11:48:17+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T10:48:15+0000
Warning	appstoreFetcher	Could not connect to appstore: cURL error 28: Operation timed out after 60000 milliseconds with 3407872 out of 5957070 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json		2024-01-10T10:15:12+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T09:43:17+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T08:43:12+0000
Warning	appstoreFetcher	Could not connect to appstore: cURL error 28: Operation timed out after 60002 milliseconds with 3375254 out of 5955869 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json		2024-01-10T07:53:15+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T07:38:26+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T06:33:12+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T05:28:12+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T04:23:19+0000
Error	richdocuments	GuzzleHttp\Exception\ConnectException: cURL error 35: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://REDACTED.HOST.COM/hosting/capabilities		2024-01-10T03:23:26+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T03:23:15+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T02:13:49+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T01:08:50+0000
Warning	appstoreFetcher	Could not connect to appstore: cURL error 35: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json		2024-01-10T01:00:05+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-10T00:03:46+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T22:58:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T21:53:46+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T20:48:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T19:43:46+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T18:38:52+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T17:33:49+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T16:28:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T15:23:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T14:23:46+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T13:18:51+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T12:13:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T11:08:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T10:03:47+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T08:58:51+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T07:53:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T06:48:45+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T05:43:45+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T04:38:48+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T03:33:45+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T02:29:22+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T01:29:19+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-09T00:24:21+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-08T23:24:19+0000

The output of your Docker config file:

sudo docker run \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart always \
--publish 80:80 \
--publish 8080:8080 \
--publish 8443:8443 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
--env NEXTCLOUD_DATADIR="/mnt/nc_data" \
--env NEXTCLOUD_MOUNT="/mnt/" \
--env SKIP_DOMAIN_VALIDATION=false \
--env NEXTCLOUD_ENABLE_DRI_DEVICE=false \
--env NEXTCLOUD_TRUSTED_DOMAINS="" \
--env SMTP_HOST="REDACTED" \
--env SMTP_SECURE=465 \
--env SMTP_NAME="REDACTEDUSER" \
--env SMTP_PASSWORD="REDACTED" \
--env MAIL_FROM_ADDRESS="REDACTED@HOST.COM" \
--env DICTIONARIES="en_GB" \
--env NEXTCLOUD_MEMORY_LIMIT=2048M \
--env NEXTCLOUD_UPLOAD_LIMIT=50G \
--env NEXTCLOUD_MAX_TIME=3600 \
--env BORG_RETENTION_POLICY="--keep-within=7d --keep-weekly=4 --keep-monthly=6" \
nextcloud/all-in-one:latest

The output of your docker Apache logs:

sudo docker logs --since=4h nextcloud-aio-apache
Error response from daemon: No such container: nextcloud-aio-apache

Please can someone help me identify what is going wrong approx. every 10+ days?
Why does it take all this effort to fix when things go wrong?
Surely there is a better way for Nextcloud AIO to detect errors/issues and attempt to fix them?

jtr · January 11, 2024, 9:26pm

Based on that top output, janus is hardly using any CPU (and almost no memory). But your load average is 2369.67 (!!!) and you have 2626 processes running so something is definitely messed up. Also, top shows nearly 10G of RAM is completely unused.

Can you take a closer look at your process list? Those 2000+ processes will give some clues maybe. Unfortunately you’ll have to wait for it to re-occur it sounds like.

What version of Docker Engine?
What apps do you have installed?
Can you share your NC config?

sunnyd24 · January 21, 2024, 4:40pm

@jtr Thanks for your time.

Good point on the 2000+ processes, I wish I had taken a full log of these when it occurred.

However, I have started to see my host CPU vmmem hovering between 1% - 15%, so starting to exhibit some of this behaviour after 10 days uptime again, but not as severe as last time yet…

Some htop dumps shows the following processes:

User                        Command                      CPU%
1. ncadmin - /lib/systemd/systemd --system --deserialize [29.5%]
2. root - runc --root /var/run/docker/runtime-runc/moby --log /run/containerd.runtme.v2.task/moby/6cf8f................................. [13.1%]
3. ncadmin - janus --config=/user/local/etc/janus/janus.jcfg --disable-colors --log-stdout --full-trickle --debug-level 3 [1.3%]

Any thoughts?

I am using AIO 7.9.1:

Docker engine: 24.0.5, build ced0996
Apps list:

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - app_api: 1.4.6
  - bruteforcesettings: 2.7.0
  - calendar: 4.6.4
  - camerarawpreviews: 0.8.4
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.5.1
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.11.3
  - external: 5.2.1
  - facerecognition: 0.9.31
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_antivirus: 5.4.1
  - files_automatedtagging: 1.17.0
  - files_external: 1.19.0
  - files_fulltextsearch: 27.0.1
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - fulltextsearch: 27.0.3
  - fulltextsearch_elasticsearch: 27.0.5
  - groupfolders: 15.3.4
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - mail: 3.5.4
  - maps: 1.2.0
  - memories: 6.2.2
  - nextcloud-aio: 0.4.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.9.2
  - notifications: 2.15.0
  - notify_push: 0.6.7
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - previewgenerator: 5.4.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - richdocuments: 8.2.4
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - spreed: 17.1.4
  - support: 1.10.0
  - survey_client: 1.15.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - encryption: 2.15.0
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0

NC AIO config:

sudo docker run \
--sig-proxy=false \
--name nextcloud-aio-mastercontainer \
--restart always \
--publish 80:80 \
--publish 8080:8080 \
--publish 8443:8443 \
--volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
--volume /var/run/docker.sock:/var/run/docker.sock:ro \
--env NEXTCLOUD_DATADIR="/mnt/nc_data" \
--env NEXTCLOUD_MOUNT="/mnt/" \
--env SKIP_DOMAIN_VALIDATION=false \
--env NEXTCLOUD_ENABLE_DRI_DEVICE=false \
--env NEXTCLOUD_TRUSTED_DOMAINS="" \
--env SMTP_HOST="REDACTED" \
--env SMTP_SECURE=465 \
--env SMTP_NAME="REDACTEDUSER" \
--env SMTP_PASSWORD="REDACTED" \
--env MAIL_FROM_ADDRESS="REDACTED@HOST.COM" \
--env DICTIONARIES="en_GB" \
--env NEXTCLOUD_MEMORY_LIMIT=2048M \
--env NEXTCLOUD_UPLOAD_LIMIT=50G \
--env NEXTCLOUD_MAX_TIME=3600 \
--env BORG_RETENTION_POLICY="--keep-within=7d --keep-weekly=4 --keep-monthly=6" \
nextcloud/all-in-one:latest

sunnyd24 · January 23, 2024, 12:30pm

Further update:

htop tree shows 4 running threads all maxing out the CPU with load average at 40 after 11 days uptime

It seems there are several processes killing the system:

User                 Command                                         CPU%
ncadmin              /lib/systemd/systemd --system --deserialize     99.5%
ncadmin              |_ /lib/systemd/systemd --user                  94-100%
                        |_(sd-pam)                                   0%

root                 |_(ionclean)                                    70-80%

Here are some htop tree screenshots 4 seconds apart:

Nextcloud logs:

Error	cron	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 7 no connection to the server		2024-01-22T03:00:54+0000
Error	core	Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 7 no connection to the server		2024-01-22T03:00:54+0000
Error	core	OC\DB\Exceptions\DbalException: An exception occurred while executing a query: SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "gf_versions_uniq_index" DETAIL: Key (file_id, "timestamp")=(8851, 1691150951) already exists.		2024-01-20T02:45:03+0000

Some further investigations:

$sudo ps -eo pid,comm,unit   
   1824 systemd         user@1000.service
   1833 (sd-pam)        user@1000.service
   172935 (ionclean)      phpsessionclean.service

$sudo ps -aux
    ncadmin     1824  6.5  0.1  33288 21344 ?        Rs   Jan11 1124:48 /lib/systemd/systemd --user
    ncadmin     1833  0.0  0.0 169516  1680 ?        S    Jan11   0:00 (sd-pam)
    root      172935 78.3  0.3 198956 39124 ?        Rs   12:32   8:42 (ionclean)

Any ideas what (sd-pam) and (ionclean) do?

This is seriously affecting access to Nextcloud and slowing down my host PC?

sunnyd24 · February 6, 2024, 4:00pm

Final request for support before I move to another service:

Issue seems to be traced to a ncadmin & root processes being run after first boot with other processes being run from this master systemd --user process after uptime has reached 13 days again, it’s repeatable!

In the latest CPU hog issue, it seems that:

Hung process id 1 "/sbin/init seems to be causing an issue. The process was run 35h 52mins ago, and uses 95-100% CPU on first core.
Hung process id 2294 “/lib/systemd/systemd --user” seems to be causing an issue. The process was run 35h 56mins ago, and uses 95-100% CPU on second core.
The processes above occasionally invokes child processes running such as (find) and (ionclean) which also ends up using 100% on third core.

root           1 11.0  0.4 199184 42000 ?        Rs   Jan24 2134:10 /sbin/init
ncadmin     2294 10.9  0.3  49152 39940 ?        Rs   Jan24 2103:47 /lib/systemd/systemd --user
ncadmin     2295  0.0  0.0 169508  1976 ?        S    Jan24   0:00 (sd-pam)

When in this state Nextcloud is unable to communicate with the outside world with DNS errors showing up in the syslog and externally connecting to hostname either through browser or Android app does not work. Even the AIO master container runs super slow when trying to log in with AIO password.

Using an strace, it seems it process id 2294 is trying to read from the network location --env NEXTCLOUD_DATADIR="/mnt/nc_data" which is mapped on Ubuntu to a SMB location on Windows host //192.168.1.50/H/NextCloud_Data on loop but I can’t see any errors logged.

read(22, "closetimeo=1\n175253 396693 0:45 "..., 1024) = 1024
read(22, "size=1048576,echo_interval=60,ac"..., 1024) = 1024
read(22, "mfsymlinks,rsize=4194304,wsize=4"..., 1024) = 1024
read(22, "7,seal,soft,nounix,serverino,map"..., 1024) = 1024
read(22, "192.168.1.50,file_mode=0777,dir_"..., 1024) = 1024
read(22, "uid=33,noforceuid,gid=0,noforceg"..., 1024) = 1024
read(22, "3.1.1,cache=strict,username=REDACTED"..., 1024) = 1024
read(22, "//192.168.1.50/H/NextCloud_Data "..., 1024) = 1024
read(22, "/ncdata rw,relatime shared:346 -"..., 1024) = 1024
read(22, ",closetimeo=1\n175175 396615 0:45"..., 1024) = 1024

Using strace on (sd-pam) process:
rt_sigtimedwait([TERM],

Using strace on /sbin/init process:

read(57, "time shared:346 - cifs //192.168"..., 1024) = 1024
read(57, "831 43508 0:45 / /mnt/nc_data rw"..., 1024) = 1024
read(57, "nterval=60,actimeo=1,closetimeo="..., 1024) = 1024
read(57, "4,wsize=4194304,bsize=1048576,ec"..., 1024) = 1024
read(57, "rino,mapposix,mfsymlinks,rsize=4"..., 1024) = 1024
read(57, ",dir_mode=0777,seal,soft,nounix,"..., 1024) = 1024
read(57, "id,addr=192.168.1.50,file_mode=0"..., 1024) = 1024
read(57, "sara,uid=33,noforceuid,gid=0,nof"..., 1024) = 1024
read(57, "s=3.1.1,cache=strict,username=su"..., 1024) = 1024
read(57, "/192.168.1.50/H/NextCloud_Data r"..., 1024) = 1024
read(57, "_data rw,relatime shared:346 - c"..., 1024) = 1024

Partial syslog dump in case it helps (for info at 3am borgbackup runs):

...
Feb  6 00:47:04 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 01:09:01 nextcloud CRON[1391003]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 01:17:01 nextcloud CRON[1396771]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb  6 01:20:14 nextcloud systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb  6 01:20:14 nextcloud systemd[1]: Finished Clean php session files.
Feb  6 01:20:14 nextcloud systemd[1]: phpsessionclean.service: Consumed 26min 22.893s CPU time.
Feb  6 01:20:14 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 01:39:01 nextcloud CRON[1412663]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 01:39:41 nextcloud systemd[1]: logrotate.service: Deactivated successfully.
Feb  6 01:39:41 nextcloud systemd[1]: Finished Rotate log files.
Feb  6 01:39:41 nextcloud systemd[1]: logrotate.service: Consumed 1h 19min 45.000s CPU time.
Feb  6 01:53:53 nextcloud systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb  6 01:53:53 nextcloud systemd[1]: Finished Clean php session files.
Feb  6 01:53:53 nextcloud systemd[1]: phpsessionclean.service: Consumed 28min 48.083s CPU time.
Feb  6 01:53:53 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 02:02:49 nextcloud systemd[1]: Starting Cleanup of Temporary Directories...
Feb  6 02:02:49 nextcloud systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Feb  6 02:02:49 nextcloud systemd[1]: Finished Cleanup of Temporary Directories.
Feb  6 02:09:01 nextcloud CRON[1435175]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 02:17:01 nextcloud CRON[1441217]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb  6 02:29:43 nextcloud systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb  6 02:29:43 nextcloud systemd[1]: Finished Clean php session files.
Feb  6 02:29:43 nextcloud systemd[1]: phpsessionclean.service: Consumed 33min 43.484s CPU time.
Feb  6 02:29:43 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 02:39:01 nextcloud CRON[1457700]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 02:57:49 nextcloud systemd[1]: Starting Message of the Day...
Feb  6 02:57:49 nextcloud systemd[1]: motd-news.service: Deactivated successfully.
Feb  6 02:57:49 nextcloud systemd[1]: Finished Message of the Day.
Feb  6 03:00:33 nextcloud dockerd[941]: time="2024-02-06T03:00:33.898821600Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:00:34 nextcloud NetworkManager[686]: <info>  [1707188434.2932] manager: (vethfb0f763): new Veth device (/org/freedesktop/NetworkManager/Devices/553)
Feb  6 03:00:34 nextcloud kernel: [1127611.909258] br-b164f87cd003: port 13(veth300fe85) entered blocking state
Feb  6 03:00:34 nextcloud kernel: [1127611.909262] br-b164f87cd003: port 13(veth300fe85) entered disabled state
Feb  6 03:00:34 nextcloud kernel: [1127611.909867] device veth300fe85 entered promiscuous mode
Feb  6 03:00:34 nextcloud NetworkManager[686]: <info>  [1707188434.2938] manager: (veth300fe85): new Veth device (/org/freedesktop/NetworkManager/Devices/554)
Feb  6 03:00:34 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 380 seen, reloading interface list
Feb  6 03:00:34 nextcloud systemd-udevd[1473813]: Using default interface naming scheme 'v249'.
Feb  6 03:00:34 nextcloud dockerd[941]: time="2024-02-06T03:00:34.432046000Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:00:34 nextcloud dockerd[941]: time="2024-02-06T03:00:34.432081600Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:00:34 nextcloud systemd-networkd[633]: veth300fe85: Link UP
Feb  6 03:00:34 nextcloud systemd-udevd[1473811]: Using default interface naming scheme 'v249'.
Feb  6 03:00:34 nextcloud dockerd[941]: time="2024-02-06T03:00:34.564589800Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:00:34 nextcloud containerd[727]: time="2024-02-06T03:00:34.678573700Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:00:34 nextcloud containerd[727]: time="2024-02-06T03:00:34.680736900Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:00:34 nextcloud containerd[727]: time="2024-02-06T03:00:34.680749900Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:00:34 nextcloud containerd[727]: time="2024-02-06T03:00:34.681698800Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/2a808d1ede917eed619ea6d84169d7c47be7146d5fae606ec48ae3b18cdc15c9 pid=1473839 runtime=io.containerd.runc.v2
Feb  6 03:01:51 nextcloud systemd[1]: Started libcontainer container 2a808d1ede917eed619ea6d84169d7c47be7146d5fae606ec48ae3b18cdc15c9.
Feb  6 03:02:52 nextcloud kernel: [1127749.899903] eth0: renamed from vethfb0f763
Feb  6 03:02:52 nextcloud kernel: [1127749.917138] IPv6: ADDRCONF(NETDEV_CHANGE): veth300fe85: link becomes ready
...
Feb  6 03:09:01 nextcloud CRON[1479153]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 03:09:01 nextcloud systemd[1]: docker-055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0.scope: Deactivated successfully.
Feb  6 03:09:01 nextcloud systemd[1]: docker-055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0.scope: Consumed 12min 25.231s CPU time.
Feb  6 03:09:01 nextcloud dockerd[941]: time="2024-02-06T03:09:01.840663000Z" level=info msg="ignoring event" container=055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb  6 03:09:01 nextcloud containerd[727]: time="2024-02-06T03:09:01.841515600Z" level=info msg="shim disconnected" id=055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0
Feb  6 03:09:01 nextcloud containerd[727]: time="2024-02-06T03:09:01.841556900Z" level=warning msg="cleaning up after shim disconnected" id=055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0 namespace=moby
Feb  6 03:09:01 nextcloud containerd[727]: time="2024-02-06T03:09:01.841566200Z" level=info msg="cleaning up dead shim"
Feb  6 03:09:01 nextcloud containerd[727]: time="2024-02-06T03:09:01.851447300Z" level=warning msg="cleanup warnings time=\"2024-02-06T03:09:01Z\" level=info msg=\"starting signal loop\" namespace=moby pid=1479154 runtime=io.containerd.runc.v2\n"
Feb  6 03:09:01 nextcloud dockerd[941]: time="2024-02-06T03:09:01.852388100Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0 daemonShuttingDown=false error="restart canceled" execDuration=23h46m6.0263727s exitStatus="{137 2024-02-06 03:08:55.8671324 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Feb  6 03:09:02 nextcloud kernel: [1128120.515667] br-b164f87cd003: port 12(vethed3f68f) entered disabled state
Feb  6 03:09:02 nextcloud systemd-networkd[633]: vethed3f68f: Lost carrier
Feb  6 03:09:02 nextcloud kernel: [1128120.516923] vethb94328b: renamed from eth0
Feb  6 03:09:03 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 378 seen, reloading interface list
Feb  6 03:09:03 nextcloud systemd-udevd[1479184]: Using default interface naming scheme 'v249'.
Feb  6 03:09:03 nextcloud NetworkManager[686]: <info>  [1707188943.2020] manager: (vethb94328b): new Veth device (/org/freedesktop/NetworkManager/Devices/566)
Feb  6 03:09:03 nextcloud systemd-networkd[633]: vethed3f68f: Link DOWN
Feb  6 03:09:03 nextcloud kernel: [1128120.825875] br-b164f87cd003: port 12(vethed3f68f) entered disabled state
Feb  6 03:09:03 nextcloud kernel: [1128120.830551] device vethed3f68f left promiscuous mode
Feb  6 03:09:03 nextcloud kernel: [1128120.830557] br-b164f87cd003: port 12(vethed3f68f) entered disabled state
Feb  6 03:09:03 nextcloud NetworkManager[686]: <info>  [1707188943.3694] device (vethed3f68f): released from master device br-b164f87cd003
Feb  6 03:09:05 nextcloud dockerd[941]: time="2024-02-06T03:09:05.858742200Z" level=warning msg="Container failed to exit within 10s of kill - trying direct SIGKILL" container=055635f5b1c219ecba39de6049967c0d410542de2b5eb1cca6ff81f98e4928d0 error="context deadline exceeded"
Feb  6 03:09:07 nextcloud systemd[1]: run-docker-netns-55f08688a982.mount: Deactivated successfully.
Feb  6 03:09:07 nextcloud systemd[1]: var-lib-docker-overlay2-d96e4bc761fb310331dfc87e3b6d3f80a0c0a6ac6a4e43f3871c5d6a559b3b10-merged.mount: Deactivated successfully.
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.466887000Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.466933000Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:09:23 nextcloud NetworkManager[686]: <info>  [1707188963.5790] manager: (veth42f9e40): new Veth device (/org/freedesktop/NetworkManager/Devices/567)
Feb  6 03:09:23 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 382 seen, reloading interface list
Feb  6 03:09:23 nextcloud NetworkManager[686]: <info>  [1707188963.5801] manager: (veth8a032db): new Veth device (/org/freedesktop/NetworkManager/Devices/568)
Feb  6 03:09:23 nextcloud systemd-udevd[1479251]: Using default interface naming scheme 'v249'.
Feb  6 03:09:23 nextcloud systemd-udevd[1479252]: Using default interface naming scheme 'v249'.
Feb  6 03:09:23 nextcloud systemd-networkd[633]: veth8a032db: Link UP
Feb  6 03:09:23 nextcloud kernel: [1128141.245986] br-b164f87cd003: port 2(veth8a032db) entered blocking state
Feb  6 03:09:23 nextcloud kernel: [1128141.245991] br-b164f87cd003: port 2(veth8a032db) entered disabled state
Feb  6 03:09:23 nextcloud kernel: [1128141.246057] device veth8a032db entered promiscuous mode
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.645664200Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.645692500Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.691719000Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:09:23 nextcloud dockerd[941]: time="2024-02-06T03:09:23.691750900Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:09:23 nextcloud containerd[727]: time="2024-02-06T03:09:23.720417600Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:09:23 nextcloud containerd[727]: time="2024-02-06T03:09:23.720580600Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:09:23 nextcloud containerd[727]: time="2024-02-06T03:09:23.720603600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:09:23 nextcloud containerd[727]: time="2024-02-06T03:09:23.720752500Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5 pid=1479294 runtime=io.containerd.runc.v2
Feb  6 03:09:31 nextcloud systemd[1]: Started libcontainer container 21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5.
Feb  6 03:09:43 nextcloud kernel: [1128161.335294] eth0: renamed from veth42f9e40
Feb  6 03:09:43 nextcloud systemd-networkd[633]: veth8a032db: Gained carrier
Feb  6 03:09:43 nextcloud kernel: [1128161.448813] IPv6: ADDRCONF(NETDEV_CHANGE): veth8a032db: link becomes ready
Feb  6 03:09:43 nextcloud kernel: [1128161.448853] br-b164f87cd003: port 2(veth8a032db) entered blocking state
Feb  6 03:09:43 nextcloud kernel: [1128161.448855] br-b164f87cd003: port 2(veth8a032db) entered forwarding state
Feb  6 03:09:43 nextcloud NetworkManager[686]: <info>  [1707188983.8388] device (veth8a032db): carrier: link connected
Feb  6 03:09:45 nextcloud systemd-networkd[633]: veth8a032db: Gained IPv6LL
Feb  6 03:10:01 nextcloud CRON[1479525]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Feb  6 03:12:04 nextcloud systemd[1]: Starting Daily man-db regeneration...
Feb  6 03:17:01 nextcloud CRON[1481147]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb  6 03:23:38 nextcloud systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb  6 03:23:38 nextcloud systemd[1]: Finished Clean php session files.
Feb  6 03:23:38 nextcloud systemd[1]: phpsessionclean.service: Consumed 16min 15.309s CPU time.
Feb  6 03:23:38 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 03:37:27 nextcloud systemd[1]: docker-21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5.scope: Deactivated successfully.
Feb  6 03:37:27 nextcloud systemd[1]: docker-21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5.scope: Consumed 7min 57.307s CPU time.
Feb  6 03:37:27 nextcloud dockerd[941]: time="2024-02-06T03:37:27.349772700Z" level=info msg="ignoring event" container=21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb  6 03:37:27 nextcloud containerd[727]: time="2024-02-06T03:37:27.351315100Z" level=info msg="shim disconnected" id=21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5
Feb  6 03:37:27 nextcloud containerd[727]: time="2024-02-06T03:37:27.351773500Z" level=warning msg="cleaning up after shim disconnected" id=21a304a510a5e0c8169b0a82dc9e2ec2cd47f7e11bd9aef08a6628c4947f7eb5 namespace=moby
Feb  6 03:37:27 nextcloud containerd[727]: time="2024-02-06T03:37:27.351786700Z" level=info msg="cleaning up dead shim"
Feb  6 03:37:27 nextcloud containerd[727]: time="2024-02-06T03:37:27.360834300Z" level=warning msg="cleanup warnings time=\"2024-02-06T03:37:27Z\" level=info msg=\"starting signal loop\" namespace=moby pid=1485943 runtime=io.containerd.runc.v2\n"
Feb  6 03:37:27 nextcloud systemd-networkd[633]: veth8a032db: Lost carrier
Feb  6 03:37:27 nextcloud kernel: [1129824.990753] br-b164f87cd003: port 2(veth8a032db) entered disabled state
Feb  6 03:37:27 nextcloud kernel: [1129824.992030] veth42f9e40: renamed from eth0
Feb  6 03:37:27 nextcloud NetworkManager[686]: <info>  [1707190647.8933] manager: (veth42f9e40): new Veth device (/org/freedesktop/NetworkManager/Devices/569)
Feb  6 03:37:27 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 382 seen, reloading interface list
Feb  6 03:37:27 nextcloud systemd-udevd[1485955]: Using default interface naming scheme 'v249'.
Feb  6 03:37:27 nextcloud systemd-networkd[633]: veth8a032db: Link DOWN
Feb  6 03:37:27 nextcloud kernel: [1129825.531131] br-b164f87cd003: port 2(veth8a032db) entered disabled state
Feb  6 03:37:27 nextcloud kernel: [1129825.536451] device veth8a032db left promiscuous mode
Feb  6 03:37:27 nextcloud kernel: [1129825.536457] br-b164f87cd003: port 2(veth8a032db) entered disabled state
Feb  6 03:37:28 nextcloud NetworkManager[686]: <info>  [1707190648.1449] device (veth8a032db): released from master device br-b164f87cd003
Feb  6 03:37:32 nextcloud systemd[1]: run-docker-netns-f042f8cd14a8.mount: Deactivated successfully.
Feb  6 03:37:32 nextcloud systemd[1]: var-lib-docker-overlay2-82593c19c2939ede9a47f1c98ee084cb8a5166a9ec6b3392e0705f2ba0d6a575-merged.mount: Deactivated successfully.
Feb  6 03:38:04 nextcloud dockerd[941]: time="2024-02-06T03:38:04.919907300Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:38:05 nextcloud NetworkManager[686]: <info>  [1707190685.3140] manager: (veth783c888): new Veth device (/org/freedesktop/NetworkManager/Devices/570)
Feb  6 03:38:05 nextcloud NetworkManager[686]: <info>  [1707190685.3148] manager: (vethbe3f0e9): new Veth device (/org/freedesktop/NetworkManager/Devices/571)
Feb  6 03:38:05 nextcloud kernel: [1129862.932505] br-b164f87cd003: port 2(vethbe3f0e9) entered blocking state
Feb  6 03:38:05 nextcloud kernel: [1129862.932509] br-b164f87cd003: port 2(vethbe3f0e9) entered disabled state
Feb  6 03:38:05 nextcloud kernel: [1129862.932845] device vethbe3f0e9 entered promiscuous mode
Feb  6 03:38:05 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 384 seen, reloading interface list
Feb  6 03:38:05 nextcloud systemd-networkd[633]: vethbe3f0e9: Link UP
Feb  6 03:38:05 nextcloud systemd-udevd[1486107]: Using default interface naming scheme 'v249'.
Feb  6 03:38:05 nextcloud systemd-udevd[1486106]: Using default interface naming scheme 'v249'.
Feb  6 03:38:05 nextcloud dockerd[941]: time="2024-02-06T03:38:05.347361500Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:38:05 nextcloud dockerd[941]: time="2024-02-06T03:38:05.347634800Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:38:05 nextcloud dockerd[941]: time="2024-02-06T03:38:05.394293400Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:38:05 nextcloud containerd[727]: time="2024-02-06T03:38:05.517629900Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:38:05 nextcloud containerd[727]: time="2024-02-06T03:38:05.517744700Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:38:05 nextcloud containerd[727]: time="2024-02-06T03:38:05.517753300Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:38:05 nextcloud containerd[727]: time="2024-02-06T03:38:05.517893800Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/df7a0bec9d27200c7c70ef0caa5de00e35fd64b122538e0ca15b29f22a360cc7 pid=1486127 runtime=io.containerd.runc.v2
Feb  6 03:38:14 nextcloud systemd[1]: Started libcontainer container df7a0bec9d27200c7c70ef0caa5de00e35fd64b122538e0ca15b29f22a360cc7.
Feb  6 03:38:26 nextcloud kernel: [1129883.891330] eth0: renamed from veth783c888
Feb  6 03:38:26 nextcloud kernel: [1129883.914396] IPv6: ADDRCONF(NETDEV_CHANGE): vethbe3f0e9: link becomes ready
Feb  6 03:38:26 nextcloud kernel: [1129883.914430] br-b164f87cd003: port 2(vethbe3f0e9) entered blocking state
Feb  6 03:38:26 nextcloud kernel: [1129883.914432] br-b164f87cd003: port 2(vethbe3f0e9) entered forwarding state
Feb  6 03:38:26 nextcloud systemd-networkd[633]: vethbe3f0e9: Gained carrier
Feb  6 03:38:26 nextcloud NetworkManager[686]: <info>  [1707190706.3045] device (vethbe3f0e9): carrier: link connected
Feb  6 03:38:27 nextcloud systemd-networkd[633]: vethbe3f0e9: Gained IPv6LL
Feb  6 03:38:50 nextcloud dockerd[941]: time="2024-02-06T03:38:50.254601200Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:38:50 nextcloud NetworkManager[686]: <info>  [1707190730.5689] manager: (veth4842b4e): new Veth device (/org/freedesktop/NetworkManager/Devices/572)
Feb  6 03:38:50 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 386 seen, reloading interface list
Feb  6 03:38:50 nextcloud NetworkManager[686]: <info>  [1707190730.5698] manager: (vethe6f56a9): new Veth device (/org/freedesktop/NetworkManager/Devices/573)
Feb  6 03:38:50 nextcloud systemd-udevd[1486363]: Using default interface naming scheme 'v249'.
Feb  6 03:38:50 nextcloud kernel: [1129908.188947] br-b164f87cd003: port 3(vethe6f56a9) entered blocking state
Feb  6 03:38:50 nextcloud kernel: [1129908.188951] br-b164f87cd003: port 3(vethe6f56a9) entered disabled state
Feb  6 03:38:50 nextcloud kernel: [1129908.189069] device vethe6f56a9 entered promiscuous mode
Feb  6 03:38:50 nextcloud systemd-networkd[633]: vethe6f56a9: Link UP
Feb  6 03:38:50 nextcloud systemd-udevd[1486362]: Using default interface naming scheme 'v249'.
Feb  6 03:38:50 nextcloud dockerd[941]: time="2024-02-06T03:38:50.674258900Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:38:50 nextcloud dockerd[941]: time="2024-02-06T03:38:50.674288300Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:38:50 nextcloud dockerd[941]: time="2024-02-06T03:38:50.943376300Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:38:51 nextcloud containerd[727]: time="2024-02-06T03:38:51.027895800Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:38:51 nextcloud containerd[727]: time="2024-02-06T03:38:51.028015900Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:38:51 nextcloud containerd[727]: time="2024-02-06T03:38:51.028024600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:38:51 nextcloud containerd[727]: time="2024-02-06T03:38:51.031118700Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/e8beafe73aebc3c1ee24be851c4b97f5e6b8c34c371eedb800e2081a457d8a92 pid=1486423 runtime=io.containerd.runc.v2
Feb  6 03:39:01 nextcloud CRON[1486488]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 03:39:03 nextcloud systemd[1]: Started libcontainer container e8beafe73aebc3c1ee24be851c4b97f5e6b8c34c371eedb800e2081a457d8a92.
Feb  6 03:39:19 nextcloud kernel: [1129936.889746] eth0: renamed from veth4842b4e
Feb  6 03:39:19 nextcloud NetworkManager[686]: <info>  [1707190759.3207] device (vethe6f56a9): carrier: link connected
Feb  6 03:39:19 nextcloud systemd-networkd[633]: vethe6f56a9: Gained carrier
Feb  6 03:39:19 nextcloud kernel: [1129936.932726] IPv6: ADDRCONF(NETDEV_CHANGE): vethe6f56a9: link becomes ready
Feb  6 03:39:19 nextcloud kernel: [1129936.932776] br-b164f87cd003: port 3(vethe6f56a9) entered blocking state
Feb  6 03:39:19 nextcloud kernel: [1129936.932778] br-b164f87cd003: port 3(vethe6f56a9) entered forwarding state
Feb  6 03:39:20 nextcloud systemd-networkd[633]: vethe6f56a9: Gained IPv6LL
Feb  6 03:39:23 nextcloud systemd[1]: phpsessionclean.service: Deactivated successfully.
Feb  6 03:39:23 nextcloud systemd[1]: Finished Clean php session files.
Feb  6 03:39:23 nextcloud systemd[1]: phpsessionclean.service: Consumed 14min 9.866s CPU time.
Feb  6 03:39:23 nextcloud systemd[1]: Starting Clean php session files...
Feb  6 03:39:40 nextcloud dockerd[941]: time="2024-02-06T03:39:40.871614600Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:39:41 nextcloud NetworkManager[686]: <info>  [1707190781.0125] manager: (vetha1ffd09): new Veth device (/org/freedesktop/NetworkManager/Devices/574)
Feb  6 03:39:41 nextcloud systemd-udevd[1486891]: Using default interface naming scheme 'v249'.
Feb  6 03:39:41 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 388 seen, reloading interface list
Feb  6 03:39:41 nextcloud systemd-udevd[1486890]: Using default interface naming scheme 'v249'.
Feb  6 03:39:41 nextcloud NetworkManager[686]: <info>  [1707190781.0141] manager: (veth3df102f): new Veth device (/org/freedesktop/NetworkManager/Devices/575)
Feb  6 03:39:41 nextcloud kernel: [1129958.633225] br-b164f87cd003: port 4(veth3df102f) entered blocking state
Feb  6 03:39:41 nextcloud kernel: [1129958.633251] br-b164f87cd003: port 4(veth3df102f) entered disabled state
Feb  6 03:39:41 nextcloud kernel: [1129958.633326] device veth3df102f entered promiscuous mode
Feb  6 03:39:41 nextcloud systemd-networkd[633]: veth3df102f: Link UP
Feb  6 03:39:41 nextcloud dockerd[941]: time="2024-02-06T03:39:41.041218000Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:39:41 nextcloud dockerd[941]: time="2024-02-06T03:39:41.041438600Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:39:41 nextcloud dockerd[941]: time="2024-02-06T03:39:41.075671500Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:39:41 nextcloud containerd[727]: time="2024-02-06T03:39:41.116871900Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:39:41 nextcloud containerd[727]: time="2024-02-06T03:39:41.117017100Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:39:41 nextcloud containerd[727]: time="2024-02-06T03:39:41.117026000Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:39:41 nextcloud containerd[727]: time="2024-02-06T03:39:41.117193400Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/b055e06ee933374c45d35a07455243c7bdf07aa17053ce406805688dccd711d8 pid=1486917 runtime=io.containerd.runc.v2
Feb  6 03:39:54 nextcloud systemd[1]: Started libcontainer container b055e06ee933374c45d35a07455243c7bdf07aa17053ce406805688dccd711d8.
Feb  6 03:40:15 nextcloud kernel: [1129993.547525] eth0: renamed from vetha1ffd09
Feb  6 03:40:15 nextcloud kernel: [1129993.590126] IPv6: ADDRCONF(NETDEV_CHANGE): veth3df102f: link becomes ready
Feb  6 03:40:15 nextcloud kernel: [1129993.590166] br-b164f87cd003: port 4(veth3df102f) entered blocking state
Feb  6 03:40:15 nextcloud kernel: [1129993.590168] br-b164f87cd003: port 4(veth3df102f) entered forwarding state
Feb  6 03:40:15 nextcloud systemd-networkd[633]: veth3df102f: Gained carrier
Feb  6 03:40:15 nextcloud NetworkManager[686]: <info>  [1707190815.9810] device (veth3df102f): carrier: link connected
Feb  6 03:40:17 nextcloud systemd-networkd[633]: veth3df102f: Gained IPv6LL
Feb  6 03:40:39 nextcloud dockerd[941]: time="2024-02-06T03:40:39.494968600Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:40:39 nextcloud NetworkManager[686]: <info>  [1707190839.7705] manager: (vethdd4f5cd): new Veth device (/org/freedesktop/NetworkManager/Devices/576)
Feb  6 03:40:39 nextcloud kernel: [1130017.389491] br-b164f87cd003: port 5(veth0fbb60f) entered blocking state
Feb  6 03:40:39 nextcloud kernel: [1130017.389495] br-b164f87cd003: port 5(veth0fbb60f) entered disabled state
Feb  6 03:40:39 nextcloud kernel: [1130017.390895] device veth0fbb60f entered promiscuous mode
Feb  6 03:40:39 nextcloud NetworkManager[686]: <info>  [1707190839.7714] manager: (veth0fbb60f): new Veth device (/org/freedesktop/NetworkManager/Devices/577)
Feb  6 03:40:39 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 390 seen, reloading interface list
Feb  6 03:40:39 nextcloud systemd-udevd[1487395]: Using default interface naming scheme 'v249'.
Feb  6 03:40:39 nextcloud systemd-udevd[1487396]: Using default interface naming scheme 'v249'.
Feb  6 03:40:39 nextcloud systemd-networkd[633]: veth0fbb60f: Link UP
Feb  6 03:40:39 nextcloud dockerd[941]: time="2024-02-06T03:40:39.806449800Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:40:39 nextcloud dockerd[941]: time="2024-02-06T03:40:39.806751800Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:40:39 nextcloud dockerd[941]: time="2024-02-06T03:40:39.860320800Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:40:39 nextcloud containerd[727]: time="2024-02-06T03:40:39.910707400Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:40:39 nextcloud containerd[727]: time="2024-02-06T03:40:39.910895800Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:40:39 nextcloud containerd[727]: time="2024-02-06T03:40:39.910922600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:40:39 nextcloud containerd[727]: time="2024-02-06T03:40:39.911239800Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/117d6fd4287a99bf6db9bbf9c1f91647163179ed5a0a62d1791a89701442e612 pid=1487419 runtime=io.containerd.runc.v2
Feb  6 03:40:55 nextcloud systemd[1]: Started libcontainer container 117d6fd4287a99bf6db9bbf9c1f91647163179ed5a0a62d1791a89701442e612.
Feb  6 03:41:16 nextcloud kernel: [1130053.847288] eth0: renamed from vethdd4f5cd
Feb  6 03:41:16 nextcloud kernel: [1130053.930934] IPv6: ADDRCONF(NETDEV_CHANGE): veth0fbb60f: link becomes ready
Feb  6 03:41:16 nextcloud kernel: [1130053.930972] br-b164f87cd003: port 5(veth0fbb60f) entered blocking state
Feb  6 03:41:16 nextcloud kernel: [1130053.930974] br-b164f87cd003: port 5(veth0fbb60f) entered forwarding state
Feb  6 03:41:16 nextcloud systemd-networkd[633]: veth0fbb60f: Gained carrier
Feb  6 03:41:16 nextcloud NetworkManager[686]: <info>  [1707190876.3217] device (veth0fbb60f): carrier: link connected
Feb  6 03:41:17 nextcloud systemd-networkd[633]: veth0fbb60f: Gained IPv6LL
Feb  6 03:41:38 nextcloud dockerd[941]: time="2024-02-06T03:41:38.755518300Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:41:38 nextcloud NetworkManager[686]: <info>  [1707190898.9263] manager: (veth4496910): new Veth device (/org/freedesktop/NetworkManager/Devices/578)
Feb  6 03:41:38 nextcloud systemd-udevd[1487918]: Using default interface naming scheme 'v249'.
Feb  6 03:41:38 nextcloud kernel: [1130076.545804] br-b164f87cd003: port 6(veth5c5126b) entered blocking state
Feb  6 03:41:38 nextcloud kernel: [1130076.545808] br-b164f87cd003: port 6(veth5c5126b) entered disabled state
Feb  6 03:41:38 nextcloud kernel: [1130076.545873] device veth5c5126b entered promiscuous mode
Feb  6 03:41:38 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 392 seen, reloading interface list
Feb  6 03:41:38 nextcloud NetworkManager[686]: <info>  [1707190898.9273] manager: (veth5c5126b): new Veth device (/org/freedesktop/NetworkManager/Devices/579)
Feb  6 03:41:38 nextcloud systemd-udevd[1487917]: Using default interface naming scheme 'v249'.
Feb  6 03:41:38 nextcloud systemd-networkd[633]: veth5c5126b: Link UP
Feb  6 03:41:38 nextcloud dockerd[941]: time="2024-02-06T03:41:38.972768900Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:41:38 nextcloud dockerd[941]: time="2024-02-06T03:41:38.972795100Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:41:39 nextcloud dockerd[941]: time="2024-02-06T03:41:39.014490000Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:41:39 nextcloud containerd[727]: time="2024-02-06T03:41:39.057364800Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:41:39 nextcloud containerd[727]: time="2024-02-06T03:41:39.057979000Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:41:39 nextcloud containerd[727]: time="2024-02-06T03:41:39.058207600Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:41:39 nextcloud containerd[727]: time="2024-02-06T03:41:39.058702200Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/0baf5e6ad5c7a922c1470e7685ed41a36dce1a366b1fc680e2b1f889423cb605 pid=1487944 runtime=io.containerd.runc.v2
Feb  6 03:41:46 nextcloud systemd[1]: Started libcontainer container 0baf5e6ad5c7a922c1470e7685ed41a36dce1a366b1fc680e2b1f889423cb605.
Feb  6 03:42:05 nextcloud kernel: [1130102.923365] eth0: renamed from veth4496910
Feb  6 03:42:05 nextcloud kernel: [1130103.011568] IPv6: ADDRCONF(NETDEV_CHANGE): veth5c5126b: link becomes ready
Feb  6 03:42:05 nextcloud kernel: [1130103.011610] br-b164f87cd003: port 6(veth5c5126b) entered blocking state
Feb  6 03:42:05 nextcloud kernel: [1130103.011612] br-b164f87cd003: port 6(veth5c5126b) entered forwarding state
Feb  6 03:42:05 nextcloud systemd-networkd[633]: veth5c5126b: Gained carrier
Feb  6 03:42:05 nextcloud NetworkManager[686]: <info>  [1707190925.4013] device (veth5c5126b): carrier: link connected
Feb  6 03:42:07 nextcloud systemd-networkd[633]: veth5c5126b: Gained IPv6LL
Feb  6 03:42:36 nextcloud dockerd[941]: time="2024-02-06T03:42:36.239514300Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:42:36 nextcloud NetworkManager[686]: <info>  [1707190956.4188] manager: (veth18afe2b): new Veth device (/org/freedesktop/NetworkManager/Devices/580)
Feb  6 03:42:36 nextcloud NetworkManager[686]: <info>  [1707190956.4197] manager: (veth4b6526f): new Veth device (/org/freedesktop/NetworkManager/Devices/581)
Feb  6 03:42:36 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 394 seen, reloading interface list
Feb  6 03:42:36 nextcloud kernel: [1130134.033792] br-b164f87cd003: port 7(veth4b6526f) entered blocking state
Feb  6 03:42:36 nextcloud kernel: [1130134.033796] br-b164f87cd003: port 7(veth4b6526f) entered disabled state
Feb  6 03:42:36 nextcloud systemd-networkd[633]: veth4b6526f: Link UP
Feb  6 03:42:36 nextcloud kernel: [1130134.050189] device veth4b6526f entered promiscuous mode
Feb  6 03:42:36 nextcloud systemd-udevd[1488501]: Using default interface naming scheme 'v249'.
Feb  6 03:42:36 nextcloud systemd-udevd[1488503]: Using default interface naming scheme 'v249'.
Feb  6 03:42:36 nextcloud dockerd[941]: time="2024-02-06T03:42:36.466685000Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:42:36 nextcloud dockerd[941]: time="2024-02-06T03:42:36.466711600Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:42:36 nextcloud dockerd[941]: time="2024-02-06T03:42:36.508618100Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:42:36 nextcloud containerd[727]: time="2024-02-06T03:42:36.602247000Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:42:36 nextcloud containerd[727]: time="2024-02-06T03:42:36.604895400Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:42:36 nextcloud containerd[727]: time="2024-02-06T03:42:36.605199200Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:42:36 nextcloud containerd[727]: time="2024-02-06T03:42:36.605522500Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/b1481c96157f1bc55f888cb2a130f0cdb9781f520e28d8f8e94060b34a81e062 pid=1488525 runtime=io.containerd.runc.v2
Feb  6 03:42:44 nextcloud systemd[1]: Started libcontainer container b1481c96157f1bc55f888cb2a130f0cdb9781f520e28d8f8e94060b34a81e062.
Feb  6 03:43:15 nextcloud kernel: [1130173.214096] eth0: renamed from veth18afe2b
Feb  6 03:43:15 nextcloud kernel: [1130173.393874] IPv6: ADDRCONF(NETDEV_CHANGE): veth4b6526f: link becomes ready
Feb  6 03:43:15 nextcloud kernel: [1130173.393909] br-b164f87cd003: port 7(veth4b6526f) entered blocking state
Feb  6 03:43:15 nextcloud kernel: [1130173.393912] br-b164f87cd003: port 7(veth4b6526f) entered forwarding state
Feb  6 03:43:15 nextcloud systemd-networkd[633]: veth4b6526f: Gained carrier
Feb  6 03:43:15 nextcloud NetworkManager[686]: <info>  [1707190995.7848] device (veth4b6526f): carrier: link connected
Feb  6 03:43:17 nextcloud systemd-networkd[633]: veth4b6526f: Gained IPv6LL
Feb  6 03:43:45 nextcloud dockerd[941]: time="2024-02-06T03:43:45.698984200Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:43:45 nextcloud kernel: [1130203.479003] br-b164f87cd003: port 8(vethd83f6d5) entered blocking state
Feb  6 03:43:45 nextcloud kernel: [1130203.479007] br-b164f87cd003: port 8(vethd83f6d5) entered disabled state
Feb  6 03:43:45 nextcloud kernel: [1130203.479148] device vethd83f6d5 entered promiscuous mode
Feb  6 03:43:45 nextcloud NetworkManager[686]: <info>  [1707191025.8662] manager: (veth05e2267): new Veth device (/org/freedesktop/NetworkManager/Devices/582)
Feb  6 03:43:45 nextcloud NetworkManager[686]: <info>  [1707191025.8672] manager: (vethd83f6d5): new Veth device (/org/freedesktop/NetworkManager/Devices/583)
Feb  6 03:43:45 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 396 seen, reloading interface list
Feb  6 03:43:45 nextcloud systemd-networkd[633]: vethd83f6d5: Link UP
Feb  6 03:43:45 nextcloud systemd-udevd[1489269]: Using default interface naming scheme 'v249'.
Feb  6 03:43:45 nextcloud systemd-udevd[1489270]: Using default interface naming scheme 'v249'.
Feb  6 03:43:45 nextcloud dockerd[941]: time="2024-02-06T03:43:45.888394700Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:43:45 nextcloud dockerd[941]: time="2024-02-06T03:43:45.888420500Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:43:45 nextcloud dockerd[941]: time="2024-02-06T03:43:45.942393500Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:43:45 nextcloud containerd[727]: time="2024-02-06T03:43:45.989218100Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:43:45 nextcloud containerd[727]: time="2024-02-06T03:43:45.989381800Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:43:45 nextcloud containerd[727]: time="2024-02-06T03:43:45.989406400Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:43:45 nextcloud containerd[727]: time="2024-02-06T03:43:45.989562500Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/8d48cb56ba93b94de469a169406b999ac30342a9108f77bce587fe810a541f16 pid=1489300 runtime=io.containerd.runc.v2
Feb  6 03:44:04 nextcloud systemd[1]: Started libcontainer container 8d48cb56ba93b94de469a169406b999ac30342a9108f77bce587fe810a541f16.
Feb  6 03:44:35 nextcloud kernel: [1130253.295902] eth0: renamed from veth05e2267
Feb  6 03:44:35 nextcloud kernel: [1130253.334420] IPv6: ADDRCONF(NETDEV_CHANGE): vethd83f6d5: link becomes ready
Feb  6 03:44:35 nextcloud kernel: [1130253.334465] br-b164f87cd003: port 8(vethd83f6d5) entered blocking state
Feb  6 03:44:35 nextcloud kernel: [1130253.334467] br-b164f87cd003: port 8(vethd83f6d5) entered forwarding state
Feb  6 03:44:35 nextcloud systemd-networkd[633]: vethd83f6d5: Gained carrier
Feb  6 03:44:35 nextcloud NetworkManager[686]: <info>  [1707191075.7248] device (vethd83f6d5): carrier: link connected
Feb  6 03:44:37 nextcloud systemd-networkd[633]: vethd83f6d5: Gained IPv6LL
Feb  6 03:44:59 nextcloud dockerd[941]: time="2024-02-06T03:44:59.442814700Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:44:59 nextcloud NetworkManager[686]: <info>  [1707191099.5733] manager: (veth6595696): new Veth device (/org/freedesktop/NetworkManager/Devices/584)
Feb  6 03:44:59 nextcloud kernel: [1130277.189278] br-b164f87cd003: port 9(vethadf73a3) entered blocking state
Feb  6 03:44:59 nextcloud kernel: [1130277.189281] br-b164f87cd003: port 9(vethadf73a3) entered disabled state
Feb  6 03:44:59 nextcloud kernel: [1130277.193583] device vethadf73a3 entered promiscuous mode
Feb  6 03:44:59 nextcloud systemd-udevd[1490147]: Using default interface naming scheme 'v249'.
Feb  6 03:44:59 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 398 seen, reloading interface list
Feb  6 03:44:59 nextcloud NetworkManager[686]: <info>  [1707191099.5742] manager: (vethadf73a3): new Veth device (/org/freedesktop/NetworkManager/Devices/585)
Feb  6 03:44:59 nextcloud systemd-udevd[1490146]: Using default interface naming scheme 'v249'.
Feb  6 03:44:59 nextcloud systemd-networkd[633]: vethadf73a3: Link UP
Feb  6 03:44:59 nextcloud dockerd[941]: time="2024-02-06T03:44:59.639484800Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:44:59 nextcloud dockerd[941]: time="2024-02-06T03:44:59.639725200Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:44:59 nextcloud dockerd[941]: time="2024-02-06T03:44:59.716153100Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:44:59 nextcloud containerd[727]: time="2024-02-06T03:44:59.770965500Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:44:59 nextcloud containerd[727]: time="2024-02-06T03:44:59.771187600Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:44:59 nextcloud containerd[727]: time="2024-02-06T03:44:59.771214400Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:44:59 nextcloud containerd[727]: time="2024-02-06T03:44:59.771385200Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/9487bb66b8aad64efeca0f1624a8e0fcb8c34fe585b1626be5b196474be3a992 pid=1490173 runtime=io.containerd.runc.v2
Feb  6 03:45:13 nextcloud systemd[1]: Started libcontainer container 9487bb66b8aad64efeca0f1624a8e0fcb8c34fe585b1626be5b196474be3a992.
Feb  6 03:45:35 nextcloud kernel: [1130312.814694] eth0: renamed from veth6595696
Feb  6 03:45:35 nextcloud kernel: [1130312.843456] IPv6: ADDRCONF(NETDEV_CHANGE): vethadf73a3: link becomes ready
Feb  6 03:45:35 nextcloud kernel: [1130312.843505] br-b164f87cd003: port 9(vethadf73a3) entered blocking state
Feb  6 03:45:35 nextcloud kernel: [1130312.843507] br-b164f87cd003: port 9(vethadf73a3) entered forwarding state
Feb  6 03:45:35 nextcloud systemd-networkd[633]: vethadf73a3: Gained carrier
Feb  6 03:45:35 nextcloud NetworkManager[686]: <info>  [1707191135.2344] device (vethadf73a3): carrier: link connected
Feb  6 03:45:36 nextcloud systemd-networkd[633]: vethadf73a3: Gained IPv6LL
Feb  6 03:45:58 nextcloud dockerd[941]: time="2024-02-06T03:45:58.957333500Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:45:59 nextcloud kernel: [1130336.737259] br-b164f87cd003: port 10(veth2c4e392) entered blocking state
Feb  6 03:45:59 nextcloud kernel: [1130336.737263] br-b164f87cd003: port 10(veth2c4e392) entered disabled state
Feb  6 03:45:59 nextcloud kernel: [1130336.745886] device veth2c4e392 entered promiscuous mode
Feb  6 03:45:59 nextcloud NetworkManager[686]: <info>  [1707191159.1358] manager: (veth74b6a6b): new Veth device (/org/freedesktop/NetworkManager/Devices/586)
Feb  6 03:45:59 nextcloud NetworkManager[686]: <info>  [1707191159.1372] manager: (veth2c4e392): new Veth device (/org/freedesktop/NetworkManager/Devices/587)
Feb  6 03:45:59 nextcloud networkd-dispatcher[692]: WARNING:Unknown index 400 seen, reloading interface list
Feb  6 03:45:59 nextcloud systemd-udevd[1490810]: Using default interface naming scheme 'v249'.
Feb  6 03:45:59 nextcloud systemd-udevd[1490812]: Using default interface naming scheme 'v249'.
Feb  6 03:45:59 nextcloud systemd-networkd[633]: veth2c4e392: Link UP
Feb  6 03:45:59 nextcloud dockerd[941]: time="2024-02-06T03:45:59.191763000Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb  6 03:45:59 nextcloud dockerd[941]: time="2024-02-06T03:45:59.191789200Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb  6 03:46:07 nextcloud dockerd[941]: time="2024-02-06T03:46:07.231142300Z" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb  6 03:46:07 nextcloud containerd[727]: time="2024-02-06T03:46:07.282744100Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Feb  6 03:46:07 nextcloud containerd[727]: time="2024-02-06T03:46:07.282945100Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Feb  6 03:46:07 nextcloud containerd[727]: time="2024-02-06T03:46:07.282955900Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Feb  6 03:46:07 nextcloud containerd[727]: time="2024-02-06T03:46:07.283317300Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/5dddee8d52534eca16d0325ec00a2bf15b6d8d92db66efd8884e78ddb43b9310 pid=1490909 runtime=io.containerd.runc.v2
Feb  6 03:46:14 nextcloud systemd[1]: Started libcontainer container 5dddee8d52534eca16d0325ec00a2bf15b6d8d92db66efd8884e78ddb43b9310.
Feb  6 04:09:01 nextcloud CRON[1504918]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 04:17:01 nextcloud CRON[1509679]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb  6 04:39:01 nextcloud CRON[1523137]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 05:02:57 nextcloud snapd[696]: storehelpers.go:791: cannot refresh: snap has no updates available: "core22", "lxd", "snapd"
Feb  6 05:09:01 nextcloud CRON[1541390]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 05:17:01 nextcloud CRON[1546171]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb  6 05:39:01 nextcloud CRON[1559563]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb  6 05:45:43 nextcloud dockerd[941]: time="2024-02-06T05:45:43.865070600Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t A" error="read udp 127.0.0.1:45954->127.0.0.53:53: i/o timeout"
Feb  6 05:45:43 nextcloud dockerd[941]: time="2024-02-06T05:45:43.865146500Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t AAAA" error="read udp 127.0.0.1:47673->127.0.0.53:53: i/o timeout"
...
Feb  6 09:17:01 nextcloud CRON[1691721]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
...
Feb  6 09:39:01 nextcloud CRON[1705162]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
...
Feb  6 10:09:01 nextcloud CRON[1723388]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
...
Feb  6 10:17:01 nextcloud CRON[1728318]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
...
Feb  6 10:38:36 nextcloud snapd[696]: autorefresh.go:638: Cannot prepare auto-refresh change due to a permanent network error: persistent network error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io: Temporary failure in name resolution
Feb  6 10:38:36 nextcloud snapd[696]: stateengine.go:149: state ensure error: persistent network error: Post "https://api.snapcraft.io/v2/snaps/refresh": dial tcp: lookup api.snapcraft.io: Temporary failure in name resolution
...
Feb  6 10:39:01 nextcloud CRON[1741754]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)...
Feb  6 10:51:41 nextcloud dockerd[941]: time="2024-02-06T10:51:41.547441800Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t AAAA" error="read udp 127.0.0.1:58496->127.0.0.53:53: i/o timeout"
Feb  6 10:51:41 nextcloud dockerd[941]: time="2024-02-06T10:51:41.547608000Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t A" error="read udp 127.0.0.1:48974->127.0.0.53:53: i/o timeout"
Feb  6 10:51:43 nextcloud dockerd[941]: time="2024-02-06T10:51:43.045647000Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t AAAA" error="read udp 127.0.0.1:35709->127.0.0.53:53: i/o timeout"
Feb  6 10:51:43 nextcloud dockerd[941]: time="2024-02-06T10:51:43.045805500Z" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;HOST.REDACTED.com.\tIN\t A" error="read udp 127.0.0.1:34790->127.0.0.53:53: i/o timeout"

.
.

Several questions come to mind, can anybody help please:

Why is Nextcloud creating a PAM session using “ncadmin” credentials?
In Nextcloud is there management of hung processes for PAM sessions?
Why would host network drives access (as nextcloud data folder) cause all these processes to hang?
Why after 13 days (like clockwork) do I start to see these type of issues?
Any ideas on how to automatically kill such hung processes which intensely hog the CPU after say 3 hours?
Is this a bug or several that I need to report to Nextcloud AIO github?

sunnyd24 · February 7, 2024, 2:21pm

A similar issue ( AIO Linode becomes unresponsive after a few days) posted recently by another user here.

An issue report opened by @szaimen on my behalf here. Thanks.

szaimen · February 8, 2024, 9:54pm

All processes that you mention here are processes that run on your host and not inside a container. That is why I am pretty sure that this is a bug in docker and/or its underlying software since AIO does not run any processes outside of containers.

My current assumption is that this might happen if the containers get recreated every day due to the daily backup. If it then recreates all the containers 13 days in a row without restarting the server, docker and/or its underlying software seems to run into problems… So one workaround would be scheduling restarts of the host regularly like every week or disabling daily backups.

This is also the reason why I suggested in another post to update docker and orher packages to hopefully fix these problems. (However I did nor investigate if there was a bug report somewhere already upstream).

sunnyd24 · February 22, 2024, 8:08pm

Thanks for the reply @szaimen, it is really appreciated.
I really hope I can find a solution for this as I really want to support the Nextcloud initiative as its great!

I take your comment on-board as to Docker or the host OS Ubuntu most likely causing the issue. The daily backup issue where containers continuously get recreated every day maybe the root cause or something related to this.

I acknowledge the workaround for scheduling regular host restarts, but that approach is a short term fix, but needs further investigation for a long term solution as many other users will probably face this in the future.

I have since updated Ubuntu and Docker packages 14 days ago, docker updates listed below:

jammy/stable amd64 containerd.io amd64 1.6.28-1
jammy/stable amd64 docker-ce-cli amd64 5:25.0.3-1~ubuntu.22.04~jammy
jammy/stable amd64 docker-ce amd64 5:25.0.3-1~ubuntu.22.04~jammy
jammy/stable amd64 docker-ce-rootless-extras amd64 5:25.0.3-1~ubuntu.22.04~jammy
jammy/stable amd64 docker-compose-plugin amd64 2.24.5-1~ubuntu.22.04~jammy

The issue has occurred again 12-14 days later (found today but may have missed the “build up”).

I have checked that docker containers and they are not indeed using CPU resources when this issue occurs, but this time caused by 3 host processes:

User              Command                                CPU%
root              /sbin/init                                100-101%
ncadmin              |_ /lib/systemd/systemd --user         96-101%
                        |_(sd-pam)                          0%

root                 |_ (.plocate)                          98-117%
                        |_ /usr/sbin/updatedb.plocate       0%

In turn, I killed one process at a time:

sigkill /lib/systemd/systemd --user removed one CPU core usage.
sigkill (.plocate) removed one CPU core usage.
/sbin/init cannot be killed as PID 1, but continues to use one CPU core at 100%.

I will raise a Docker or Ubuntu issue for this once I have more information.

A few questions:

Could it be related to following AIO backup option as this happens at least once every 10-13 days, unlikely I am sure?

Automatically update all containers, the mastercontainer and on saturdays your Nextcloud apps

When you mention a bug report upstream, do you mean in Docker or Ubuntu or somewhere else?
If so, any guidance on where to start investigating would be great?
Is there a way to change the update frequency in the AIO interface, to say, once a week, I haven’t seen it but maybe a feature request for custom update schedules in GitHub?
Meanwhile, can you confirm if which “user” creates all Nextcloud docker processes?