Directory trees grind sync to a crawl

ℹ️ Support
, , , ,
1

Nextcloud version: 29.0.6
Operating system and version: Fedora Linux 40
Apache or nginx version: 2.4.61
PHP version: 8.2.23

The issue you are facing:

Using the desktop client for linux and syncing a folder of various documents gives wildly differing speeds depending on what’s transferring. Larger files will upload at the hardware limit of 60-80mb/s, but once it starts syncing smaller items (of which there are 50k+) it’ll only transfer 5 or so at a time in the worst case. Eventually, though, the client gets to the point where it only updates transfer progress in increments of exactly 100 files every so often.

Another thing that’s been a major headache is various files will cause the client to throw errors, the most common being “failed to open stream”/“could not locate file.” In a few cases certain files cause strange behavior (looping, progress regression) until said files are removed.

config.php:

<?php 
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'upgrade.disable-web' => true,
  'instanceid' => 'oc75gsf1b4c3',
  'passwordsalt' => '[redacted]',
  'secret' => '[redacted]',
  'trusted_domains' => 
  array (
    0 => '192.168.12.197:8080',
    1 => '192.168.193.111:8080',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '29.0.6.1',
  'overwrite.cli.url' => 'http://192.168.12.197:8080',
  'dbname' => 'nextcloud',
  'dbhost' => 'nextcloud-db.dns.podman',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '[redacted]',
  'installed' => true,                                                  
);

The only nextcloud log output is just a ton of entries like this:

{
  "reqId": "IGCar3qDK3IKCpBVpqEG",
  "level": 3,
  "time": "2024-09-07T08:08:40+00:00",
  "remoteAddr": "192.168.12.132",
  "user": "[redacted]",
  "app": "PHP",
  "method": "POST",
  "url": "/remote.php/dav/bulk",
  "message": "file_put_contents(/var/www/html/data/[redacted]/unknown-26.png): Failed to open stream: No such file or directory at /var/www/html/lib/private/Files/Storage/Local.php#339",
  "userAgent": "Mozilla/5.0 (Linux) mirall/3.13.33.13 (build 25182) (Nextcloud, manjaro-6.6.46-1-MANJARO ClientArchitecture: x86_64 OsArchitecture: x86_64)",
  "version": "29.0.6.1",
  "data": {
    "app": "PHP"
  },
  "id": "66dc0a08a5b8e"
}

An entry that appears frequently in the host logs:

nextcloud
[Sat Sep 07 07:46:37.178462 2024] [access_compat:error] [pid 268:tid 268] [client 10.89.2.1:59892] AH01797: client denied by server configuration: /var/www/html/data/.ocdata

CODE_FILE
    src/ctr_logging.c
CODE_FUNC
    write_journald
CODE_LINE
    329
CONTAINER_ID
    5aa3ed8b8842
CONTAINER_ID_FULL
    5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef
CONTAINER_NAME
    nextcloud
PRIORITY
    3
SYSLOG_IDENTIFIER
    nextcloud
_BOOT_ID
    c0c07c89f5d44131990b71c817f925c1
_CAP_EFFECTIVE
    1ffffffffff
_CMDLINE
    /usr/bin/conmon --api-version 1 -c 5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef -u 5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef/userdata -p /run/containers/storage/overlay-containers/5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef/userdata/pidfile -n nextcloud --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef/userdata/oci-log --conmon-pidfile /run/containers/storage/overlay-containers/5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.imagestore=/usr/lib/containers/storage --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef
_COMM
    conmon
_EXE
    /usr/bin/conmon
_GID
    0
_HOSTNAME
    kentucky-homelab
_MACHINE_ID
    c7a8af996f914300bb75601feec80044
_PID
    3916
_RUNTIME_SCOPE
    system
_SELINUX_CONTEXT
    system_u:system_r:container_runtime_t:s0
_SOURCE_REALTIME_TIMESTAMP
    1725695197178565
_SYSTEMD_CGROUP
    /machine.slice/libpod-conmon-5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef.scope
_SYSTEMD_INVOCATION_ID
    27a5a222862746148c9a0203de60197f
_SYSTEMD_SLICE
    machine.slice
_SYSTEMD_UNIT
    libpod-conmon-5aa3ed8b88429fc8beb9625f732731a7796ac1bc0033203ca9d4f6ef4a96cfef.scope
_TRANSPORT
    journal
_UID
    0
__CURSOR
    s=27ce399206814fa1bc9287ed1df4ca44;i=f0890;b=c0c07c89f5d44131990b71c817f925c1;m=939efb4887;t=62182b937eafd;x=92d2073f75067f79
__MONOTONIC_TIMESTAMP
    634027460743
__REALTIME_TIMESTAMP
    1725695197178621
__SEQNUM
    985232
__SEQNUM_ID
    27ce399206814fa1bc9287ed1df4ca44

Thanks for looking!

2

This shouldn’t happen. Something weird is going on here. What’s your underlying volumes configuration for this app container? And what’s the storage underlying that?

3

It’s being run off an mdraid 10 array at /data/container-data/nextcloud/

The host is currently on an old usb drive at the moment. It does seem like it gets a bit more activity during transfers for some reason, but swap and everything is turned off.

SELinux also isn’t giving any errors.

Docker compose:

version: '3.8'

services:
  nextcloud_db:
    container_name: nextcloud-db
    image: mariadb:10.6
    restart: unless-stopped
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    networks:
      - cloudnet
    volumes:
      - ./db:/var/lib/mysql:Z
    environment:
      - MYSQL_ROOT_PASSWORD=${NC_MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${NC_MYSQL_PASSWORD}
      - MYSQL_DATABASE=${NC_MYSQL_DATABASE}
      - MYSQL_USER=${NC_MYSQL_USER}

  nextcloud:
    container_name: nextcloud
    image: nextcloud
    restart: unless-stopped
    networks:
      - cloudnet
    ports:
      - ${NC_PORT}:80
    volumes:
      - ./nextcloud:/var/www/html:Z
    environment:
      - PHP_MEMORY_LIMIT=${NC_PHP_MEMORY_LIMIT} # 20G
      - PHP_UPLOAD_LIMIT=${NC_PHP_UPLOAD_LIMIT} # 30G
      - NEXTCLOUD_DATA_DIR=/var/www/html/data
      - MYSQL_PASSWORD=${NC_MYSQL_PASSWORD}
      - MYSQL_DATABASE=${NC_MYSQL_DATABASE}
      - MYSQL_USER=${NC_MYSQL_USER}
      - MYSQL_HOST=nextcloud-db.dns.podman

networks:
  cloudnet:
    name: cloudnet
    driver: bridge

Thanks

4

I also get this error on the latest 29.0.6:

[Sun Sep 08 11:57:41.850487 2024] [access_compat:error] [pid 70:tid 70] [client 172.22.0.5:38984] AH01797: client denied by server configuration: /var/www/html/data/.ocdata

On the overview page I get:

Error occurred while checking server setup
5

I finally just gave up on the desktop client when I couldn’t figure out why it was giving errors with large files and went to rclone for documents and kopia for backups instead, which has worked so much better, albeit with a bit of fine-tuning. Something like this with rclone:

rclone sync \
        --dry-run
        --log-file rclone.log \
        --log-level DEBUG \
        --progress \
        --transfers 5 \
        --checkers 10 \
        --contimeout 60s \
        --timeout 300s \
        --retries 6 \
        --low-level-retries 10 \
        --stats 1s \
        --stats-file-name-length 0 \
        --fast-list \
         [source path] Nextcloud:[remote path]

Relatively heavy on the resources of your destination machine, so make sure it doesn’t bog things down too much before raising the --transfers and --checkers flags. (I prefer a 1:2 ratio). Remove the --dry-run flag after making sure it runs as expected.

Obviously this isn’t a solution, just an alternative. I don’t know why the official desktop client is so problematic. I’ll probably end up writing a script for running automatic syncs with rclone if something like that doesn’t exist already.