How to speed up initial sync of large folder

Support intro

Sorry to hear you’re facing problems. :slightly_frowning_face:

The community help forum (help.nextcloud.com) is for home and non-enterprise users. Support is provided by other community members on a best effort / “as available” basis. All of those responding are volunteering their time to help you.

If you’re using Nextcloud in a business/critical setting, paid and SLA-based support services can be accessed via portal.nextcloud.com where Nextcloud engineers can help ensure your business keeps running smoothly.

Getting help

In order to help you as efficiently (and quickly!) as possible, please fill in as much of the below requested information as you can.

Before clicking submit: Please check if your query is already addressed via the following resources:

(Utilizing these existing resources is typically faster. It also helps reduce the load on our generous volunteers while elevating the signal to noise ratio of the forums otherwise arising from the same queries being posted repeatedly).

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 32.0.3
  • Operating system and version (e.g., Ubuntu 24.04):
    • Ubuntu 22.04.5
  • Web server and version (e.g, Apache 2.4.25):
    *
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • Nginx 1.18.0
  • PHP version (e.g, 8.3):
    • 8.4.16
  • Is this the first time you’ve seen this error? (Yes / No):
    • yes
  • When did this problem seem to first start?
    • a few days ago, after installing AOI for the first time
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • AIO
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • no

Summary of the issue you are facing:

I have installed AIO on my server and I am now trying to evaluate it as an alternative for Seafile with extra features.

I have a large folder with mostly images and short videos: about 367GB in 78613 files and folders. I tried to sync it with a desktop client on a Linux Mint Cinnamon PC. It was extremely slow and barely seemed to make progress. Also, the desktop client’s settings GUI reacted very slowly (> 30s to open a menu). I let the desktop client run for several days, and still only a small fraction of the files are synced.

So I decided to try another approach. I copied all files into /var/lib/docker/volumes/nextcloud_aio_nextcloud_data/_data/Johan/files/Fotos directly on the server. I changed ownership of all files recursively to www-data:www-data. Then I ran:

docker exec -it nextcloud-aio-nextcloud /usr/local/bin/php /var/www/html/occ files:scan --all

Most of the files are still not shown in the nextcloud web interface. The files that are shown may be those that were already synced by the desktop client.

  1. Is it expected that the desktop client is so slow ?

  2. What can I do to make nextcloud recognize the directly copied files?

Proposed additional step before running occ files:scan

You already mentioned that ownership is fixed to www-data, which is correct.
However, even with correct ownership, inconsistent POSIX permissions on directories or files can still cause Nextcloud to silently skip files during scanning, especially with large datasets.

These steps are not theoretical.
I have them documented in my own migration notes, collected while migrating a Nextcloud server, and I consistently apply them as “steps of certainty” before every large import or re-scan.

Normalize directory permissions

find . -type d -exec chmod 750 {} \;

This ensures:

  • execute permission on directories so Nextcloud can actually traverse them

  • full read/write/execute access for www-data

  • eliminates permission-related skips during recursive scans

Normalize file permissions

find . -type f -exec chmod 640 {} \;

This ensures:

  • all files are readable and writable by the web server

  • avoids unreadable files being silently ignored during indexing or checksum generation

Then run the scan

sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ files:scan-app-data
sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ files:scan --all

Why this may help in your case

During my own server migration, I observed that:

  • ownership alone was not sufficient

  • mixed or inherited permissions caused some files to be skipped without visible errors

  • scans completed successfully, yet not all files appeared in the UI

Normalizing permissions before scanning has proven to be a reliable safety step to ensure that all files are actually reachable by occ files:scan.


Note: This post was written with the help of an AI assistant as a writing aid only. The opinions, solutions, and technical recommendations are fully based on my personal experience.
More about how and why I use AI to write forum posts:
:right_arrow: Is there limitations to installing Nextcloud via CT template on Proxmox - #4 by vawaver

1 Like

Even after executing the above commands, not all files are shown in the web interface. In fact, nothing changed.

I tried a sanity check: used the web interface to create a new folder Foobar, and then checked the folder /var/lib/docker/volumes/nextcloud_aio_nextcloud_data/_data/Johan/files on the server. I cannot find the new folder on the server. Somehow, there seems to be a disconnect between the folders and files shown in the web interface and those present on the server.

Glad for the update.
Based on this behavior, this no longer looks like a scan or permission issue.

Please double-check the following first:

  1. External storage
    Confirm that this user is not using any external storage (Local / SMB / NFS / object storage).
    If external storage is in use, files created via the web UI will not appear in the default Docker data volume.

  2. Correct user / home storage
    Make sure you are inspecting the filesystem path that actually belongs to this user.

  3. Client details
    Please specify:

  • the Nextcloud desktop client version

  • whether it is an AppImage or installed from distribution repositories, or other source (Flathub)

Before doing any further scans or experiments, make sure to back up all files.

For reference: I run a similar data volume on my private server, and the initial synchronization itself was not a major issue.
Of course, the total sync time also strongly depends on the internet connection speed.

Thanks for your reply.

  1. External storage

I am not sure how to check this. I am not aware of any external storage, but I have done so many experiments before posting here, that I am no longer sure. How can I check to be sure?

This is the command I used to start nextcloud on the server:
```
sudo docker run \
–init \
–sig-proxy=false \
–name nextcloud-aio-mastercontainer \
–restart always \
–publish 127.0.0.1:8081:8080 \
–env APACHE_PORT=11000 \
–env APACHE_IP_BINDING=0.0.0.0 \
–env NEXTCLOUD_MEMORY_LIMIT=1024M \
–volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
–volume /var/run/docker.sock:/var/run/docker.sock:ro \

  1. Correct user / home storage
    Logged in in the web interface as Johan. Copied my fotos to /var/lib/docker/volumes/nextcloud_aio_nextcloud_data/_data/Johan/files/Fotos, and created test folder Foobar in that folder.
  2. Client details
    I am using Nextcloud-4.0.4-x86_64.AppImage

Good to know that a sync of this size should work.

Thanks for reminding me to check my internet speed. That might indeed be an issue: currently around 2 Mbps, while I am paying for 30 Mbps. Will check Monday with my provider. Server has a 10 Gbps network interface. Anyway, I will try again when that is fixed.

```

1 Like

OK I figured what is wrong, not yet how to fix it.

At some point during my experiments, I decided to use an external folder (on the host) instead of the docker volume nextcloud_aio_nextcloud_data to store the data. I added this option to the docker run command:

--env NEXTCLOUD_DATADIR=/srv/nextcloud/ncdata

Later, I changed my mind. So I did this:

sudo docker stop nextcloud-aio-mastercontainer
sudo docker rm nextcloud-aio-mastercontainer
sudo docker run .... # options without --env NEXTCLOUD_DATADIR, as shown above

I assumed that this would get rid of the external NEXTCLOUD_DATADIR, but it didn’t: files are still synced to /srv/nextcloud/ncdata

So the question is: how can I reverse the decision to use an external folder to store the data?

That behavior is expected.

Once NEXTCLOUD_DATADIR is used in Nextcloud All-in-One, the path is persisted in the configuration, not only in the container runtime arguments.
Removing and re-creating the mastercontainer does not revert the data directory.

The clean and supported way to reverse this is to properly reset the entire AIO instance, exactly as described here:
https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-properly-reset-the-instance

This will:

  • remove stored configuration

  • detach the external data directory

  • allow AIO to recreate and use the default Docker volume again

Before doing this, make a full backup of all data.

After the reset, I strongly recommend not using long docker run commands, but switching to a Docker Compose setup (using docker-compose.yml file).
And commands:

# Start docker 
docker compose up -d
# Stop docker
docker compose down

It is easier to read, maintain, and avoids exactly this type of configuration confusion.

You can see a complete working example here (AIO + Proxmox + Nginx Proxy Manager + Watchtower):
https://help.nextcloud.com/t/proxmox-nginx-nextcloud-aio-watchtower-deployment-guide/233643?u=vawaver

1 Like

For completeness: syncing with the desktop client is still not working. I documented the issues in this post: Slow and weird desktop client behavior (Linux AppImage 4.0.4)

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.