Syncing a large Folder (1TB+) with windows client

[/details]

Nextcloud version (eg, 12.0.2): 18.0.1
Operating system and version (eg, Ubuntu 17.04):Ubuntu 18.04.4
Apache or nginx version (eg, Apache 2.4.25): Server version: Apache/2.4.29 (Ubuntu) Server built: 2019-09-16T12:58:48

PHP version (eg, 7.1): PHP 7.2.24-0ubuntu0.18.04.3

The issue you are facing:

I am running my nextcloud vm from hanssonit.se on an openmediavault machine. The data storage is located on a shared folder that is mounted via nfs. And a Windows client is connected via 1GB Lan to the server.
So far the Cloud is up and running. The issu that i have might just be a limit of the cloud that i have reached. I have a 1.3TB Data folder with more than 200k Files in several subfolders. Now i try to sync this folder to my cloud with the nextcloud Windows client. The sync process takes a very long time to compare the local and the remote Data and the copy process takes even longer. so far after 5 days (24h per day) the sync process managed to upload the wirst 900GB of data. Once and a while i get sync erros and the connection has to be reastablished.

Is there a recommended maximum folder size or number of files?

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Sync large datafolder with nextcloud via Nextcloud client

The output of your Nextcloud log in Admin > Logging:

[webdav] Fatal: Doctrine\DBAL\Exception\DeadlockException: An exception occurred while executing 'UPDATE "oc_filecache" SET "mtime" = GREATEST("mtime", ?), "etag" = ? WHERE ("storage" = ?) AND ("path_hash" IN ('d41d8cd98f00b204e9800998ecf8427e', '45b963397aa40d4a0063e0d85e4fe7a1', '349e9c2afa1127f9d55a7492a3265ec5', '5bec4b2da49bebbdd8ddc03263f717f0', 'd02c97767f81cc66e6a3cb84802645f5', 'a20875e06ea4543749f356007478c024', 'bf6870c40251d93d6634809edd46be25'))' with params [1583117001, "5e5c72c9a0eb4", 4]:

SQLSTATE[40P01]: Deadlock detected: 7 ERROR:  deadlock detected
DETAIL:  Process 12555 waits for ShareLock on transaction 564868; blocked by process 12554.
Process 12554 waits for ShareLock on transaction 564867; blocked by process 12555.
HINT:  See server log for query details.
CONTEXT:  while rechecking updated tuple (1942,181) in relation "oc_filecache" at <<closure>>

 0. /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php line 169
    Doctrine\DBAL\Driver\AbstractPostgreSQLDriver->convertException("An exception oc ... "", Doctrine\DBAL\Dr ... ]})
 1. /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php line 145
    Doctrine\DBAL\DBALException::wrapException(Doctrine\DBAL\Driver\PDOPgSql\Driver {}, Doctrine\DBAL\Dr ... ]}, "An exception oc ... "")
 2. /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php line 1063
    Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Doctrine\DBAL\Driver\PDOPgSql\Driver {}, Doctrine\DBAL\Dr ... ]}, "UPDATE \"oc_fil ... )", {1: 1583117001,2: "5e5c72c9a0eb4",3: 4})
 3. /var/www/nextcloud/lib/private/DB/Connection.php line 220
    Doctrine\DBAL\Connection->executeUpdate("UPDATE \"oc_fil ... )", [1583117001,"5e5c72c9a0eb4",4], [1,2,1])
 4. /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php line 203
    OC\DB\Connection->executeUpdate("UPDATE \"oc_fil ... )", {dcValue1: 15831 ... 4}, {dcValue1: 1,dcValue2: 2,dcValue3: 1})
 5. /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php line 215
    Doctrine\DBAL\Query\QueryBuilder->execute()
 6. /var/www/nextcloud/lib/private/Files/Cache/Propagator.php line 101
    OC\DB\QueryBuilder\QueryBuilder->execute()
 7. /var/www/nextcloud/lib/private/Files/Cache/HomePropagator.php line 49
    OC\Files\Cache\Propagator->propagateChange("*** sensitive parameter replaced ***", 1583117001, 539347)
 8. /var/www/nextcloud/lib/private/Files/Cache/Updater.php line 138
    OC\Files\Cache\HomePropagator->propagateChange("*** sensitive parameter replaced ***", 1583117001, 539347)
 9. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php line 291
    OC\Files\Cache\Updater->update("*** sensitive parameters replaced ***")
10. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
    OCA\DAV\Connector\Sabre\File->put(null)
11. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
    OCA\DAV\Connector\Sabre\Directory->createFile("DSC_9469.JPG", null)
12. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
    Sabre\DAV\Server->createFile("files/Ben/Bilde ... G", null, null)
13. <<closure>>
    Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
14. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
15. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
16. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
17. /var/www/nextcloud/apps/dav/lib/Server.php line 319
    Sabre\DAV\Server->exec()
18. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
19. /var/www/nextcloud/remote.php line 165
    require_once("/var/www/nextcl ... p")

PUT /remote.php/dav/files/Ben/Bilder/strukturiert/BW_Bilder/fussballbilderklein/kaatz/DSC_9469.JPG
from 192.168.10.31 by Ben at 2020-03-02T03:43:22+01:00

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

PASTE HERE

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

i would not sync all data in one run rather add data step by step to the nextcloud sync folder

1 Like

You should use the v2.3.3 of the nextcloud windows client or the v2.6.4
If you still have deadlocks you should ask help because your VM needs some tweaking.

Thanks for the quick Reply,
i just updated the client to 2.6.4 stable and i plan to give the vm a host only network adapter to use it for the nfs share. So the data won’t have to pass the real network adapter twice. And maybe increase the processor cores from 3 to 4.

Give updates on this issue after.

Update:
So i seperatet the external Interface and a “host only” adapter for the internal data handling. That improved the performance significant, the same data sync progress would estimate 3-4 days instead of 6 days. But i still got an i/o error after 6 hours transfering data to the cloud.
After that i put the entire vm on the system ssd of the host (data still on hdd Raid via nfs), that also improved the overall performance so the hole sync might take about 2-3 days. And hopefully i will not get any i/o errors anymore. If i will still get errors i will update here.
Maybe someone with comparable data and usecase can tell me about the hardware setup and experiences.
My server hardware specs: i7-2600, 20gb ddr3, 512gb ssd, 5x8tb hdd as md raid6

I had done 4To of data (files around 2Mo and 128Mo) in 2 Days by local gigabyte lan.
Client OSX v2.3.3
Server Qnap with ubuntu 18.04
Nginx - MariaDB 10.3 - Nextcloud 16
Intel 4core at 2,6Ghz 20TB Raid6
Only a few files had failled.

Update:
my last change was to change from NFS share to SMB/CIFS share for the data folder. With this the performance increased again to about 15-16 hours for the same transfer. Sysbench showed that specialy asyncronus rw is much better with smb/cifs.

1 Like