Difficulties when uploading a large amount of small files

ℹ️ Support
1

Hi

When I upload a folder that contains a large amount of small files (say several dozens) and sub-folders I get an “abort” message.

My php.ini is defined with these parameters:

php_value upload_max_filesize = 16G
php_value post_max_size = 16G
php_value max_input_time 3600
php_value max_execution_time 3600
output_buffering=off

I have no warnings in my “Admin” page and I also configured memory caching using APC.

Logging shows the following:

Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`,`lock`,`ttl`) SELECT ?,?,? FROM `oc_file_locks` WHERE `key` = ? HAVING COUNT(*) = 0' with params ["files\/f0250313cd61be54c7b669e3b9ff8d2e", 1, 1517222292, "files\/f0250313cd61be54c7b669e3b9ff8d2e"]: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 128: Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 1015: Doctrine\DBAL\DBALException driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'INSERT INTO `oc...', Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Connection.php - line 213: Doctrine\DBAL\Connection->executeUpdate('INSERT INTO `oc...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Adapter.php - line 114: OC\DB\Connection->executeUpdate('INSERT INTO `oc...', Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Connection.php - line 251: OC\DB\Adapter->insertIfNotExist('*PREFIX*file_lo...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Lock/DBLockingProvider.php - line 118: OC\DB\Connection->insertIfNotExist('*PREFIX*file_lo...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Lock/DBLockingProvider.php - line 163: OC\Lock\DBLockingProvider->initLockField('files/f0250313c...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Common.php - line 704: OC\Lock\DBLockingProvider->acquireLock('files/f0250313c...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Common->acquireLock('files/FinTech P...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/FinTech P...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/FinTech P...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1931: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/FinTech P...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 2041: OC\Files\View->lockPath('/FinTech Partne...', 1, false)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1332: OC\Files\View->lockFile('/FinTech Partne...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1379: OC\Files\View->getCacheEntry(Object(OCA\FilesAccessControl\StorageWrapper), 'files/FinTech P...', '/FinTech Partne...')
/srv/dev-disk-by-label-HDD/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 214: OC\Files\View->getFileInfo('/admin/files/Fi...')
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 1178: OCA\DAV\Connector\Sabre\Directory->getChild('Screenshots')
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 590: Sabre\DAV\Server->createCollection('FinTech Partner...', Object(Sabre\DAV\MkCol))
[internal function] Sabre\DAV\CorePlugin->httpMkcol(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105: call_user_func_array(Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 479: Sabre\Event\EventEmitter->emit('method MKCOL', Array)
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 254: Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/srv/dev-disk-by-label-HDD/nextcloud/apps/dav/appinfo/v1/webdav.php - line 76: Sabre\DAV\Server->exec()
/srv/dev-disk-by-label-HDD/nextcloud/remote.php - line 162: require_once('/srv/dev-disk-b...')
{main}

System details:
Nextcloud V. 12.0.5
Debian Jessie
mysql Ver 15.1 Distrib 10.2.12-MariaDB
intel i3 1.8GHz

Any ideas?

Thanks

2

Ok, I’ve managed to 90% solve this by changing the following line in php.ini:

max_file_uploads = 20000

(before the change it was 20 or something)

However, I still get an error message when uploading a single bulk of 1000-2000 files and subfolders. Much less errors than before, but if it includes lots of sub-folders and files I still get an error.

Help?

Logging shows:

Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`,`lock`,`ttl`) SELECT ?,?,? FROM `oc_file_locks` WHERE `key` = ? HAVING COUNT(*) = 0' with params ["files\/3a46ef690a513ac9a663aff2f086a3e1", 1, 1517236528, "files\/3a46ef690a513ac9a663aff2f086a3e1"]: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 128: Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 1015: Doctrine\DBAL\DBALException driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'INSERT INTO `oc...', Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Connection.php - line 213: Doctrine\DBAL\Connection->executeUpdate('INSERT INTO `oc...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Adapter.php - line 114: OC\DB\Connection->executeUpdate('INSERT INTO `oc...', Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/DB/Connection.php - line 251: OC\DB\Adapter->insertIfNotExist('*PREFIX*file_lo...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Lock/DBLockingProvider.php - line 118: OC\DB\Connection->insertIfNotExist('*PREFIX*file_lo...', Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Lock/DBLockingProvider.php - line 163: OC\Lock\DBLockingProvider->initLockField('files/3a46ef690...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Common.php - line 704: OC\Lock\DBLockingProvider->acquireLock('files/3a46ef690...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Common->acquireLock('files/Documents...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/Documents...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 588: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/Documents...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1931: OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/Documents...', 1, Object(OC\Lock\DBLockingProvider))
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 2041: OC\Files\View->lockPath('/Documents/FinT...', 1, false)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1332: OC\Files\View->lockFile('/Documents/FinT...', 1)
/srv/dev-disk-by-label-HDD/nextcloud/lib/private/Files/View.php - line 1379: OC\Files\View->getCacheEntry(Object(OCA\FilesAccessControl\StorageWrapper), 'files/Documents...', '/Documents/FinT...')
/srv/dev-disk-by-label-HDD/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 214: OC\Files\View->getFileInfo('/admin/files/Do...')
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 1178: OCA\DAV\Connector\Sabre\Directory->getChild('Screenshots')
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 590: Sabre\DAV\Server->createCollection('Documents/FinTe...', Object(Sabre\DAV\MkCol))
[internal function] Sabre\DAV\CorePlugin->httpMkcol(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105: call_user_func_array(Array, Array)
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 479: Sabre\Event\EventEmitter->emit('method MKCOL', Array)
/srv/dev-disk-by-label-HDD/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 254: Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/srv/dev-disk-by-label-HDD/nextcloud/apps/dav/appinfo/v1/webdav.php - line 76: Sabre\DAV\Server->exec()
/srv/dev-disk-by-label-HDD/nextcloud/remote.php - line 162: require_once('/srv/dev-disk-b...')
{main}
3

Help, please? Anyone?
I’m getting errors upon bulk uploads (dozens of files), and I really want to solve this…
Except that everything works perfect.

Thanks