Some files not syncing

ℹ️ Support
1

Hi,

I’ve recently set up a new Nextcloud instance, and I am really impressed with the possibilities. However, I have recently discovered that a small number of files (maybe 3 or 4 out of hundreds or thousands) are not uploading to the server when starting a new sync to an existing folder on my computer (using Mac desktop sync client) or, similarly, when choosing to sync a folder from the server to a new location on a different computer there are always a few files that don’t make it.

The most annoying thing is that I don’t seem to be able to force a resync in the desktop client to clean up the missing files.

In looking a bit more closely, I can see that there are two different problems. One is that, when doing a sync of a large number of files (in either direction), I usually get a small number of files that hit a file lock conflict. I think this is a known issue (for example Fatal WebDav error in log). Errors in the log look like the following:

Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO 'oc_file_locks' ('key','lock','ttl') SELECT ?,?,? FROM 'oc_file_locks' WHERE 'key' = ? HAVING COUNT(*) = 0' with params ["files\/a7b054b2aa6352173a82883cc364ad4e", 1, 1497240775, "files\/a7b054b2aa6352173a82883cc364ad4e"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/a7b054b2aa6352173a82883cc364ad4e' for key 'lock_key_index'

The conflicting locks seem to eventually “go away” after a while (querying oc_file_locks table on my mariadb), so presumably the problem is short lived. Mind you, I am surprised at how long it takes for locks to disappear - 30 minutes maybe? Not sure if this is normal?

While this problem is already ringing alarm bells, the much bigger problem is that I can’t do anything to get my desktop sync client to reattempt syncing those files. The sync client reports that everything is up to date, and the related “internal server errors” disappear in the sync status. I have tried a “php occ files:scan --all”, but this only seems to scan on the server side.

I have also tried quitting, pausing and logging out on the sync client, but it always tells me everything is sync’ed once sync is resumed, even though the files remain missing.

This is a showstopper for me. I am so impressed with what Nextcloud can do, but if I can’t trust it to sync 100% of my files, it is game over.

I would love some help to straighten this out if I can.

Nextcloud version: 12.0.0 (running in docker container, linuxserver/nextcloud image)
Operating system and version: Debian Jesse (Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux)
Apache or nginx version: nginx 1.12.0
PHP version: 7.1.5

Is this the first time you’ve seen this error?:
No, it always happens when I do large syncs up or down.

Can you reliably replicate it? (If so, please outline steps):
Yes, so long as the sync is “large enough” although I’m not quite sure what the bar for “large enough” is.

The issue you are facing:
As above

The output of your Nextcloud log in Admin > Logging:
Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO oc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/a7b054b2aa6352173a82883cc364ad4e", 1, 1497240775, "files\/a7b054b2aa6352173a82883cc364ad4e"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/a7b054b2aa6352173a82883cc364ad4e' for key 'lock_key_index' 2017-06-12T13:12:55+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/15b37ed2e1ef92068aa0d1be5dfca1b2", 1, 1497240671, "files\/15b37ed2e1ef92068aa0d1be5dfca1b2"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/15b37ed2e1ef92068aa0d1be5dfca1b2' for key 'lock_key_index' 2017-06-12T13:11:11+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/393465b8c58e386a34d06659ddcb8ded", 1, 1497240629, "files\/393465b8c58e386a34d06659ddcb8ded"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/393465b8c58e386a34d06659ddcb8ded' for key 'lock_key_index' 2017-06-12T13:10:29+1000 Fatal webdav Sabre\DAV\Exception\ServiceUnavailable: HTTP/1.1 503 System in maintenance mode. 2017-06-12T12:37:48+1000 Fatal webdav Sabre\DAV\Exception\ServiceUnavailable: HTTP/1.1 503 System in maintenance mode. 2017-06-12T12:37:15+1000 Fatal webdav Sabre\DAV\Exception\ServiceUnavailable: HTTP/1.1 503 System in maintenance mode. 2017-06-12T12:37:12+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/27bd9a27e73c62c07a182ebe9d8adfa9", 1, 1497238009, "files\/27bd9a27e73c62c07a182ebe9d8adfa9"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/27bd9a27e73c62c07a182ebe9d8adfa9' for key 'lock_key_index' 2017-06-12T12:26:49+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/f99d1a523c3f4fdb3e337f8f514e0a94", 1, 1497237952, "files\/f99d1a523c3f4fdb3e337f8f514e0a94"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/f99d1a523c3f4fdb3e337f8f514e0a94' for key 'lock_key_index' 2017-06-12T12:25:53+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/15b37ed2e1ef92068aa0d1be5dfca1b2", 1, 1497237906, "files\/15b37ed2e1ef92068aa0d1be5dfca1b2"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/15b37ed2e1ef92068aa0d1be5dfca1b2' for key 'lock_key_index' 2017-06-12T12:25:06+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/5ac830caab5079486b7ce961017c8277", 1, 1497237872, "files\/5ac830caab5079486b7ce961017c8277"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/5ac830caab5079486b7ce961017c8277' for key 'lock_key_index' 2017-06-12T12:24:32+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey= ? HAVING COUNT(*) = 0' with params ["files\/754078e90fe248e96586dfa48cbdd0af", 1, 1497237846, "files\/754078e90fe248e96586dfa48cbdd0af"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/754078e90fe248e96586dfa48cbdd0af' for key 'lock_key_index' 2017-06-12T12:24:06+1000 Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTOoc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locksWHEREkey = ? HAVING COUNT(*) = 0' with params ["files\/3c68613e20df4a0f61fefa1e4239c93c", 1, 1497237846, "files\/3c68613e20df4a0f61fefa1e4239c93c"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/3c68613e20df4a0f61fefa1e4239c93c' for key 'lock_key_index' 2017-06-12T12:24:06+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T12:09:26+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T12:00:09+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:56:53+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:55:41+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:55:04+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:54:46+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:54:36+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:54:36+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:54:33+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:54:30+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:30:45+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:21:34+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:21:23+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:20:11+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:19:35+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:19:16+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:19:06+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:19:01+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b7eaa07a150c82782887b5d7a866da1e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:58+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:34+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b39bf893b4d364468b5e9c28d1e4ecb9") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:30+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b39bf893b4d364468b5e9c28d1e4ecb9") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:20+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b39bf893b4d364468b5e9c28d1e4ecb9") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:16+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:15+1000 Error PHP DateTime::__construct(): Failed to parse time string ("b39bf893b4d364468b5e9c28d1e4ecb9") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:12+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:18:00+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:42+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:42+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:33+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:33+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:28+1000 Error PHP DateTime::__construct(): Failed to parse time string ("d3c553d7ff7c1a098ccc54281a5cdd2e") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:25+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:23+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:14+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:09+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:09+1000 Error PHP DateTime::__construct(): Failed to parse time string ("4d06ce29a5cc317b982eb9488e8129f8") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:17:06+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:15:23+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:15:12+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:14:36+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:14:18+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:14:08+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:14:03+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:13:59+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:13:52+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:13:47+1000 Error PHP DateTime::__construct(): Failed to parse time string ("5152dc4aac7c6fae7fb00a4d62a5ea72") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:13:44+1000 Error PHP DateTime::__construct(): Failed to parse time string ("cb3fb4cc636839de28cf79118e83caef") at position 0 ("): Unexpected character at /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php#133 2017-06-12T11:13:18+1000...

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
<?php $CONFIG = array ( 'memcache.local' => '\\OC\\Memcache\\APCu', 'datadirectory' => '/data', 'instanceid' => 'xxx', 'passwordsalt' => 'xxx', 'secret' => 'xxx', 'trusted_domains' => array ( 0 => 'xxx:444', 1 => 'xxx', ), 'overwrite.cli.url' => 'https://xxx', 'overwrite' => 'xxx', 'overwriteprotocol' => 'https', 'dbtype' => 'mysql', 'version' => '12.0.0.29', 'dbname' => 'nextcloud', 'dbhost' => 'xxx:3306', 'dbport' => '', 'dbtableprefix' => 'oc_', 'dbuser' => 'chbmb', 'dbpassword' => 'xxx', 'installed' => true, 'mail_from_address' => 'xxx', 'mail_smtpmode' => 'smtp', 'mail_smtpauthtype' => 'LOGIN', 'mail_domain' => 'xxx', 'mail_smtpauth' => 1, 'mail_smtphost' => 'xxx', 'maintenance' => false, 'logtimezone' => 'Australia/Hobart', );

Screen Shot 2017-06-12 at 1.13.57 pm.png1686×746 208 KB

2

A little bit more fiddling, a little bit more detail. I mentioned that file locks appear to stay for a very long time in the oc_file_locks table. I looked more carefully… entries in the table do indeed stay for a long time, but the lock column is getting set to zero pretty quickly, which I assume means the lock is released when it should be. That’s good.

The other thing I’ve noticed is that if I do something (e.g. duplicate another file) in a directory on my computer where a file has failed to sync down from the server, then this seems to have the effect of forcing that particularly directory to be resync’d. That is: shortly after duplicating a different file in the relevant directory, the missing file appears (and a notification appears in the sync client to tell me the missing file has arrived).

That’s fine if I know the file is missing, but not so good if I don’t.

Very concerning behaviour, but maybe these additional clues help people more knowledgable than me…

3

does this relate to your Problems?

4

Hi,

Thanks for the suggestion, but no, that isn’t related. The quoted case involves multiple users. In my case, I have only one user, and I can trigger the problem with a single large sync up or down from a single computer (via desktop sync client - macOS).

It seems quite easy to trigger the problem. This afternoon I was able to trigger one constraint violation on file locks in the mariadb just on a four (small) file sync up to the server. Zero workload on my nextcloud server. No other logged in users.

It bothers me that this happens so readily. Are others experiencing this too? And does anyone have any feedback on how to force a client side rescan / resync?

Thanks,

Nick

1 Like
5

I’ve recently set up a new Nextcloud 12.0.3 instance.

Two days ago I uploaded 4 folders via the osx folder sync client, today I went to download them and kept getting a “file not found” error. They show in the web interface, but I get a similar error from there. The sync client shows everything green and synced even with a “force resync” yet when i actually go to the folder on the server, the files are indeed missing altogether, meaning they likely never got uploaded.

I don’t think I can trust a system that loses files with no warning and all green lights. I’d like to fix the issue. Anyone have any ideas?

Looking at the logs it looks like it started with:
Undefined offset: 3 at /Volumes/www/nextCloud/apps/activity/lib/FilesHooks.php#620

Afterwords there are a ton of:
unable to rename, file does not exists
and
Sabre\DAV\Exception\ServiceUnavailable: Could not open file

Failing to write a file is fine, if it knows there was a problem, and retries or alerts me, but carrying on as if everything is okay is a bit frightening and will result in lost data, especially if it is showing files in the web interface that don’t actually exist on the server. Seems there needs to be some more error handling and checks in the file handling code?

6

I was faced to the same problem, some GB of data and maybe 30 files did not sync.
I set Nextcloud into maintenance mode and deleted all entries in the file locking table.
Then i checked the logs, but found nothing helpful.
Uploading a blocked file via Browser Interface showed a different message “…database gone…”.
All other sites on the same machine like Wordpress etc had no problem at all.

SOLUTION:
I set the mySQL parameter wait_timeout to a higher value.
Values like 10s or 30s seem to make problems!

After that i just moved the blocked files out of Nextcloud and back into Nextcloud.
It’s working perfect now.

Maybe this is helpful for some guys, greets to all.

7

I am afraid that this is not solved yet. I am getting this with a bunch of very small files in 14.0.0:

Error downloading http://xxx/nextcloud/remote.php/dav/files/xxx/ar/beta_level-03.cfg - server replied: Internal Server Error (An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`,`lock`,`ttl`) SELECT ?,?,? FROM `oc_file_locks` WHERE `key` = ? HAVING COUNT(*) = 0' with params ["files\/abeb23d80afcc5d192d9c0f7b56fe046", 1, 1537479982, "files\/abeb23d80afcc5d192d9c0f7b56fe046"]:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/abeb23d80afcc5d192d9c0f7b56fe046' for key 'lock_key_index')

In my case, it doesn’t seem to be a timeout issue.

Eventually, the problem seems to go away on following attempts, but it leaves me a bit anxious about what is happening with my data (and that I will miss any actual errors because the log is very cluttered by this).

Any help will be greatly appreciated.

Edit: Found an ownCloud PR where a similar issue is discussed. A some sort of a race condition would explain this behaviour pretty well…

8

Bunches of small files are known to create tons of databse queries. Since this is related to the file-locking feature, you should use redis as filelocking cache reducing the load on your *sql database. More tuning on the database cache can then further improve the upload speed.

9

Thank you for the tip, I will definitely look into it!

But I still think the errors shouldn’t happen, even if the default setup is less efficient in my scenario…

10

Feel free to open a bug report.