External Storage S3 Modified Time Incorrect

ℹ️ Support
, , ,
1

Nextcloud version: 16.0.5 and 17.0.2
Operating system and version: Ubuntu 18.04.3 and NethServer 7.7.1908 (CentOS 7)
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.29 and Apache 2.4.6
PHP version (eg, 7.1): 7.3.11-1 and 7.2.24

The issue you are facing:
I am trying to migrate my data from one Nextcloud instance to a new one. The original instance (16.0.5 on Ubuntu) has local storage for the files. The target instance (17.0.2 on NethServer) I’m using Wasabi S3 as external storage mounted at / for the user.

I tried using rclone to sync between the WebDAV nodes of each NC. When I did this, I noticed that the new instance was always failing to get the modified date. I had to restart the transfer a few times (800GB+ of data) and each time it wanted to sync everything because the modified date was incorrect.

I gave up on this approach and instead started syncing from WebDAV -> S3 directly using rclone. When I do this, the files are able to sync, and the sync can detect that the files are the same date. When I check on the S3 files I can see the correct modified time. When I look in Nextcloud, all of the Modified times are the date the sync was done.

I tried to do a files:scan and it went through and successfully found all the files. Now, all of the file modify dates are set to the time when I did the scan.

I have seen many forum posts talking about issues with modify dates but I haven’t seen any recent solutions for NC 17+.

One post suggested running this query:
UPDATE oc_filecache SET mtime=storage_mtime

I looked in this table though and the storage_mtime is the same as mtime, so it appears that Nextcloud is not bringing over any of the modification date information from S3.

Any help on this would be greatly appreciated!

Is this the first time you’ve seen this error?: Y

The output of your Nextcloud log in Admin > Logging:

In the logs there are two main things. This one I read on the forums is just a small bug and can be safely ignored

fclose(): supplied resource is not a valid stream resource at /usr/share/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php#688

However, I also see some entries like this:

Doctrine\DBAL\Exception\LockWaitTimeoutException: An exception occurred while executing 'UPDATE `oc_filecache` SET `mtime` = ?, `etag` = ?, `storage_mtime` = ?, `checksum`=? WHERE (`mtime` <> ? OR `etag` <> ? OR `storage_mtime` <> ? OR `checksum` <> ? OR `mtime` IS NULL OR `etag` IS NULL OR `storage_mtime` IS NULL OR `checksum` IS NULL) AND `fileid` = ? ' with params [1580515102, "5e34e62ea021e", 1580515102, "", 1580515102, "5e34e62ea021e", 1580515102, "", 110756]: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction

So it seems like it may be trying to update the modify times but is failing for some reason.

2

I tried to update the target Nextcloud, hoping that it would help the issue, but it got stuck half way through. So I re-installed it (18.0), since all the files are on S3 anyway, and it started up just fine. After a file rescan it still has the wrong modification dates, and no errors in the log.

3

July 19, Nextcloud 22, and currently facing the same issue, using the same exact scenario.

If I mount it via davfs, and touch a file using a custom date, the file will get the correct modification date, but Nextcloud will report it changed just now :confused:

1 Like
4

The core problem is that S3 doesn’t store two times (modified and created), and if I remember correctly, there is not much support for doing a PATCH to update that specific attribute. This could lead to trying to write the item to the S3 bucket again (with the forced timestamp), but that would lead to extra S3 usage which incurs a fee.

The other way to approach this would be for Nextcloud to maintain internal metadata for that file, which it does not seem equipped to do.

In the end, I gave up on this approach and just mounted a network drive and installed Nextcloud to there. S3 might be a viable option on a fresh install, but it seems to break in every possible way when trying to migrate an instance TO an S3 bucket to act as the file storage for Nextcloud.

5

Hi. Any news on this issue?