Unable to revert to prior revisions after upgrade / Missing data

Operating system: Linux 5.15.0-1036-oracle #42-Ubuntu SMP Thu May 18 17:58:30 UTC 2023 aarch64

Webserver: Apache/2.4.56 (Debian) (apache2handler)

Database: sqlite3 3.34.1

PHP version: 8.2.6

Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, bcmath, exif, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, sysvsem, zip, Zend OPcache

Nextcloud version: 26.0.2 - 26.0.2.1

The issue you are facing:

Hi Nextcloud Community,

I just upgraded to version 26.0.2 last night using the latest image, and have run into a troubling issue. Upon opening a text document via the mobile app (which was already logged into a session), while pasting text the content in the entire document simply disappeared. I figured this might have been inadvertent error on my part, so I planned to revert the changes.

I noticed that the file’s activity history is accurate. However, the version history only shows the Current Version, and the thumbnail image is broken. If I attempt to revert changes, I’m unable to do so - there is no link available when I hover over the button.

The text app version is 3.7.2.

I’ve never had version history or content disappear. This is very unsettling given the sensitivity of the data I store in Nextcloud. I have old backups of my Nextcloud volume, but I’m not sure how practical it would be to restore that. The files are stored in Object Storage using the External S3 File Storage extension. Nextcloud uses a sqlite database stored in a volume, which is mounted.

I’m aware a recent fix for a CVE-2023-32318 made a change to the text editing feature and the way sessions are handled, but I had upgraded from 26.0.1 which contained the fix and it was fine.

It should be noted that if I save a different existing text document after this issue occurred, the version history populates properly. Therefore, this might be isolated to the first text document that I opened after the upgrade, which may have had its revision history corrupted.

If there’s an easy way to restore this data, or if this issue has been mentioned before, I’m curious to hear input from the community. I don’t want this to happen again. I want to determine the root cause to ensure that my data is safe.

Here are screenshots:

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

I’m likely not able to replicate this issue, but:

  1. Open a text document after an upgrade
  2. Inadvertently clear the data or have it disappear via the mobile app
  3. Try and fail to revert the document

The output of your Nextcloud log in Admin > Logging:

I’ve noticed no errors in the logs other than a deprecation warning after the upgrade. I don’t think this is related since these errors occurred the night prior:

|Error|PHP|Using ${var} in strings is deprecated, use {$var} instead at /var/www/html/3rdparty/scssphp/scssphp/src/Compiler.php#3491||2023-06-07T23:37:18-0400|
|---|---|---|---|—|

|Error|PHP|Using ${var} in strings is deprecated, use {$var} instead at /var/www/html/3rdparty/scssphp/scssphp/src/Compiler.php#3491||2023-06-07T23:37:18-0400|

|Info|updater|\OC\Updater::maintenanceDisabled: Turned off maintenance mode||2023-06-07T23:36:56-0400|

|Info|updater|\OC\Updater::updateEnd: Update successful||2023-06-07T23:36:56-|

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

{
    "htaccess.RewriteBase": "\/",
    "memcache.local": "\\OC\\Memcache\\Memcached",
    "apps_paths": [
        {
            "path": "\/var\/www\/html\/apps",
            "url": "\/apps",
            "writable": false
        },
        {
            "path": "\/var\/www\/html\/custom_apps",
            "url": "\/custom_apps",
            "writable": true
        }
    ],
    "overwritehost": ā€œredacted.example.com",
    "overwriteprotocol": "https",
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "127.0.0.1:9000"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "sqlite3",
    "version": "26.0.2.1",
    "overwrite.cli.url": "https:\/\/127.0.0.1:9000",
    "installed": true,
    "memcache.distributed": "\\OC\\Memcache\\Memcached",
    "memcached_servers": [
        [
            "fc00:0:0:1::10",
            11211
        ]
    ],
    "memcached_options": {
        "14": 50,
        "15": 50,
        "19": 50,
        "20": 50,
        "8": 50,
        "-1001": true,
        "16": true,
        "18": true
    },
    "app_install_overwrite": [
        "files_ebookreader",
        "epubreader"
    ],
    "maintenance": false,
    "mail_smtpmode": "smtp",
    "mail_smtpsecure": "tls",
    "mail_sendmailmode": "smtp",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "587",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauthtype": "PLAIN",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "loglevel": 2
}

The output of your Apache/nginx/system log in /var/log/____:

No errors within the time period that this occurred.

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

No errors within the time period that this occurred.

Snappy? Docker?

When say ā€œmobile appā€, are you referring to the official Android client, the official iOS official, or via a mobile browser?

Unfortunately this is going to be very challenging to troubleshoot and fix (if it’s a bug) if it can’t be reproduced. :-/

The logs you provided aren’t related. Can you find the logs for the transactions from the client to the NC server for your Quotes.txt immediately before and after the disappearing / overwrite event?

Since you’re using External Storage - and it’s configured as Primary Storage - it’s possible to bypass NC and access your Object Storage directly. Depending on your Object Storage environments configuration, you may have access to a a prior version of the file there. e.g.

Using versioning in S3 buckets - Amazon Simple Storage Service (off by default)

AWS S3 Compatible API - Dramatically Lower Your Costs (on by default)

MinIO | Bucket and Object Versioning (depends)

P.S. I know this isn’t helpful now, but it’s a good reminder to head off problems like this in the future: When using Object Storage - or any external storage - whether Primary or not, it’s important to integrate it into your data backup strategies. This is handled outside of NC. And since this data is stored outside of the datadirectory it isn’t automatically picked up in your server data backups.

I’m using the latest docker image.

I appreciate the feedback.

Regarding the mobile application, I’m using the official IOS client.

The logs were provided because that’s what the template requested. I will search the transaction logs again. I didn’t see any errors in my logs within the time period that this occurred, however.

Since the files are in object storage, I do keep backups. I was able to restore the file, but without the versioning information. I have noticed that since the upgrade that the activity log is associated with all of my text documents, but none appear to have the versioning information associated with them.

If the revisions are not kept between upgrades, this would explain why I couldn’t roll back the changes. It’s possible that I might have inadvertently deleted the content when I opened the file with the mobile application and had nothing to roll back to. Since the file was at that point empty, that would explain the broken thumbnail.

Should the versions of a document edited over time be kept between upgrades? If so, where is that content stored? My volume mount point is to /var/www/html. If the revision information is stored elsewhere, that might explain why that information wasn’t kept after the upgrade.

Thanks,
Dave

The actual files (prior versions) are stored in [datadirectory]/[user]/files_versions (even for External Storage - or at least they are on my NC26 test instance).

Revisions are definitely retained by NC through upgrades (or they’re supposed to be).

Which S3 provider/platform are you using out of curiosity? The only time I’ve seen the broken preview image (and version history) on External Storage has been during recent testing with Google (GCS) to try to understand their various S3 compatibility quirks.

I don’t have a lot of time today to dig further into this, but I did find a PR that looked interesting: