User files failing to decrpyt after manual upgrade to v28 (default encryption module)

Nextcloud version: 28.0.1
Operating system and version: Linux 4.18.0-425.3.1.lve.3.el7h
PHP version: 8.1.25

The issue you are facing:
After manually upgrading to NC 28, user files are not decrypting.

I am unable to disable the encryption module. It is locked on, which seems to be expected/intended behavior for NC.

I attempted to upgrade from NC v27 to NC v28, using the web updater. Of course, that broke as usual. It doesn’t like my DB server for some reason, so it throws errors and fails. Then I tried using OCC through the terminal. That broke too, just like the web updater. So I did a manual upgrade…I uploaded the new NC files to the server, copied in my user files and config file, and ran the upgrade command in OCC. It worked! Then I just reinstalled all of the apps. However, user files (documents, etc.) are now coming through in their encrypted from. And everywhere the NC Sync application is running, the files were overwritten with the encrypted versions. Each user file now contains this, followed by many hyphens and gibberish.

HBEGIN:oc_encryption_module:OC_DEFAULT_MODULE:cipher:AES-256-CTR:signed:true:useLegacyFileKey:false:encoding:binary:HEND------------------------

I have backups, but still, I’d like to fix it directly. Did the key not import properly when I performed the upgrade? Or is there another solution to this?

Is this the first time you’ve seen this error? (Y/N): Yes. Previous upgrades in this same manner have been successful.

Steps to replicate it:

  1. Perform an unsuccessful automated upgrade from v27 (latest version) to v38 (latest version) through the GUI upgrade tool.
  2. When that fails, try upgrading through OCC.
  3. When that fails, try a manual upgrade, as described in NC documentation (successfully)

The output of your Nextcloud log in Admin > Logging:

??
Is this supposed to be a menu option? If so, see the screenshot and remarks below.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '--------',
  'passwordsalt' => '--------',
  'secret' => '--------',
  'trusted_domains' => 
  array (
    0 => '--------',
  ),
  'datadirectory' => '--------',
  'dbtype' => 'mysql',
  'version' => '28.0.1.1',
  'overwrite.cli.url' => '--------',
  'dbname' => '--------',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => '--------',
  'mysql.utf8mb4' => true,
  'dbuser' => '--------',
  'dbpassword' => '--------',
  'installed' => true,
  'app_install_overwrite' => 
  array (
    0 => 'cpanelmailsync',
    1 => 'occweb',
    2 => 'listman',
    3 => 'hibp',
    4 => 'holiday_calendars',
    5 => 'dropit',
    6 => 'weather',
    7 => 'secrets',
    8 => 'metadata',
    9 => 'carnet',
    10 => 'files_downloadactivity',
  ),
  'has_rebuilt_cache' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 0,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => '--------',
  'mail_domain' => '--------',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '--------',
  'mail_smtphost' => '--------',
  'mail_smtpname' => '--------',
  'mail_smtppassword' => '--------',
);

  • Are you using a master key or per user keys?
  • Can you provide the output of occ app:list as well?

Since it may be related somehow - or maybe it isn’t but it should not be happening: what types of errors and fails? If you get a chance, check (or post) your updater.log.

Yes. Logs are available in the Web UI under Administration settings->Logging. They can also be accessed directly in the file nextcloud.log located in your datadirectory. It may have clues, particularly if you can look both at the point of the upgrade as well as while triggering any transaction that spits out raw encrypted files (apparently).

Hi, thank you for your reply!

I am using master keys. I never figured out how to do per-user keys safely. There is a NC plugin which seems to offer that, but has received abysmal reviews. Therefore, I never tried it.

Here is the current app list, as requested. I should do a little cleanup.

Enabled:
  - activity: 2.20.0
  - admin_audit: 1.18.0
  - announcementcenter: 6.7.0
  - bruteforcesettings: 2.8.0
  - calendar: 4.6.1
  - carnet: 0.25.2
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contacts: 5.5.0
  - dav: 1.29.1
  - dropit: 0.4.0
  - duplicatefinder: 1.1.4
  - encryption: 2.16.0
  - external: 5.3.1
  - federatedfilesharing: 1.18.0
  - files: 2.0.0
  - files_accesscontrol: 1.18.0
  - files_downloadactivity: 1.16.0
  - files_external: 1.20.0
  - files_lock: 28.0.1
  - files_pdfviewer: 2.9.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - fileslibreofficeedit: 1.1.0
  - forms: 4.0.0
  - geoblocker: 0.5.13
  - groupfolders: 16.0.1
  - holiday_calendars: 0.3.0
  - impersonate: 1.15.0
  - listman: 27.1.4
  - login_notes: 1.4.0
  - lookup_server_connector: 1.16.0
  - memegen: 1.0.6
  - metadata: 0.19.0
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.1
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - onlyoffice: 9.0.0
  - password_policy: 1.18.0
  - phonetrack: 0.7.7
  - photos: 2.4.0
  - polls: 6.0.1
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - quota_warning: 1.18.0
  - registration: 2.3.0
  - secrets: 1.5.2
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - side_menu: 3.11.2
  - socialsharing_email: 3.0.1
  - suspicious_login: 6.0.0
  - tasks: 0.15.0
  - text: 3.9.1
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - twofactor_email: 2.7.4
  - twofactor_nextcloud_notification: 3.8.0
  - twofactor_totp: 10.0.0-beta.2
  - twofactor_webauthn: 1.3.2
  - updatenotification: 1.18.0
  - user_status: 1.8.1
  - user_usage_report: 1.12.0
  - viewer: 2.2.0
  - weather: 1.7.7
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - circles: 28.0.0-dev (installed 24.0.0)
  - contactsinteraction: 1.9.0 (installed 1.9.0)
  - dashboard: 7.8.0 (installed 7.1.0)
  - federation: 1.18.0 (installed 1.11.0)
  - files_reminders: 1.1.0 (installed 1.1.0)
  - firstrunwizard: 2.17.0 (installed 2.10.0)
  - logreader: 2.13.0 (installed 2.13.0)
  - recommendations: 2.0.0 (installed 1.0.0)
  - related_resources: 1.3.0 (installed 1.3.0)
  - socialsharing_diaspora: 3.0.1 (installed 3.0.1)
  - support: 1.11.0 (installed 1.11.0)
  - survey_client: 1.16.0 (installed 1.16.0)
  - systemtags: 1.18.0 (installed 1.11.0)
  - user_ldap: 1.19.0

The upgrade error I receive seems to be indicating a DB timeout, even though the DB server is still online and responsive to other requests. Still, this is cheap hosting, so it could be the server’s fault. I never had this problem with a different hosting provider I previously used. I recieve the following error in both the WebUI, and through OCC: "Doctrine\DBAL\Exception\ConnectionLost: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away"

Where would I look for the “updater.log” file? I’ve never understood where to find the error logs, which causes extra difficulty when troubleshooting.

Alright, so Admin settings…at the risk of sounding helpless…where next? here is the nav bar. Am I just not seeing it, or is it missing for some reason?

However, I did find the general log in the data folder. It contains some moderately sensitive stuff, so perhaps I can DM those to you?

Hello,

after reading your topic I have exactly the same problem
after updating nextcloud result I can no longer access these files have you found a solution in the meantime?

Unfortunately, I did not find a solution, @TruckStone . I restored most files from an external backup, but a few low priority ones I have just lost access to.

This indicates some very sloppy crypto handling, so now I find myself wondering if I need to deploy a new instance of NextCloud with encryption disabled, and migrate everything over there. Or perhaps I need to give OC (the name which shall not be named) another look…

For your issue, TruckStone, I’ve noticed that NC keeps data related to older software versions. Might it be possible to rebuild an older version of NC (or somehow revert the upgrade), access the files properly, export all of the (decrypted) files from that instance, then import them into NC 28 for re-encryption?

Edit: I just found this: GitHub - nextcloud/encryption-recovery-tools: This project contains tools to recover files that have been encrypted with the Nextcloud End-to-End Encryption or Nextcloud Server-Side Encryption.
I wonder if that might be useful.

At least one significant part of the problem is the result of the above list in your config. This is the list of apps that have previously force enabled by the admin (at some point previously - may not have been recently). Force enabling overrides Server’s built-in protections against loading app versions that are incompatible with the running version of Nextcloud Server. The logs you DM’d also confirm this (they are many errors resulting from these apps being enabled).

You need to review the manually entries for each of these apps in the appstore and confirm they’re actually ready for NC28. I only checked two (files_downloadactivity and metadata) and they’re definitely not compatible with NC28. Disable any that are not compatible with NC28.

It’s still possible you could have something in addition to the above going on, but that’s a biggie that has to be sorted out before troubleshooting anything else since that state will create all sorts of unpredictable problems.

I’d also clear that entire entry from your config.php.

Yes, Carnet and Holiday Calendars both have compatibility issues with v28. “Secrets” I got working fine, and I think I’ve disabled the rest in my recent cleanup efforts.

But could a random addon cause NC to just forget that it needs to decrypt files??

But could a random addon cause NC to just forget that it needs to decrypt files??

Definitely. The encryption app itself is an add-on (not that that is necessary for this to happen - just mentioning that to make it clear that apps are quite powerful).

A misbehaving app can break just about anything. Some are particularly likely to cause certain types of problems, perhaps. For example, the files_downloadactivity app registers itself to be notified (and, in turn, then executed) every time a file is read. And it was crashing every single time…

I already disabled that particular addon, and was still experiencing the issue. But fair enough, that could conceivably cause any number of issues.

But here’s what I would think is an important point. All files preexisting the upgrade (even if duplicated) are presented in the encrypted form. Any files uploaded since the v28 upgrade are presented in decrypted form, even when they are overwritten on top of the old pre-28 files.

So the issue would seem to have been a one-time event at the time of upgrade, at which point NC “forgot” that all user files were encrypted. Addons were disabled during the upgrade, so I don’t see them messing things up during that process, unless I’m missing something.

Does NC decrypt user files during the upgrade process? Perhaps it tried unsuccessfully to decrypt the files, performed the upgrade, then encrypted them a second time (perhaps even under a new server key, if that rotates)? That’s just a total guess…

honestly i’m disappointed with necxtcloud i had almost no extra modules installed
in short
@nextcloud thank you for answering this question I don’t think I’m an isolated case

I just built a powershell script that will analyze each file regardless for my computer of format and will just read to see if there is HBEGIN:oc_encryption_module at the beginning and at the end will tell the number of files and where they are located.

so that I can build him a tomb … and see the extent of the damage

I will try to restore an old version of nextcloud from my vm except that the data in helle same sound, no backup so not sure it will work

hello
quick question which web server
despite deleting and re-uploading all data I still have problems …
do you think that the type of web server may have an impact, even if it’s not logical?

at the moment I have nginx 25 to take advantage of http3 h3 and UDP
before I was on apache2 a few months ago

if other people have the same problem, don’t hesitate to react.

Hello,
I’m using nginx as well, and was when I encountered this issue.

Since the upgrade, my files have been handled properly, but I’m due for another minor upgrade. I suppose that will crypto-shred all of my files again…