[Nextcloudpi] Automatic update to 29.0.10 crashed system

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 28.0.13 → 29.0.10 update
  • Operating system and version (e.g., Ubuntu 24.04):
    • Nextcloudpi on RaspberyPi-4
  • Web server and version (e.g, Apache 2.4.25):
    • Apache 2.4.62-1~deb12u2
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • unknown
  • PHP version (e.g, 8.3):
    • 8.1.31
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • Automatic upgrade 28.0.12 → 28.0.13 seemed to work on Dec-6, 2024
    • Automatic upgrade 28.0.13 → 29.0.10 led to system unusable
    • unclear if the first upgrade already introduced (hidden) issues.
  • Installation method (e.g. AIO, NCP, Bare Metal/Archive, etc.)
    • in Nextcloudpi nc-autoupdate-nc was set; after the crash now deactivated.
  • Are you using Cloudflare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

I’m using nextcloudpi on a rpi4. My system was on 28.0.12 and got automatically updated once the 28.0.13 was available on Dec-6, 8:02 CET. There were some messages regarding disabled incompatible apps such as calendar, contacts… you find the full 28.012->28.0.13 update log at the bottom.

Before the system tried to upgrade several times to 29.0.9 which was withdrawn from the update servers.

This morning, my system did not work, I couldn’t even login via the web-API. Error message: I’m logging in from an non-trusted domain.

Tonight, my system automatically tried to upgrade to 29.0.10 which failed with an error message; the automatically following roll-back did not succeed and left the system in an unusable state --some relevant entries in trusted-domains disappeared. Full log at the bottom.

Looking into config.php: it got really messed up. Besides some missing entries in trusted_domains, there was a

huge block /w strange messages

‘overwrite.cli.url’ => 'An unhandled exception has been thrown:
RedisException: No such file or directory in /var/www/nextcloud/lib/private/RedisFactory.php:123
Stack trace:
#0 /var/www/nextcloud/lib/private/RedisFactory.php(123): Redis->pconnect()
#1 /var/www/nextcloud/lib/private/RedisFactory.php(167): OC\RedisFactory->create()

After the system not working this morning, if found this announcement:

Obviously, my system did automatically upgrade to 28.0.13 before it was withdrawn.

I restored config.php from a backup, did a reboot and the system seems to work right now. However, the RedisException above is again back in config.php…

Questions:

  • any idea what was the root cause could be for the in reality non-successful Rollback? Here strange message are

    • Nextcloud is not installed - only a limited number of commands are availableCommand "upgrade" is not defined. Abort
    • likewise the error messages during the roll back
  • How to stop automatic upgrades until the current issues /w 28.0.13 and 30.0.3 are resolved?

  • Would you recommend to roll-back to 28.0.12?
    If yes: how to perform that?

Thanks for your support in advance,
Thomas

Log entries

ncp-autoupdate-nc.txt from Dec-6, 8:02 CET, 28.0.12->28.0.13
/etc/cron.daily/ncp-autoupdate-nc:
Running as unit: ncp-update-nc.service
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0 100  817k  100  817k    0     0   499k      0  0:00:01  0:00:01 --:--:--  499k
Current   Nextcloud version 28.0.12
Requested Nextcloud version latest
Selected  Nextcloud version 28.0.13
INFO: You have requested an update to the latest available version that can be performed directly, which is '28.0.13'. Updates can only performed to the next major version (or the latest minor version of the current major version). If you run ncp-update-nc again after this update completes, a new version might be available.
Download Nextcloud 28.0.13...
Back up current instance...
check free space...
Maintenance mode enabled
backup database...
backup files...
backup /var/www/nextcloud-bkp_20241206_1733468218.tar.gz generated
Maintenance mode disabled
Storing backup at '/var/www/nextcloud-bkp_20241206_1733468218-28.0.12.tar.gz'...
Install Nextcloud 28.0.13...
Fix permissions...
Upgrade...
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Disabled incompatible app: calendar
Disabled incompatible app: contacts
Disabled incompatible app: notes
Disabled incompatible app: notify_push
Disabled incompatible app: richdocuments
Disabled incompatible app: richdocumentscode_arm64
Disabled incompatible app: tasks
Disabled incompatible app: user_migration
Starting code integrity check...
Finished code integrity check
Update successful
Turned off maintenance mode
Resetting log level
Done.
Done.
All tables already up to date!
 - Repair MySQL collation
     - All tables already have the correct collation -> nothing to do
 - Clean tags and favorites
     - 0 tags of deleted users have been removed.
     - 0 tags for delete files have been removed.
     - 0 tag entries for deleted tags have been removed.
     - 0 tags with no entries have been removed.
 - Repair invalid shares
 - Move .step file of updater to backup location
 - Add move avatar background job
     - Repair step already executed
 - Add preview cleanup background jobs
 - Migrate oauth2_clients table to nextcloud schema
     - Update the oauth2_access_tokens table schema.
     - Update the oauth2_clients table schema.
     - Delete clients (and their related access tokens) with the redirect_uri starting with oc:// or ending with *
 - Fix potential broken mount points
     - No mounts updated
 - Repair language codes
 - Add log rotate job
 - Clear frontend caches
     - Image cache cleared
     - JS cache cleared
 - Clear every generated avatar
 - Add preview background cleanup job
 - Queue a one-time job to cleanup old backups of the updater
 - Cleanup invalid photocache files for carddav
 - Add background job to cleanup login flow v2 tokens
 - Remove potentially over exposing share links
     - No need to remove link shares.
 - Clear access cache of projects
 - Reset generated avatar flag
 - Keep legacy encryption enabled
 - Check encryption key format
 - Remove old dashboard app config data
 - Add job to cleanup the bruteforce entries
 - Queue a one-time job to check for user uploaded certificates
 - Repair DAV shares
 - Add background job to set the lookup server share state for users
 - Add token cleanup job
 - Clean up abandoned apps
 - Add possibly missing system config
 - Add AI tasks cleanup job
 - Queue a job to generate metadata
 - Cache logo dimension to fix size in emails on Outlook
     - Logo dimensions are already known
 - Add cleanup-deleted-users background job
 - Remove shares of old group memberships
 - Repair mime types
 - Validate the phone number and store it in a known format for search
     - WARNING: Can not validate phone numbers without `default_phone_region` being set in the config file
 - Handle outdated scheduling events
     - Cleaning up old scheduling events
 - Upgrading Circles App
 - Fix component of birthday calendars
     - 4 birthday calendars updated.
 - Regenerating birthday calendars to use new icons and fix old birthday events without year
     - Repair step already executed
 - Fix broken values of calendar objects
    0 [>---------------------------]
 - Registering building of calendar search index as background job
     - Repair step already executed
 - Register building of social profile search index as background job
     - Repair step already executed
 - Registering background jobs to update cache for webcal calendars
     - Added 0 background jobs to update webcal calendars
 - Registering building of calendar reminder index as background job
     - Repair step already executed
 - Clean up orphan event and contact data
     - 0 events without a calendar have been cleaned up
     - 0 properties without an events have been cleaned up
     - 0 changes without a calendar have been cleaned up
     - 0 cached events without a calendar subscription have been cleaned up
     - 0 changes without a calendar subscription have been cleaned up
     - 0 contacts without an addressbook have been cleaned up
     - 0 properties without a contact have been cleaned up
     - 0 changes without an addressbook have been cleaned up
 - Remove activity entries of private events
     - Removed 0 activity entries
 - Clean up old calendar subscriptions from deleted users that were not cleaned-up
 0/1 [>---------------------------]   0%
 1/1 [============================] 100%
     - 0 calendar subscriptions without an user have been cleaned up
 - Remove invalid object properties
     - 0 invalid object properties removed.
 - Fix the share type of guest shares when migrating from ownCloud
 - Copy the share password into the dedicated column
 - Set existing shares as accepted
 - Clean up meta table
 - Show a hint about the new editor to existing users
 - Set binary permissions
 - Update OAuth token expiration times
 - Switches from default updater server to the customer one if a valid subscription is available
     - Repair step already executed
 - Send an admin notification if monthly report is disabled
 - Force-reset all Text document sessions
 - Initialize migration of background images from dashboard to theming app
 - Add background job to check for backup codes
 - Populating added database structures for workflows
Update completed successfully.
Backup stored at /NextcloudData/ncdata/data/ncp-update-backups/nextcloud-bkp_20241206_1733468218-28.0.12.tar.gz
Clean up...
Nextcloud update finished successfully.
ncp-autoupdate-nc from Dec-7, 7:42 CET: failure 28.0.13->29.0.10
/etc/cron.daily/ncp-autoupdate-nc:
Running as unit: ncp-update-nc.service
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  24  817k   24  202k    0     0   983k      0 --:--:-- --:--:-- --:--:--  983k 100  817k  100  817k    0     0  2803k      0 --:--:-- --:--:-- --:--:-- 2799k
Current   Nextcloud version 28.0.13
Requested Nextcloud version latest
Selected  Nextcloud version 29.0.10
INFO: You have requested an update to the latest available version that can be performed directly, which is '29.0.10'. Updates can only performed to the next major version (or the latest minor version of the current major version). If you run ncp-update-nc again after this update completes, a new version might be available.
Download Nextcloud 29.0.10...
Back up current instance...
check free space...
Maintenance mode enabled
backup database...
backup files...
backup /var/www/nextcloud-bkp_20241207_1733552742.tar.gz generated
Maintenance mode disabled
Storing backup at '/var/www/nextcloud-bkp_20241207_1733552742-28.0.13.tar.gz'...
Install Nextcloud 29.0.10...
Fix permissions...
Upgrade...
Nextcloud is not installed - only a limited number of commands are available


  Command "upgrade" is not defined.


Abort
Clean up...
Rolling back to backup /var/www/nextcloud-bkp_20241207_1733552742-28.0.13.tar.gz...
PHP Warning:  Undefined array key "dbtableprefix" in Command line code on line 1
check free space...
extracting backup file /var/www/nextcloud-bkp_20241207_1733552742-28.0.13.tar.gz...
restore files...
Restoring old '/var/www/nextcloud/data' to '/var/www/nextcloud/data'...
restore database...
No datadir found in backup
Maintenance mode disabled
Starting scan for user 1 out of 4 (xxxx)
Starting scan for user 2 out of 4 (yyyy)
Starting scan for user 3 out of 4 (zzzz)
Starting scan for user 4 out of 4 (www)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 1153    | 10357 | 0   | 356     | 0       | 0      | 00:00:10     |
+---------+-------+-----+---------+---------+--------+--------------+
Installing template 'php/opcache.ini.sh'...
System config value tempdirectory set to string /NextcloudData/ncdata/data/tmp
System config value logfile set to string /NextcloudData/ncdata/data/nextcloud.log
System config value trusted_domains => 11 set to string 2003:ed:ef33:e600:d5e0:c700:911f:d1ae
System config value trusted_domains => 1 set to string 192.168.178.100
System config value trusted_domains => 14 set to string nextcloudpi
Scanning AppData for files

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 13964   | 6640  | 00:05:35     |
+---------+-------+--------------+
Cleanup...
Rollback successful. Nothing was updated
Nextcloud update failed (or was installed already).

Configuration

Nextcloud

occ config:list system
{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "7": "nextcloudpi",
            "5": "nextcloudpi.local",
            "8": "nextcloudpi.lan",
            "3": "eisenhans.mooo.com",
            "11": "2003:ed:ef33:e600:d5e0:c700:911f:d1ae",
            "1": "192.168.178.100",
            "14": "nextcloudpi",
            "20": "eisenhans.port0.org"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.13.1",
        "overwrite.cli.url": "An unhandled exception has been thrown:\nRedisException: read error on connection to \/var\/run\/redis\/redis.sock:0 in \/var\/www\/nextcloud\/lib\/private\/Memcache\/Redis.php:76\nStack trace:\n#0 \/var\/www\/nextcloud\/lib\/private\/Memcache\/
Redis.php(76): Redis->get()\n#1 \/var\/www\/nextcloud\/lib\/private\/App\/InfoParser.php(58): OC\\Memcache\\Redis->get()\n#2 \/var\/www\/nextcloud\/lib\/private\/App\/AppManager.php(724): OC\\App\\InfoParser->parse()\n#3 \/var\/www\/nextcloud\/lib\/private\/AppFramework\/App
.php(72): OC\\App\\AppManager->getAppInfo()\n#4 \/var\/www\/nextcloud\/lib\/private\/legacy\/OC_App.php(154): OC\\AppFramework\\App::buildAppNamespace()\n#5 \/var\/www\/nextcloud\/lib\/private\/AppFramework\/Bootstrap\/Coordinator.php(119): OC_App::registerAutoloading()\n#6 
\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Bootstrap\/Coordinator.php(90): OC\\AppFramework\\Bootstrap\\Coordinator->registerApps()\n#7 \/var\/www\/nextcloud\/lib\/base.php(700): OC\\AppFramework\\Bootstrap\\Coordinator->runInitialRegistration()\n#8 \/var\/www\/nextc
loud\/lib\/base.php(1200): OC::init()\n#9 \/var\/www\/nextcloud\/console.php(48): require_once('...')\n#10 \/var\/www\/nextcloud\/occ(11): require_once('...')\n#11 {main}:\/\/eisenhans.mooo.com\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/NextcloudData\/ncdata\/data\/tmp",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "maintenance": false,
        "logfile": "\/NextcloudData\/ncdata\/data\/nextcloud.log",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "loglevel": "2",
        "log_type": "file",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "theme": "",
        "data-fingerprint": "9a4a1edbdf315fda14b1aabdbac8cde8"
    },
    "apps": {
        "activity": {
            "enabled": "yes",
            "installed_version": "2.20.0",
            "types": "filesystem"
        },
        "backgroundjob": {
            "lastjob": "2897"
        },
        "calendar": {
            "enabled": "yes",
            "installed_version": "4.7.16",
            "types": ""
        },
        "circles": {
            "enabled": "yes",
            "installed_version": "28.0.0",
            "loopback_tmp_path": "\/",
            "loopback_tmp_scheme": "https",
            "maintenance_run": "0",
            "maintenance_update": "{\"3\":1733596802,\"2\":1733598903,\"1\":1733599203}",
            "migration_22": "1",
            "migration_run": "0",
            "types": "filesystem,dav"
        },
        "cloud_federation_api": {
            "enabled": "yes",
            "installed_version": "1.11.0",
            "types": "filesystem"
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "logging"
        },
        "contacts": {
            "enabled": "yes",
            "installed_version": "5.5.3",
            "types": "dav"
        },
        "contactsinteraction": {
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "dav"
        },
        "core": {
            "backgroundjobs_mode": "cron",
            "emailTestSuccessful": "0",
            "files_metadata": "{\"photos-original_date_time\":{\"value\":null,\"type\":\"int\",\"indexed\":true,\"editPermission\":0},\"photos-size\":{\"value\":null,\"type\":\"array\",\"indexed\":false,\"editPermission\":0},\"photos-exif\":{\"value\":null,\"type\":\"array\",\"indexed\":false,\"editPermission\":0},\"photos-ifd0\":{\"value\":null,\"type\":\"array\",\"indexed\":false,\"editPermission\":0},\"photos-gps\":{\"value\":null,\"type\":\"array\",\"indexed\":false,\"editPermission\":0},\"photos-place\":{\"value\":null,\"type\":\"string\",\"indexed\":true,\"editPermission\":0},\"files-live-photo\":{\"value\":null,\"type\":\"string\",\"indexed\":false,\"editPermission\":0}}",
            "files_metadata_installed": "1",
            "installedat": "1700071777.2858",
            "lastcron": "1733599203",
            "lastupdatedat": "0",
            "metadataGenerationDone": "true",
            "moveavatarsdone": "yes",
            "oc.integritycheck.checker": "[]",
            "previewsCleanedUp": "1",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "vendor": "nextcloud"
        },
        "dashboard": {
            "enabled": "yes",
            "installed_version": "7.8.0",
            "types": ""
        },
                "dav": {
            "buildCalendarReminderIndex": "yes",
            "buildCalendarSearchIndex": "yes",
            "builtSocialSearchIndex": "yes",
            "chunks_migrated": "1",
            "enabled": "yes",
            "installed_version": "1.29.2",
            "regeneratedBirthdayCalendarsForYearFix": "yes",
            "types": "filesystem"
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": ""
        },
        "federation": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "authentication"
        },
        "files": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "mimetype_version": "28.0.13.1",
            "types": "filesystem"
        },
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.20.0",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "2.9.0",
            "types": ""
        },
        "files_reminders": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": "dav"
        },
        "files_rightclick": {
            "enabled": "no",
            "installed_version": "1.6.0",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "installed_version": "1.20.0",
            "types": "filesystem"
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "filesystem,dav"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.21.0",
            "types": "filesystem,dav"
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.17.0",
            "types": "logging"
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.13.0",
            "types": "logging"
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.16.0",
            "types": "authentication"
        },
        "news": {
            "enabled": "no",
            "installed_version": "24.0.0",
            "types": ""
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.17.0",
            "pub_date": "Thu, 24 Oct 2019 00:00:00 +0200",
            "types": "logging"
        },
        "nextcloudpi": {
            "enabled": "yes",
            "installed_version": "0.0.2",
            "types": ""
        },
        "notes": {
            "enabled": "yes",
            "installed_version": "4.11.0",
            "types": ""
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.16.0",
            "types": "logging"
        },
        "notify_push": {
            "base_endpoint": "https:\/\/nextcloudpi\/push",
            "cookie": "***REMOVED SENSITIVE VALUE***",
            "enabled": "yes",
            "installed_version": "0.7.0",
            "types": "filesystem"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.16.4",
            "types": "authentication"
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "authentication"
        },
                "photos": {
            "enabled": "yes",
            "installed_version": "2.4.0",
            "lastPlaceMappedUser": "ncp",
            "lastPlaceMappingDone": "true",
            "types": "dav,authentication"
        },
        "preview": {
            "jpeg_quality": "60"
        },
        "previewgenerator": {
            "enabled": "yes",
            "heightSizes": "256",
            "installed_version": "99.99.99",
            "squareSizes": "32 256",
            "types": "filesystem",
            "widthSizes": "256 384"
        },
        "privacy": {
            "enabled": "yes",
            "installed_version": "1.12.0",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "prevent_group_restriction"
        },
        "recommendations": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "types": ""
        },
        "related_resources": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": ""
        },
        "richdocuments": {
            "disable_certificate_verification": "yes",
            "enabled": "yes",
            "installed_version": "8.3.12",
            "types": "prevent_group_restriction",
            "wopi_url": "An unhandled exception has been thrown:\nRedisException: read error on connection to \/var\/run\/redis\/redis.sock:0 in \/var\/www\/nextcloud\/lib\/private\/Memcache\/Redis.php:76\nStack trace:\n#0 \/var\/www\/nextcloud\/lib\/private\/Memcache\/Redis.php(76): Redis->get()\n#1 \/var\/www\/nextcloud\/lib\/private\/App\/InfoParser.php(58): OC\\Memcache\\Redis->get()\n#2 \/var\/www\/nextcloud\/lib\/private\/App\/AppManager.php(724): OC\\App\\InfoParser->parse()\n#3 \/var\/www\/nextcloud\/lib\/private\/AppFramework\/App.php(72): OC\\App\\AppManager->getAppInfo()\n#4 \/var\/www\/nextcloud\/lib\/private\/legacy\/OC_App.php(154): OC\\AppFramework\\App::buildAppNamespace()\n#5 \/var\/www\/nextcloud\/lib\/private\/AppFramework\/Bootstrap\/Coordinator.php(119): OC_App::registerAutoloading()\n#6 \/var\/www\/nextcloud\/lib\/private\/AppFramework\/Bootstrap\/Coordinator.php(90): OC\\AppFramework\\Bootstrap\\Coordinator->registerApps()\n#7 \/var\/www\/nextcloud\/lib\/base.php(700): OC\\AppFramework\\Bootstrap\\Coordinator->runInitialRegistration()\n#8 \/var\/www\/nextcloud\/lib\/base.php(1200): OC::init()\n#9 \/var\/www\/nextcloud\/console.php(48): require_once('...')\n#10 \/var\/www\/nextcloud\/occ(11): require_once('...')\n#11 {main}:\/\/eisenhans.mooo.com\/apps\/richdocumentscode_arm64\/proxy.php?req="
        },
        "richdocumentscode_arm64": {
            "enabled": "yes",
            "installed_version": "24.4.1002",
            "types": ""
        },
        "serverinfo": {
            "cached_count_filecache": "32162",
            "cached_count_storages": "7",
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": ""
        },
        "settings": {
            "enabled": "yes",
            "installed_version": "1.10.1",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "filesystem"
        },
        "support": {
            "SwitchUpdaterServerHasRun": "yes",
            "enabled": "yes",
            "installed_version": "1.11.1",
            "types": "session"
        },
        "survey_client": {
            "enabled": "yes",
            "installed_version": "1.16.0",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.18.0",
            "types": "logging"
        },
        "tasks": {
            "enabled": "yes",
            "installed_version": "0.16.1",
            "types": ""
        },
        "text": {
            "enabled": "yes",
            "installed_version": "3.9.2",
            "types": "dav"
        },
        "theming": {
            "backgroundMime": "image\/png",
            "enabled": "yes",
            "installed_version": "2.3.0",
            "logoDimensions": "500x500",
            "logoMime": "image\/svg+xml",
            "name": "NextCloudPi",
            "slogan": "***REMOVED SENSITIVE VALUE***",
            "types": "logging",
            "url": "***REMOVED SENSITIVE VALUE***"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.17.0",
            "types": ""
        },
        "updatenotification": {
            "enabled": "no",
            "installed_version": "1.17.0",
            "types": ""
        },
        "user_migration": {
            "enabled": "yes",
            "installed_version": "5.0.0",
            "types": ""
        },
        "user_status": {
            "enabled": "yes",
            "installed_version": "1.8.1",
            "types": ""
        },
                "viewer": {
            "enabled": "yes",
            "installed_version": "2.2.0",
            "types": ""
        },
        "weather_status": {
            "enabled": "yes",
            "installed_version": "1.8.0",
            "types": ""
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "2.10.0",
            "types": "filesystem"
        }
    }
}

Apps

occ app:list
Enabled:
  - activity: 2.20.0
  - calendar: 4.7.16
  - circles: 28.0.0
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contacts: 5.5.3
  - contactsinteraction: 1.9.0
  - dashboard: 7.8.0
  - dav: 1.29.2
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_external: 1.20.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - firstrunwizard: 2.17.0
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - nextcloud_announcements: 1.17.0
  - nextcloudpi: 0.0.2
  - notes: 4.11.0
  - notifications: 2.16.0
  - notify_push: 0.7.0
  - oauth2: 1.16.4
  - password_policy: 1.18.0
  - photos: 2.4.0
  - previewgenerator: 99.99.99
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - richdocuments: 8.3.12
  - richdocumentscode_arm64: 24.4.1002
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - support: 1.11.1
  - survey_client: 1.16.0
  - systemtags: 1.18.0
  - tasks: 0.16.1
  - text: 3.9.2
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - user_migration: 5.0.0
  - user_status: 1.8.1
  - viewer: 2.2.0
  - weather_status: 1.8.0
  - workflowengine: 2.10.0
Disabled:
  - admin_audit: 1.18.0
  - bruteforcesettings: 2.8.0
  - encryption: 2.16.0
  - suspicious_login: 6.0.0
  - twofactor_totp: 10.0.0-beta.2
  - updatenotification: 1.18.0 (installed 1.17.0)
  - user_ldap: 1.19.0

My 2ct:

  • reboot server
  • disable ncp-update-nc in cli ncp-config or :4443webui
  • (no rollback / only restore backup recommended)
  • but here I would try ncp-update-nc not 0 for latest but 29.0.10 for dedicated choice

Hi geoW,

thanks for the quick response.

  • I did reboot the server already after my “fixes”, forgot to mention that
  • I disabled ncp-update via the webui --will manually monitor potential failures or withdraws first and then maybe only update to the pre-last version. I think later if and how a “delayed automatic update” could be realized.
  • How to restore a specific nextcloud version? I’d like to restore 28.0.12 as 28.0.13 got also withdrawn. So right now, I’m on a buggy version, which is maybe one reason why the update to the “non-withdrawn” version 29.0.10 did not work.
    • I’m not sure what exactly is backed-up. I guess it’s
      • ncp (nextcloudpi, today at v1.55.3)
      • nc (nextcloud, today at 28.0.13)
      • my data
    • Ideally only a backup of nc 28.0.12 is restored. Do you know where to find that backup?
    • Or would ncp-update-nc 28.0.12 do the job, i.e. not restoring a backup? Is that what you had in mind? The usage of ncp-update-nc indicates that I can select any arbitrary version. Do you --or anyone else in this forum-- have experience with that approach?

Thanks,
Thomas

It depends on your setup, take a look at the logs, or better download the log (webui) and search for backup.

There you should put 29.0.10 instead of 0 for latest.
If this upgrade fails, examine the error messages.
If link-sharing is not needed/used I would stay on withdrawn version and wait for announced hotfix.

Hi geoW,

I’m not sure if manually upgrading to 29.0.10 as you suggest is the best trial.
The last automated update to 29.0.10 did fail --that was the start of the disaster I’m facing.

The previous automated upgrade from 28.0.12 to the later withdrawn 28.0.13 did work.

Ideally, I’d like to downgrade to 28.0.12 with ncp-update-nc 28.0.12. I thought, I’ve seen downgrading is not supported, but cannot find a reference.

Do you think ncp-update-nc 28.0.12 will work from my current 28.0.13?

If yes, the manual (or automated) update to the recently (since four days not withdrawn) published 28.0.14 should work.

That’s nice, now I would try to go to 28.014

It is in nextcloud admin-doc, there it is about mjnor numbers, but here with ncp there is even one direction - forward; going back is as usually done by backup/restore.

You have been shown much but not all we need :grinning:. I assume your inctance is running on debian 11 bullseye?
We need the errror messages of wrong going updates.

Time is running short in christmas seasen and nc28 going EOL.
Take the pressure from your production instance, setup a test-nc on a spare sd-card starting with an image -say- 1.52.x and run all upgrades, test going backwards if you like, but is is not as easy at it seems to be, so even running forward you will run in to issues, keep in mind, there will be an dist-upgrade on your way, reading here and on github issues may help you.

No, I’m on bookwork already.
I started my nextcloudpi production system end of 2023 and all automated updates went well so far. While checking ncp.log early December, if found

NCP doesn't support Nextcloud versions greater than 28 with Debian 11

and learned Debian updates need to be triggered manually via ncp-dist-upgrade. I did that and the update went fine, except some minor, opcache related error messages which I could fix.

I’ve added those in my very first posting under Log Entries above. If you could have a look into the failed 28.0.13 → 29.0.10 update: that would be awesome.

…maybe even using a copy of the production system w/o the data, where I could test your suggested update to 28.0.14. My system runs directly from an USB-SDD, no sdcard, but re-creating an sdcard image should work. I’ll have some time during the Xmas break.

yes, it did already help :wink: thanks for your support so far.

Having a recent backup.
Now I would run an nc update 0 (latest version), it should get 28.0.14,
or throw error messages.

On cli do the following commands:

sudo ncc maintenance:repair
pi@nextcloudpi:~ $ sudo ncc maintenance:repair
 - Repair MySQL collation
     - All tables already have the correct collation -> nothing to do
 - Clean tags and favorites
     - 0 tags of deleted users have been removed.
     - 0 tags for delete files have been removed.
     - 0 tag entries for deleted tags have been removed.
     - 0 tags with no entries have been removed.
 - Repair invalid shares
 - Move .step file of updater to backup location
 - Add move avatar background job
     - Repair step already executed
 - Add preview cleanup background jobs
 - Migrate oauth2_clients table to nextcloud schema
     - Update the oauth2_access_tokens table schema.
     - Update the oauth2_clients table schema.
     - Delete clients (and their related access tokens) with the redirect_uri starting with oc:// or ending with *
 - Repair language codes
 - Add log rotate job
 - Clear frontend caches
     - Image cache cleared
     - JS cache cleared
 - Clear every generated avatar
 - Add preview background cleanup job
 - Queue a one-time job to cleanup old backups of the updater
 - Cleanup invalid photocache files for carddav
 - Add background job to cleanup login flow v2 tokens
 - Remove potentially over exposing share links
     - No need to remove link shares.
 - Clear access cache of projects
 - Reset generated avatar flag
 - Keep legacy encryption enabled
 - Check encryption key format
 - Remove old dashboard app config data
 - Add job to cleanup the bruteforce entries
 - Queue a one-time job to check for user uploaded certificates
 - Repair DAV shares
 - Add background job to set the lookup server share state for users
 - Add token cleanup job
 - Clean up abandoned apps
 - Add possibly missing system config
 - Add AI tasks cleanup jobs
 - Queue a job to generate metadata
 - migrate lazy config values
 - Cache logo dimension to fix size in emails on Outlook
     - Logo dimensions are already known
 - Remove legacy ".ocdata" file
 - Add cleanup-deleted-users background job
 - AppAPI Daemons configuration GPU params update
     - Daemons configuration GPU params updated: 0
 - Upgrading Circles App
 - Fix component of birthday calendars
     - 3 birthday calendars updated.
 - Regenerating birthday calendars to use new icons and fix old birthday events without year
     - Repair step already executed
 - Fix broken values of calendar objects
    0 [->--------------------------]
 - Registering building of calendar search index as background job
     - Repair step already executed
 - Register building of social profile search index as background job
     - Repair step already executed
 - Registering background jobs to update cache for webcal calendars
     - Added 0 background jobs to update webcal calendars
 - Registering building of calendar reminder index as background job
     - Repair step already executed
 - Clean up orphan event and contact data
     - 0 events without a calendar have been cleaned up
     - 0 properties without an events have been cleaned up
     - 0 changes without a calendar have been cleaned up
     - 0 cached events without a calendar subscription have been cleaned up
     - 0 changes without a calendar subscription have been cleaned up
     - 0 contacts without an addressbook have been cleaned up
     - 0 properties without a contact have been cleaned up
     - 0 changes without an addressbook have been cleaned up
 - Remove activity entries of private events
     - Removed 0 activity entries
 - Clean up old calendar subscriptions from deleted users that were not cleaned-up
    0 [----->----------------------]
     - 0 calendar subscriptions without an user have been cleaned up
 - Remove invalid object properties
     - 0 invalid object properties removed.
 - Fix the share type of guest shares when migrating from ownCloud
 - Copy the share password into the dedicated column
 - Set existing shares as accepted
 - Register Maps MIME types
     - Registering the mimetype...
     - The mimetype was successfully registered.
 - Remove the unused News update job
     - Job does not exist, all good
 - Clean up meta table
 - Show a hint about the new editor to existing users
 - Set binary permissions
 - Update OAuth token expiration times
 - init metadata
 - Switches from default updater server to the customer one if a valid subscription is available
     - Repair step already executed
 - Send an admin notification if monthly report is disabled
 - Force-reset all Text document sessions
 - Initialize migration of background images from dashboard to theming app
 - Add background job to check for backup codes
 - Populating added database structures for workflows
pi@nextcloudpi:~ $

A quick check of database health.

And:

sudo systemctl status ncp-update-nc.service
pi@nextcloudpi:~ $ sudo systemctl status ncp-update-nc.service
Unit ncp-update-nc.service could not be found.
pi@nextcloudpi:~ $

Last one shows whether there is a hanging update job, with which error the last update job ended.

I decided. to do small steps and firstly work on a sdcard copy of my SDD-based production system. Adapting cmdline.txt in the boot partition and /etc/fstab in the root partition with the correct PARTUUIDs is straigt forward. Copying the data to a slow sdcard took a while though.

As you suggested, I did execute the two validation steps prior to the update.

sudo ncc maintenance:repair went well, minor output differences to your version above
  • Repair MySQL collation
    • All tables already have the correct collation → nothing to do
  • Clean tags and favorites
    • 0 tags of deleted users have been removed.
    • 0 tags for delete files have been removed.
    • 0 tag entries for deleted tags have been removed.
    • 0 tags with no entries have been removed.
  • Repair invalid shares
  • Move .step file of updater to backup location
  • Add move avatar background job
    • Repair step already executed
  • Add preview cleanup background jobs
  • Migrate oauth2_clients table to nextcloud schema
    • Update the oauth2_access_tokens table schema.
    • Update the oauth2_clients table schema.
    • Delete clients (and their related access tokens) with the redirect_uri starting with oc:// or ending with *
  • Fix potential broken mount points
    • No mounts updated
  • Repair language codes
  • Add log rotate job
  • Clear frontend caches
    • Image cache cleared
    • JS cache cleared
  • Clear every generated avatar
  • Add preview background cleanup job
  • Queue a one-time job to cleanup old backups of the updater
  • Cleanup invalid photocache files for carddav
  • Add background job to cleanup login flow v2 tokens
  • Remove potentially over exposing share links
    • No need to remove link shares.
  • Clear access cache of projects
  • Reset generated avatar flag
  • Keep legacy encryption enabled
  • Check encryption key format
  • Remove old dashboard app config data
  • Add job to cleanup the bruteforce entries
  • Queue a one-time job to check for user uploaded certificates
  • Repair DAV shares
  • Add background job to set the lookup server share state for users
  • Add token cleanup job
  • Clean up abandoned apps
  • Add possibly missing system config
  • Add AI tasks cleanup job
  • Queue a job to generate metadata
  • Cache logo dimension to fix size in emails on Outlook
    • Logo dimensions are already known
  • Add cleanup-deleted-users background job
  • Upgrading Circles App
  • Fix component of birthday calendars
    • 4 birthday calendars updated.
  • Regenerating birthday calendars to use new icons and fix old birthday events without year
    • Repair step already executed
  • Fix broken values of calendar objects
    0 [>---------------------------]
  • Registering building of calendar search index as background job
    • Repair step already executed
  • Register building of social profile search index as background job
    • Repair step already executed
  • Registering background jobs to update cache for webcal calendars
    • Added 0 background jobs to update webcal calendars
  • Registering building of calendar reminder index as background job
    • Repair step already executed
  • Clean up orphan event and contact data
    • 0 events without a calendar have been cleaned up
    • 0 properties without an events have been cleaned up
    • 0 changes without a calendar have been cleaned up
    • 0 cached events without a calendar subscription have been cleaned up
    • 0 changes without a calendar subscription have been cleaned up
    • 0 contacts without an addressbook have been cleaned up
    • 0 properties without a contact have been cleaned up
    • 0 changes without an addressbook have been cleaned up
  • Remove activity entries of private events
    • Removed 0 activity entries
  • Clean up old calendar subscriptions from deleted users that were not cleaned-up
    0/1 [>---------------------------] 0%
    1/1 [============================] 100%
    • 0 calendar subscriptions without an user have been cleaned up
  • Remove invalid object properties
    • 0 invalid object properties removed.
  • Fix the share type of guest shares when migrating from ownCloud
  • Copy the share password into the dedicated column
  • Set existing shares as accepted
  • Clean up meta table
  • Show a hint about the new editor to existing users
  • Set binary permissions
  • Update OAuth token expiration times
  • Switches from default updater server to the customer one if a valid subscription is available
    • Repair step already executed
  • Send an admin notification if monthly report is disabled
  • Force-reset all Text document sessions
  • Initialize migration of background images from dashboard to theming app
  • Add background job to check for backup codes
  • Populating added database structures for workflows

sudo systemctl status ncp-update-nc.service did lead to exactly the same output as yours.

ncp-update-nc 0 did identify 28.0.14 as the latest release and went fine.

here's the ncp-update-nc log

Running as unit: ncp-update-nc.service
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0^M100 5588 0 5588 0 0 11578 0 --:–:-- --:–:-- --:–:-- 11569^M100 804k 0 804k 0 0 1322k 0 --:–:-- --:–:-- --:–:-- 1323k
Current Nextcloud version 28.0.13
Requested Nextcloud version latest
Selected Nextcloud version 28.0.14
INFO: You have requested an update to the latest available version that can be performed directly, which is ‘28.0.14’. Updates can only performed to the next major version (or the latest minor version of the current major version). If you run ncp-update-nc again after this update completes, a new version might be available.
Download Nextcloud 28.0.14…
Back up current instance…
check free space…
Maintenance mode enabled
backup database…
backup files…
backup /var/www/nextcloud-bkp_20241218_1734526943.tar.gz generated
Maintenance mode disabled
Storing backup at ‘/var/www/nextcloud-bkp_20241218_1734526943-28.0.13.tar.gz’…
Install Nextcloud 28.0.14…
Fix permissions…
Upgrade…
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Disabled incompatible app: calendar
Disabled incompatible app: contacts
Disabled incompatible app: notes
Disabled incompatible app: notify_push
Disabled incompatible app: richdocuments
Disabled incompatible app: richdocumentscode_arm64
Disabled incompatible app: tasks
Disabled incompatible app: user_migration
Starting code integrity check…
Finished code integrity check
Update successful
Turned off maintenance mode
Resetting log level
Done.
Done.
All tables already up to date!

  • Repair MySQL collation
    • All tables already have the correct collation → nothing to do
  • Clean tags and favorites
    • 0 tags of deleted users have been removed.
    • 0 tags for delete files have been removed.
    • 0 tag entries for deleted tags have been removed.
    • 0 tags with no entries have been removed.
  • Repair invalid shares
  • Move .step file of updater to backup location
  • Add move avatar background job
    • Repair step already executed
  • Add preview cleanup background jobs
  • Migrate oauth2_clients table to nextcloud schema
    • Update the oauth2_access_tokens table schema.
    • Update the oauth2_clients table schema.
    • Delete clients (and their related access tokens) with the redirect_uri starting with oc:// or ending with *
  • Fix potential broken mount points
    • No mounts updated
  • Repair language codes
  • Add log rotate job
  • Clear frontend caches
    • Image cache cleared
    • JS cache cleared
  • Clear every generated avatar
  • Add preview background cleanup job
  • Queue a one-time job to cleanup old backups of the updater
  • Cleanup invalid photocache files for carddav
  • Add background job to cleanup login flow v2 tokens
  • Remove potentially over exposing share links
    • No need to remove link shares.
  • Clear access cache of projects
  • Reset generated avatar flag
  • Keep legacy encryption enabled
  • Check encryption key format
  • Remove old dashboard app config data
  • Add job to cleanup the bruteforce entries
  • Queue a one-time job to check for user uploaded certificates
  • Repair DAV shares
  • Add background job to set the lookup server share state for users
  • Add token cleanup job
  • Clean up abandoned apps
  • Add possibly missing system config
  • Add AI tasks cleanup job
  • Queue a job to generate metadata
  • Cache logo dimension to fix size in emails on Outlook
    • Logo dimensions are already known
  • Add cleanup-deleted-users background job
  • Remove shares of old group memberships
  • Repair mime types
  • Validate the phone number and store it in a known format for search
    • WARNING: Can not validate phone numbers without default_phone_region being set in the config file
  • Handle outdated scheduling events
    • Cleaning up old scheduling events
  • Upgrading Circles App
  • Fix component of birthday calendars
    • 4 birthday calendars updated.
  • Regenerating birthday calendars to use new icons and fix old birthday events without year
    • Repair step already executed
  • Fix broken values of calendar objects
    0 [>---------------------------]
  • Registering building of calendar search index as background job
    • Repair step already executed
  • Register building of social profile search index as background job
    • Repair step already executed
  • Registering background jobs to update cache for webcal calendars
    • Added 0 background jobs to update webcal calendars
  • Registering building of calendar reminder index as background job
    • Repair step already executed
  • Clean up orphan event and contact data
    • 0 events without a calendar have been cleaned up
    • 0 properties without an events have been cleaned up
    • 0 changes without a calendar have been cleaned up
    • 0 cached events without a calendar subscription have been cleaned up
    • 0 changes without a calendar subscription have been cleaned up
    • 0 contacts without an addressbook have been cleaned up
    • 0 properties without a contact have been cleaned up
    • 0 changes without an addressbook have been cleaned up
  • Remove activity entries of private events
    • Removed 0 activity entries
  • Clean up old calendar subscriptions from deleted users that were not cleaned-up
    0/1 [>---------------------------] 0%
    1/1 [============================] 100%
    • 0 calendar subscriptions without an user have been cleaned up
  • Remove invalid object properties
    • 0 invalid object properties removed.
  • Fix the share type of guest shares when migrating from ownCloud
  • Copy the share password into the dedicated column
  • Set existing shares as accepted
  • Clean up meta table
  • Show a hint about the new editor to existing users
  • Set binary permissions
  • Update OAuth token expiration times
  • Switches from default updater server to the customer one if a valid subscription is available
    • Repair step already executed
  • Send an admin notification if monthly report is disabled
  • Force-reset all Text document sessions
  • Initialize migration of background images from dashboard to theming app
  • Add background job to check for backup codes
  • Populating added database structures for workflows
    Update completed successfully.
    Backup stored at /NextcloudData/ncdata/data/ncp-update-backups/nextcloud-bkp_20241218_1734526943-28.0.13.tar.gz
    Clean up…
    Nextcloud update finished successfully.

One question here: There’s a section talking about Disabled incompatible apps like

Disabled incompatible app: calendar
Disabled incompatible app: contacts

However, it seems after the update, none of the apps listed is disabled. Any comment on this?

I’ll wait a few more days to see if there are upcoming issues /w 28.0.14 and will update also my production system.

In theory, I could right away try another ncp-update-nc 0 which should lead to 29.0.10.

  • Would you recommend that?
    The first attempt to upgrade to 29.0.10 (from the buggy 28.0.13) did not work.
  • How to check before running the actual update what the next version is planned to be installed?
    Seems ncp-update-nc automatically performs an update once there is one. Ideally, there is a potential possibility to stop the update once the identified version number is displayed.
  • I’m of course following Releases · nextcloud/server · GitHub and that looks like v29.0.10 is the next logical release, even if that one is more than a week older than the current v28.0.14.

A somewhat related question regarding php-versions:

  • Do I need to perform php-updates manually, or is there a possibility that nextcloudpi automatically updates to the recommended php-version?
  • According to Releases and PHP versions · nextcloud/server Wiki · GitHub, already my very first nextcloudpi installation in Dec-2023 was php-wise behind: ncp was 27.1.3.2 and php8.1 whereas php8.2 was recommended.
  • bookworm supports up to php8.4; I know how to update that for debian and appache. So, if there is no automatic way, I know what to do --need to put it on my maintenance task list.

Very nice, all went as expected in my eyes, even the messages about the apps - I don’t Know others. Some health check - ok. So your instance should run nicely, take watch on the admin-account logs, ncp data and so on.

Yes, of course, nc28 goes EOL end this month/year, it got its last update with 28.0.14 after the regular monthly update 28.0.13 was withdrawn, so the delay, sure you read about.
And this is all you can do in front of updating, read the changelogs on github nc and ncp.

It should be done without user intervention, but you should keep an eye on it and remember if you have to intervene. But now your nextcloud should run as on the first day. :grinning: