FIXED: Web interface inaccessible after upgrade from NC28 to NC29, but sync still running

oucil · September 24, 2024, 6:20pm

Nextcloud version: 29.0.3
Operating system and version: Rocky Linux 9.4
Apache or nginx version: Apache 2.4.57
PHP version: 8.3.9

The issue you are facing:
I can no longer access the web interface, I’m getting the following error…

Internal Server Error

The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log.
Technical details

    Remote Address: ############
    Request ID: 1013vo1ZeeorDerWcgTy

The overall system continues to function and sync as expected, only the front end web interface is inaccessible. My phone, tablet, and desktop are all able to connect and browse normally using the NC native apps/clients.

I should also mention, this began with NC29.0.0, and that the only reason I’m on NC29.0.3 is because I was hoping it was an issue that would resolve with a new point release.

I perform all of my upgrades via an ansible playbook I’ve been maintaining for years. I’ve successfully upgraded countless times since NC version 10. I upgrade through every point release, and in this case was up to NC28.10 before making the jump to NC29.

I remove the entire NC root directory after backing up only the config.php file. My data dir is in a different location, as is my cache dir.

The basic process is:

Download, expand, and place all NC files into the new root.
Replace the config.php with my backed up copy
Re-apply file ownerships
Disable maintenance mode
Run the OCC update from the CLI (which completes successfully)
Remove any /ext_apps/ if there are any
Import all pre-config settings from a json file…

{
    "system": {
        "trusted_domains": ["localhost","mydomain.com"],
        "log_type": "syslog",
        "logfile": "",
        "loglevel": "3",
        "redis": {
            "host": "/run/redis/redis.sock",
            "port": "0",
            "dbindex": "3",
            "timeout": "0.0"
        },
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "cache_path": "/path/to/cache",
        "apps_paths": {
          "0": {
            "path": "/path/to/nc_root/apps",
            "url": "/apps",
            "writable": false
          },
          "1": {
            "path": "/path/to/nc_root/ext_apps",
            "url": "/ext_apps",
            "writable": true
          }
        },
        "upgrade.disable-web": true,
        "defaultapp": "apporder",
        "default_phone_region": "CA",
        "blacklisted_files": {},
        "mail_smtpmode": "smtp",
        "mail_smtphost": "127.0.0.1",
        "mail_smtpport": 25,
        "mail_smtpauth": true,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpname": "...",
        "mail_smtppassword": "..."
    }
}

Run the NC database maintenance scripts

    - db:convert-filecache-bigint
    - db:add-missing-columns
    - db:add-missing-indices
    - db:add-missing-primary-keys

Disable the App Store
Register maintenance cron job
Re-install and enable all modules I use:

    - announcementcenter
    - calendar
    - contacts
    - files_external
    - fulltextsearch
    - fulltextsearch_elasticsearch
    - files_fulltextsearch
    - keeweb
    - maps
    - memories
    - metadata
    - phonetrack
    - previewgenerator
    - side_menu
    - spreed
    - tasks
    - twofactor_totp

Import external mounts list from a json file
Import app settings from a json file

{
    "apps": {
        "preview": {
            "jpeg_quality ": "80"
        },
        "files": {
            "max_chunk_size": "20971520"
        },
        "fulltextsearch": {
            "app_navigation": "1",
            "enabled": "yes",
            "search_platform": "OCA\\FullTextSearch_ElasticSearch\\Platform\\ElasticSearchPlatform",
            "types": ""
        },
        "fulltextsearch_elasticsearch": {
            "analyzer_tokenizer": "standard",
            "elastic_host": "http://localhost:9200",
            "elastic_index": "nextcloud",
            "enabled": "yes",
            "types": ""
        },
        "files_fulltextsearch": {
            "enabled": "yes",
            "files_audio": "0",
            "files_encrypted": "0",
            "files_external": "0",
            "files_federated": "0",
            "files_group_folders": "1",
            "files_image": "0",
            "files_local": "1",
            "files_office": "1",
            "files_pdf": "1",
            "files_size": "20",
            "types": "filesystem"
        },
        "theming": {
            "enabled": "yes",
            "cachebuster": "7",
            "imprintUrl": "....",
            "name": "....",
            "privacyUrl": "....",
            "slogan": "....",
            "url": "....",
            "types": "logging"
        }
    }
}

Register remote file scan cron job
Refresh all SELinux file contexts
Restart Apache
Start thumbnail generation process.

All of this works without issue and the upgrade completes successfully. All of my apps reconnect to the server without errors. But as soon as I try to bring up the web interface I get the following errors in my syslog:

Since I can't access the web interface, the following are from the syslog after making a single request for the login screen...

Sep 24 11:04:09 ############ Nextcloud[272142]: {"reqId":"1013vo1ZeeorDerWcgTy","level":3,"time":"2024-09-24T15:04:09+00:00","remoteAddr":"#########","user":"admin","app":"index","method":"GET","url":"/index.php/apps/files/","message":"{\"Exception\":\"ReflectionException\",\"Message\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\",\"Code\":-1,\"Trace\":[{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":469,\"function\":\"__construct\",\"class\":\"ReflectionClass\",\"type\":\"->\",\"args\":[\"OCA\\\\Files\\\\Controller\\\\AjaxController\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":161,\"function\":\"getAttributeRoutes\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"files\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":279,\"function\":\"loadRoutes\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"files\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":330,\"function\":\"findMatchingRoute\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"/apps/files/\"]},{\"file\":\"/srv/apps/nextcloud/lib/base.php\",\"line\":1050,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"/apps/files/\"]},{\"file\":\"/srv/apps/nextcloud/index.php\",\"line\":49,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\",\"args\":[]}],\"File\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"Line\":469,\"message\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\",\"exception\":{},\"CustomMessage\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\"}","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0","version":"29.0.3.4"}

Sep 24 11:04:10 ############ Nextcloud[272142]: {"reqId":"eHhT8qk0U6SbA5X7giaj","level":3,"time":"2024-09-24T15:04:10+00:00","remoteAddr":"#########","user":"admin","app":"index","method":"GET","url":"/index.php/apps/files/preview-service-worker.js","message":"{\"Exception\":\"ReflectionException\",\"Message\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\",\"Code\":-1,\"Trace\":[{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":469,\"function\":\"__construct\",\"class\":\"ReflectionClass\",\"type\":\"->\",\"args\":[\"OCA\\\\Files\\\\Controller\\\\AjaxController\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":161,\"function\":\"getAttributeRoutes\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"files\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":279,\"function\":\"loadRoutes\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"files\"]},{\"file\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"line\":330,\"function\":\"findMatchingRoute\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"/apps/files/preview-service-worker.js\"]},{\"file\":\"/srv/apps/nextcloud/lib/base.php\",\"line\":1050,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"/apps/files/preview-service-worker.js\"]},{\"file\":\"/srv/apps/nextcloud/index.php\",\"line\":49,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\",\"args\":[]}],\"File\":\"/srv/apps/nextcloud/lib/private/Route/Router.php\",\"Line\":469,\"message\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\",\"exception\":{},\"CustomMessage\":\"Class \\\"OCA\\\\Files\\\\Controller\\\\AjaxController\\\" does not exist\"}","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0","version":"29.0.3.4"}

From my research into the error, it doesn’t seem to be a prevelant one so there’s not much out there, the controller being called don’t exist in NC29???

This is where I’m lost. I’ve tried many things to resolve this:

Reboot server
Run occ files:cleanup
Clear NC js and css cache files manually
Run occ maintenance:repair
Manually update/re-install the files module with occ app:update files but it’s always found and up to date
Re-run occ upgrade
Disabled/removed ALL add-ons then re-perform all of the above.

Nothing has had any effect, always the same errors.

Curiously, the occ integrity:check-core is reporting a long list of files that are failing even through I’m deleting all old files prior to the upgrade and replacing with only what comes from the TAR, and then whatever is installed with the extra apps.

Here’s a small selection:

  - EXTRA_FILE:
    - lib/private/legacy/OC_DB.php:
      - expected:
      - current: 25aed9b41eaba325b81e4e9f9d916ff8af2effd203501de0927a807809da4e645386de882761aa593d146f0922e4ea46d245cb187e28485842d1d3a9c6f7079f
    - lib/private/legacy/OC_DB_StatementWrapper.php:
      - expected:
      - current: 9930f72820a83d3da286819918febbac76974431e56fdebf9b7b9e76369a4582a2e5dcff7818d28e3a765487dfb0486526d6e8aa3f9b7dd912370365eeca3597
    - lib/private/Migration/MetadataManager.php:
      - expected:
      - current: e19a3b714a68e9fd0d2aad234ab8a4bfc14ad104e658d6b5a8025e1761cfd11e6579496b06af2c0431b2d2194b035bdaeb35e1ee400e29d6a5ebe6a0790b178e
    - lib/private/Migration/Exceptions/AttributeException.php:
      - expected:
      - current: d5a813ba5256ab2cc84927f4912a45f5090b68293e21f7fc7b899d903f855443b27c756073656996e31aab5343bba25fa0c8d8ece72319f847619dcca1f92f82
    - lib/private/Repair/RepairLogoDimension.php:
      - expected:
      - current: a02e8784eeca2a93b2084a2c0407d147ef870b2c5700c028c3ff2f27e5afbc57bf81c6f0e41a446cebad55f7c28d2389fbd12d1f8e29c3c656284374bf4ae88c

    .......

There are more than 300 files listed (the list is over 600KB) from what I can see that are considered “EXTRA”.

Here’s my current config.php for refrence…

<?php
$CONFIG = array (
  'passwordsalt' => '##########',
  'secret' => '##########',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'domain1.com',
    2 => 'domain2.com',
  ),
  'datadirectory' => '/path/to/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '29.0.3.4',
  'overwrite.cli.url' => 'http://localhost',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/run/mysqld/mysqld.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '##########',
  'dbpassword' => '##########',
  'installed' => true,
  'instanceid' => '##########',
  'log_type' => 'syslog',
  'logfile' => '',
  'loglevel' => '3',
  'redis' =>
  array (
    'host' => '/run/redis/redis.sock',
    'port' => '0',
    'dbindex' => '3',
    'timeout' => '0.0',
  ),
  'filelocking.enabled' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'cache_path' => '/path/to/nextcloud/cache',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/path/to/nextcloud_root/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/path/to/nextcloud_root/ext_apps',
      'url' => '/ext_apps',
      'writable' => true,
    ),
  ),
  'upgrade.disable-web' => true,
  'defaultapp' => 'apporder',
  'default_phone_region' => 'CA',
  'blacklisted_files' =>
  array (
  ),
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => '127.0.0.1',
  'mail_smtpport' => 25,
  'mail_smtpauth' => true,
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpname' => '##########',
  'mail_smtppassword' => '##########',
  'appstoreenabled' => 'false',
  'maintenance' => false,
  'memories.exiftool_no_local' => true,
  'memories.vod.path' => '/path/to/nextcloud_root/ext_apps/memories/bin-ext/go-vod-amd64',
  'memories.vod.ffmpeg' => '/bin/ffmpeg',
  'memories.vod.ffprobe' => '/bin/ffprobe',
  'memories.db.triggers.fcu' => true,
);

Would love some help, thank you!

jtr · September 24, 2024, 7:57pm

I can’t tell you exactly what went wrong within your process for upgrading, but I can say that OC_DB.php hasn’t been in Nextcloud since v25. So it appears you have some mismatch of files within your installation folder.

oucil · September 24, 2024, 9:19pm

I think that may be the hint I needed, looks like the removal of the nextcloud root folder stopped working a while ago, silently, so it hasn’t been removing the old files when I’d thought it was.

Appreciate your help!

system · October 2, 2024, 9:19pm

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.