Upgrade from to - all files "gone". Symlinked Group Folder as culprit?

Nextcloud version (eg, 20.0.5):
Operating system and version (eg, Ubuntu 20.04): Docker production-fpm
Apache or nginx version (eg, Apache 2.4.25): ?
PHP version (eg, 7.4): ?

The issue you are facing:
After the update, the MariaDB container (10.5) ran crazy for about half an hour (disk writes on end), but finally it “died down”.
Since then, the Windows and Android clients as well as Firefox and Edge (yes, all caches cleared) do not show any files.
The files are on the servers storage just fine, all permissions unchanged (33:33 755) and within the container, I can access the data directory /var/www/data (where /html/config/config.php points to) without problems.
occ files:scan --all yields an error in one of the groupfolders since it’s directory is symlinked into the data dir (other zfs dataset) but that was no problem beforehand.
occ groupfolders:list shows the configured groupfolders but no size for the problematic groupfolder number 4 (see error log below).

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. upgrade nextcloud:production-fpm from to
  2. restart the docker stack and wait until MariaDB has done it’s db upgrade
  3. try to connect a nextcloud client or open den web interfac in Firefox or Edge: “files? which files? you never had any!”

The output of your Nextcloud log in Admin > Logging:

{"reqId":"bcOsdODizIo3l87c4BhT","level":3,"time":"2023-12-19T21:19:39+00:00","remoteAddr":"","user":"georg","app":"index","method":"GET","url":"/apps/recommendations/api/recommendations","message":"Exception thrown: OCP\\Files\\NotPermittedException","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0","version":"","exception":{"Exception":"OCP\\Files\\NotPermittedException","Message":"Could not create folder \"/__groupfolders/4\"","Code":0,"Trace":[{"function":"newFolder","class":"OC\\Files\\Node\\Folder","type":"->","args":["4"]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Helper/LazyFolder.php","line":63,"function":"call_user_func_array","args":[[["OC\\Files\\Node\\Folder"],"newFolder"],["4"]]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Helper/LazyFolder.php","line":143,"function":"__call","class":"OCA\\GroupFolders\\Helper\\LazyFolder","type":"->","args":["newFolder",["4"]]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Mount/MountProvider.php","line":294,"function":"newFolder","class":"OCA\\GroupFolders\\Helper\\LazyFolder","type":"->","args":["4"]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Mount/MountProvider.php","line":209,"function":"getFolder","class":"OCA\\GroupFolders\\Mount\\MountProvider","type":"->","args":[4]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Mount/MountProvider.php","line":162,"function":"getMount","class":"OCA\\GroupFolders\\Mount\\MountProvider","type":"->","args":[4,"/NCadm/files/Medien",31,-3,null,["OC\\Files\\Storage\\StorageFactory"],true,["OC\\User\\User"],["OCA\\GroupFolders\\ACL\\ACLManager"],[[],[],[]]]},{"function":"OCA\\GroupFolders\\Mount\\{closure}","class":"OCA\\GroupFolders\\Mount\\MountProvider","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/custom_apps/groupfolders/lib/Mount/MountProvider.php","line":143,"function":"array_map","args":[["Closure"],["*** sensitive parameters replaced ***"]]},{"file":"/var/www/html/lib/private/Files/Config/MountProviderCollection.php","line":89,"function":"getMountsForUser","class":"OCA\\GroupFolders\\Mount\\MountProvider","type":"->","args":[["OC\\User\\User"],["OC\\Files\\Storage\\StorageFactory"]]},{"file":"/var/www/html/lib/private/Files/Config/MountProviderCollection.php","line":138,"function":"getMountsFromProvider","class":"OC\\Files\\Config\\MountProviderCollection","type":"->","args":[["OCA\\GroupFolders\\Mount\\MountProvider"],["OC\\User\\User"],["OC\\Files\\Storage\\StorageFactory"]]},{"file":"/var/www/html/lib/private/Files/SetupManager.php","line":215,"function":"addMountForUser","class":"OC\\Files\\Config\\MountProviderCollection","type":"->","args":[["OC\\User\\User"],["OC\\Files\\Mount\\Manager"],["Closure"]]},{"file":"/var/www/html/lib/private/Files/SetupManager.php","line":318,"function":"OC\\Files\\{closure}","class":"OC\\Files\\SetupManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/Files/SetupManager.php","line":214,"function":"setupForUserWith","class":"OC\\Files\\SetupManager","type":"->","args":[["OC\\User\\User"],["Closure"]]},{"file":"/var/www/html/lib/private/Files/SetupManager.php","line":498,"function":"setupForUser","class":"OC\\Files\\SetupManager","type":"->","args":[["OC\\User\\User"]]},{"file":"/var/www/html/lib/private/Files/Mount/Manager.php","line":231,"function":"setupForProvider","class":"OC\\Files\\SetupManager","type":"->","args":["/NCadm/files",["OC\\Files\\Mount\\LocalHomeMountProvider"]]},{"file":"/var/www/html/lib/private/Files/Node/Root.php","line":444,"function":"getMountsByMountProvider","class":"OC\\Files\\Mount\\Manager","type":"->","args":["/NCadm/files",["OC\\Files\\Mount\\LocalHomeMountProvider"]]},{"file":"/var/www/html/lib/private/Files/Node/LazyUserFolder.php","line":82,"function":"getByIdInPath","class":"OC\\Files\\Node\\Root","type":"->","args":[5774116,"/NCadm/files"]},{"file":"/var/www/html/lib/private/Share20/Share.php","line":191,"function":"getById","class":"OC\\Files\\Node\\LazyUserFolder","type":"->","args":[5774116]},{"file":"/var/www/html/apps/recommendations/lib/Service/RecentlySharedFilesSource.php","line":116,"function":"getNode","class":"OC\\Share20\\Share","type":"->","args":[]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/apps/recommendations/lib/Service/RecentlySharedFilesSource.php","line":112,"function":"array_map","args":[["Closure"],["*** sensitive parameters replaced ***"]]},{"file":"/var/www/html/apps/recommendations/lib/Service/RecommendationService.php","line":88,"function":"getMostRecentRecommendation","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":[["OC\\User\\User"],7]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/apps/recommendations/lib/Service/RecommendationService.php","line":87,"function":"array_reduce","args":[[["OCA\\Recommendations\\Service\\RecentlyCommentedFilesSource"],["OCA\\Recommendations\\Service\\RecentlyEditedFilesSource"],"*** sensitive parameters replaced ***"],["Closure"],[]]},{"file":"/var/www/html/apps/recommendations/lib/Controller/RecommendationController.php","line":64,"function":"getRecommendations","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":[["OC\\User\\User"]]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"index","class":"OCA\\Recommendations\\Controller\\RecommendationController","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Recommendations\\Controller\\RecommendationController"],"index"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Recommendations\\Controller\\RecommendationController"],"index"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Recommendations\\Controller\\RecommendationController","index",["OC\\AppFramework\\DependencyInjection\\DIContainer"],["recommendations.recommendation.index"]]},{"file":"/var/www/html/lib/base.php","line":1068,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/recommendations/api/recommendations"]},{"file":"/var/www/html/index.php","line":38,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/lib/private/Files/Node/Folder.php","Line":162,"CustomMessage":"Exception thrown: OCP\\Files\\NotPermittedException"},"id":"658208eb700a5"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'port' => 6379,
  'instanceid' => '***',
  'passwordsalt' => '***',
  'secret' => '***',
  'trusted_domains' =>
  array (
    0 => 'nextcloud',
    1 => '***',
  'datadirectory' => '/var/www/data',
  'dbtype' => 'mysql',
  'version' => '',
  'overwrite.cli.url' => 'https://**redacted**',
  'dbname' => 'nextcloud',
  'dbhost' => '',
  'dbport' => '',
  'dbtableprefix' => '',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '***',
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_from_address' => '***',
  'mail_domain' => '***',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => '***',
  'mail_smtpport' => '587',
  'mail_smtpname' => '***',
  'mail_smtppassword' => '***',
  'ldapIgnoreNamingRules' => false,
  'default_language' => 'de',
  'loglevel' => 0,
  'defaultapp' => 'calendar,files',
  'maintenance' => false,
  'trusted_proxies' =>
  array (
    0 => '',
  'app_install_overwrite' =>
  array (
    0 => 'flowupload',
    1 => 'tasks',
    2 => 'twofactor_admin',
    3 => 'ransomware_detection',
    4 => 'keeweb',
    5 => 'files_external_onedrive',
    6 => 'occweb',
    7 => 'apporder',
    8 => 'contacts',
    9 => 'external',
    10 => 'ownbackup',
    11 => 'qownnotesapi',
    12 => 'quota_warning',
    13 => 'spreed',
    14 => 'uploaddetails',
    15 => 'caniupdate',
    16 => 'files_readmemd',
    17 => 'ocdownloader',
    18 => 'ocsms',
    19 => 'files_inotify',
    20 => 'breezedark',
    21 => 'files_clipboard',
    22 => 'w2g2',
    23 => 'geoblocker',
    24 => 'guests',
    25 => 'activitylog',
    26 => 'passman',
    27 => 'drop_account',
    28 => 'weather',
    29 => 'side_menu',
    30 => 'groupfolders',
    31 => 'files_markdown',
    32 => 'ransomware_protection',
    33 => 'announcementcenter',
    34 => 'nextbackup',
    35 => 'duplicatefinder',
  'theme' => '',
  'share_folder' => '/Mit_mir_geteilt',
  'skeletondirectory' => '',
  'memory_limit' => '2G',
  'max_execution_time' => '3600',
  'max_input_time' => '180',
  'default_phone_region' => 'DE',
  'mail_sendmailmode' => 'smtp',
  'data-fingerprint' => '88d57053ed86be48f181bcb0af4adef9',

The output of your Apache/nginx/system log in /var/log/____:

not found

The interesting part is: a user without group folders can use the instance just fine. All users with access to (any) groupfolder (not just the problematic “4”) hit the wall…

Why does NC try to create the group folder “4”? It’s a symlink to another dataset! Even if it would be a normal directory, a create must fail too!
Were additional “sanity checks” implemented from 27.1.3 to 27.1.5? “Directory does not exist - create it!” Not very sane, if you as me…

ls -l within the datadirectorys __groupfolders directory:

total 27K
drwxr-xr-x  6 www-data www-data  7  11.05.2023 22:05  ./
drwxrwx--- 34 www-data www-data 41  18.12.2023 22:12  ../
drwxr-xr-x 23 www-data www-data 75  24.10.2023 09:10  2/
drwxr-xr-x 27 www-data www-data 33  20.04.2023 17:04  3/
lrwxrwxrwx  1 www-data www-data 30  16.11.2023 17:11  4 -> /mir23/Medien/__groupfolders/4/
drwxr-xr-x  4 www-data www-data  5  01.07.2023 23:07  trash/
drwxr-xr-x  5 www-data www-data  6  11.05.2023 23:05  versions/

Again, this rather crude symlink layout worked before the upgrade.
Downgrading is no option since I didn’t bother to take a snapshot before this ‘minor’ upgrade (and just reverting the version number within the config.php doesn’t seem to be sufficient).

Any ideas?

I am no docker specialist, so I cannot help you on your core problem but are all those apps installed:

Some of them are really old and could cause really weird problems while others are fully supported and do not have to be maintained in this array.

Much luck,

Hi @ernolf,
thank you for taking your time and thanks for the hint concerning the countless apps.

Our instance is running for five years now so there may be some old stuff in it nobody cares about anymore (neither we nor the programmers). So yes, some decluttering is necessary.

But what do you mean by “do not have to be maintained in this array”?
I thought this array is maintained by the apps [plugin|part|?] (what do you call this piece of core functionality?)?
How would I be supposed to shrink this array other than deinstalling apps?

In general, is it advisable to setup a new clutter-free instance after all this time?
I did not pollute the instance on purpose (and at least it’s not Windows) but the problem I’m facing really is weird.

After all, aren’t symlinks allowed within the data folder? Nextcloud is no OS and should work with the filesystem it is presented with.

(We had a several drive crashes within the former software RAID5 so while restoring the backup we decided to move from md on ext4 to mirrored zfs storage with smaller zpools (2TB max) so backup/restore would be easier. Without symlinks we would have to setup a big zpool like the md RAID before which is harder to handle, backup-wise, and is prone to faults since one degraded mirror affects the whole zpool).

If we would lean towards setting up a new instance, the additional space (for the groupfolders) would have to be setup on “external storage”, right?

And you can cd 4 here just fine (from within your app container) and see all your files at the filesystem level? Just want to make sure it’s not something obvious before we dig deeper: like /mir23 not being mounted on your host (or, just as critically, in your app Docker container) at the moment or something.

It may not be related to your current issue (well, it’s possible since it overrides compatibility checks). It’s also not well documented, but this parameter indicates you’ve previously force enabled some apps. In your case, a bunch of apps it appears.

Force enabled apps don’t meet requirements to run in your current Nextcloud environment. It also means that going forward updates to those apps (either triggered by new app version releases… or by upgrading your NC Server instance) ignore compatibility /requirements checks. This is asking for trouble.

You’re correct that you didn’t add those entries directly, but something you’ve done historically has added those entries indirectly.

I always get a bit concerned when I see that entry pop up in a config because it means that those apps may get re-deployed in the future even if they’re incompatible with your environment. Essentially: remove all the entries unless you have incompatible apps force installed for a good reason.

EDIT: To clarify, it doesn’t mean that every app listed there is currently incompatible with your NC instance. It just means that at some point in the past they were force installed at a time maybe when they were not compatible. You can manually review compatible by grabbing each of those listed apps and looking them up in the app store either by browsing it or by going to https://apps.nextcloud.com/apps/<appname>.

EDIT2: Since groupfolders is listed there it stood out given your problem report. Though it supports v27.1.5 so likely not the culprit at the moment. Another app could in theory be causing the problem but no evidence of that currently.

1 Like

Hi Josh,
I just checked and yes, I can cd into that folder from within the app container.
While checking all symlinks I stumbled upon some owned by root, but since they are 777 that shouldn’t have been a problem in the first place.

And you’re generally on the right track asking for checking correct mounts - after modifying the instance to adapt to the new storage situation (about six weeks ago), at first I simply forgot to mount the datasets containing the symlink targets into the containers (app and nginx). Took me a while to figure that out… :wink:

1 Like

Hi @jtr again,
I’ve found the real culprit.

– Me –

While editing the split up config some days ago, I demolished the file containing 'localstorage.allowsymlinks' => true,. So the restart during the container update ignored that setting from there on, causing the problem.

I am sorry for the fuss I created. :grimacing:

But while we’re at it:
Can I simply delete the 'app_install_overwrite' => array ( ... ),?
As “promised” to @ernolf, I decluttered the apps disabling all I think aren’t currently used, leaving me with these:

Activity 2.19.0
Activity Log 0.0.1
Breeze Dark 27.0.0
Brute-force settings 2.7.0
Calendar 4.6.0
Can I update? 0.2.0
Cloud Python API 0.1.8
Contacts 5.5.0
Custom menu 3.11.2
Deleted files 1.17.0
External storage support 1.19.0
File sharing 1.19.0
Flow Upload 1.1.3
Group folders 15.3.2
Guests 2.5.1
Impersonate 1.14.0
INotify file watcher 0.1.16
Log Reader 2.12.0
Maps 1.2.0
MediaDC 0.3.7
Metadata 0.19.0
Monitoring 1.17.0
Notes 4.9.1
Notifications 2.15.0
Password policy 1.17.0
PhoneTrack 0.7.7
Polls 5.4.2
Privacy 1.11.0
QOwnNotesAPI 23.12.0
Right click 1.6.0
Share by mail 1.17.0
Tasks 0.15.0
Two-Factor TOTP Provider 9.0.0
Update notification 1.17.0
User status 1.7.0
Versions 1.20.0

which are quite basic, maintained and should not need that “force” setting.

And lastly, should I mark your first post as the solution? It lead to my discovery of my config.php issue after all…

1 Like

Well, you can safe remove the array if you want.
It only comes to effect, at the moment you want to enable a not compatible app as @jtr explained.
When an app gets an actualization, it normaly fits your version annyhow.
After a server upgrade (e.g. from 27.1 to 28) all not compatible apps will get deactivated. If you want to try if one or more of those apps run without side effects even though it is not marked as supported, then you can force again in the builtin appstore. That way, the built in app store will create a new ‘app_install_overwrite’ array with that special app(s).
After it is running, you can leave it in this array as long as it is still considered unsupported. It is also a kind of overview so that you always know exactly which apps are running out of order for you and still work.

I have written a command line app store that takes care of this task fully automatic. Unfortunately, it doesn’t (yet) work on containers like Docker.

Much luck,

1 Like

OK, deleted the array.
Hopefully I’ll remember this when the next update comes…

I tried your script nonetheless - but failed.

Could you nudge me in the right direction?

Nah, mark yours where you figured out the underlying config parameter change. :slight_smile:
Glad you discovered the solution!

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.