Occ files:scan outputs "Path not found:" for some users

ℹ️ Support
, ,
1

Nextcloud version (eg, 20.0.5): 28.0.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04
Apache or nginx version (eg, Apache 2.4.25): 2.4.52
PHP version (eg, 7.4): 8.1

The issue you are facing:

When I run occ files:scan --all the scan process works for most of the users, but for some the commend outputs the following error as scan result:

Path not found: /username

When I do this for the single user with occ files:scan username I get the following result:

Starting scan for user 1 out of 1 (username)
Path not found: /username
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 1320    | 28103 | 0   | 0       | 0       | 1      | 00:00:07     |
+---------+-------+-----+---------+---------+--------+--------------+

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

It just happens, I don’t know how to replicate it.

The output of your Nextcloud log in Admin > Logging:

(there is no log for the scan)

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

(there is no relevant config which affects the scan as it only happens for 8 of about 50 users)

The output of your Apache/nginx/system log in /var/log/____:

(this log is way too big! we talk about an instance for more than 50 users!)

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

(there are no errors in the log)

2

Every config is relevant!

I wrote a function for you to check, wether all directories exist. You may have to adapt the variable NC_DIR to your needs because you did not provide that information.
It needs jq: (apt-get install jq)
Then copy this function and paste it inside of your terminal:

check_userdirs(){
    declare NC_DIR="/var/www/nextcloud"
    declare -A colors=([R]='\033[1;31m' [G]='\033[1;32m' [Y]='\033[0;33m' [B]='\033[0;34m' [NC]='\033[0m')
    for color in "${!colors[@]}"; do eval "$color() { echo -ne '${colors[$color]}'; }";done
    occ_call="sudo -u www-data php -f $NC_DIR/occ"
    for user in $(echo $($occ_call user:list --output=json | jq -r 'keys_unsorted[] | @base64')); do
        user="$(echo "$user" | base64 --decode)"
        echo "processing User `Y`$user`N`"
        userdir="$($occ_call user:info --output json -- "$user"  | jq -r '.user_directory')"
        if [ -d "$userdir" ]; then
            echo "Userdirectory \"`B`$userdir`N`\" `G`exists and is a directory`N`"
        elif [ -z "$userdir" ]; then
            echo "Userdirectory `R`is not defined`N`"
        else
            echo "Userdirectory \"`B`$userdir`N`\" `R`does not exist or is not a directory`N`"
        fi
        echo "----------------"
    done
}

… then simply run it with

check_userdirs

Much luck,
ernolf

3

Well - I this case don’t think so - but here you are:

<?php
$CONFIG = array (
  'instanceid' => '*** REMOVED FOR SECURITY REASONS ***',
  'passwordsalt' => '*** REMOVED FOR SECURITY REASONS ***',
  'secret' => '*** REMOVED FOR SECURITY REASONS ***',
  'trusted_domains' => 
  array (
    0 => 'mydomain.example',
  ),
  'datadirectory' => '/var/www/mydomain.example/data',
  'skeletondirectory' => '',
  'overwrite.cli.url' => 'https://mydomain.example',
  'dbtype' => 'mysql',
  'version' => '28.0.1.1',
  'installed' => true,
  'htaccess.RewriteBase' => '/',
  'maintenance' => false,
  'dbname' => '*** REMOVED FOR SECURITY REASONS ***',
  'dbhost' => 'localhost',
  'dbuser' => '*** REMOVED FOR SECURITY REASONS ***',
  'dbpassword' => '*** REMOVED FOR SECURITY REASONS ***',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/run/redis/redis-server.sock',
    'port' => 0,
    'dbindex' => 0,
  ),
  'knowledgebaseenabled' => false,
  'theme' => '',
  'simpleSignUpLink.shown' => false,
  'loglevel' => 2,
  'log_authfailip' => true,
  'mysql.utf8mb4' => true,
  'mail_from_address' => '*** REMOVED FOR SECURITY REASONS ***',
  'mail_domain' => '*** REMOVED FOR SECURITY REASONS ***',
  'mail_smtpmode' => 'sendmail',
  'mail_smtphost' => 'mymailserver',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpname' => '*** REMOVED FOR SECURITY REASONS ***',
  'mail_smtppassword' => '*** REMOVED FOR SECURITY REASONS ***',
  'updater.release.channel' => 'stable',
  'trashbin_retention_obligation' => 'auto, 30',
  'versions_retention_obligation' => 'auto, 180',
  'app_install_overwrite' => 
  array (
    0 => 'apporder',
    1 => 'deck',
    2 => 'keeweb',
    3 => 'appointments',
    4 => 'drawio',
    5 => 'bookmarks',
    6 => 'spreed',
    7 => 'unroundedcorners',
    8 => 'fulltextsearch',
    9 => 'fulltextsearch_elasticsearch',
    10 => 'files_fulltextsearch',
    11 => 'files_fulltextsearch_tesseract',
    12 => 'passwords',
    13 => 'gpxpod',
    14 => 'dicomviewer',
    15 => 'files_bpm',
  ),
  'has_rebuilt_cache' => true,
  'default_language' => 'de',
  'default_phone_region' => 'DE',
  'check_data_directory_permissions' => false,
  'log_rotate_size' => 52428800,
  'profile.enabled' => false,
  'preview_max_x' => 1920,
  'preview_max_y' => 1920,
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '465',
  'trusted_proxies' => 
  array (
    0 => '*** REMOVED FOR SECURITY REASONS ***',
    1 => '*** REMOVED FOR SECURITY REASONS ***',
    2 => '127.0.0.1',
    3 => '::1',
  ),
  'auth.bruteforce.protection.enabled' => true,
  'ratelimit.protection.enabled' => true,
  'reference_opengraph' => false
);
4

Thanks for the help - but this is not the problem.

Every user directory does exist and as expected the function does not show any problem either :frowning:. For all users the script reports:

----------------
processing User username
Userdirectory "/var/www/mydomain.example/data/username" exists and is a directory

Where username is of course the name of each user which was checked.

I will put a backup of the server to my development machine and check what is going on XDebug and will post an update here if I find something. It will just take a while, since we talk about more than 400 GB of data.

Last time when this happened in Nextcloud 27 it only affected one single user and I could fix this by exporting the user with the migration app, deleting the account and then creating and importing the user again. After that, the problem did not show up - until I updated to Nextcloud 28 and it came back. But now it affects more users.

5

Good. Then we have at least ruled that out.

Strange! :thinking:
I don’t know right now what could be going on :cry:

Much luck,
ernolf

6

The strange thing is, that the file scan itself seems in fact to work. If a new file is copied to the user data folder via scp the file scan will happily add it to the oc_filecache table and also report that update. However it still will complain about “Path not found”.

7

Try

occ user:sync-account-data

I don’t know if that helps, but it can’t break anything either

Much luck,
ernolf

8

Thanks for the hint - did not change anything.

9

I now had a chance to check the code in XDebug.

The issue is using shared Collectives.

The users on that instance started using Collectives recently and around the same time that “Path not found” issue started showing up.

At some point the file scanner (apps/files/lib/Command/Scan.php) will try to find the physical folder for a shared Collective which is seen als “folder”. However there is no real folder with that name in the users files folder. That folder only exists as “virtual” folder - it is visible in the web UI and you can use Collectives to edit files, but there is no physical folder with that name.

So what happens is this:

  1. There is a collective named Foobar shared with a circle where the user is a member, however the user does not have a physical folder within /var/www/nextcloud/data/username/files/Collectives/Foobar since it is only a shared folder which will only exist in its original source.
  2. The scanner will see Collectives/Foobar in the file list of the user to be scanned.
  3. The method \OC\Files\Utils\Scanner::scan() will then fail, because that folder must not be scanned at all since it does not exist.

The message “Path not found” is confusing, because in fact it means “there was at least one file which I could not find, but I don’t even know how it was named” since at the point, where the exception is thrown, the original file name is already lost.

I have to do a bit more research here - because regular shared files or folders work fine and I wonder, why only Collectives cause a problem here. I need to describe a way how to reproduce that and then I will create a GitHub issue for that as well.

3 Likes
10

Can you check if you hit a mount point of type CollectiveMountPoint or CollectiveStorage?
See
collectives/lib/Mount/CollectiveStorage.php at main · nextcloud/collectives · GitHub and
collectives/lib/Mount/CollectiveMountPoint.php at main · nextcloud/collectives · GitHub

SharedStorage is skipped at server/lib/private/Files/Utils/Scanner.php at 5125c34f011651821d3a1d6e3ce2d0337ab75fe9 · nextcloud/server · GitHub

So might be that this needs to be skipped as well (leaving aside a better solution)

3 Likes
11

Thanks for the hint. Yes CollectiveMountPoint is hit in \OCA\Collectives\Mount\MountProvider::getMountsForUser(). I will see, if I can adapt the SharedStorage method as a workaround.

1 Like
12

Yes - that did the trick:

			// don't scan received local shares, these can be scanned when scanning the owner's storage
			if ($storage->instanceOfStorage(SharedStorage::class)
                || $storage->instanceOfStorage(CollectiveStorage::class)
            ) {
				continue;
			}

I will create a GitHub issue and PR for that as soon as possible. Thanks for your help.

The GitHub issue including a Pull Request for the workaround can be found here: [Bug]: Shared Collectives cause "Path not found" when scannig files with "occ files:scan --all" · Issue #42496 · nextcloud/server · GitHub

2 Likes
13

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.