Moving files, files:scan problems

CZLOWIEK · July 20, 2023, 9:47am

Hi,

the problem rose when we had a system crash and had to restore a couple of corrupted directories from backup. They were in Nexcloud data directory structures.

After moving files we wanted to trigger rescan.

It cannot be done with maintenance:mode --on, because “There are no commands defined in the “files” namespace.”

When running

sudo -u www-data php occ files:scan --all

Got

Path not found: /Administrator

Path not found: /User

for every user. Tried to solve it with

sudo -u www-data php occ files:scan --all --home-only

Ok, no such errors but at one user got (maybe not related):

Exception during scan: An exception occurred while executing a query: SQLSTATE[40P01]: Deadlock detected: 7 ERROR:  deadlock detected
DETAIL:  Process 120449 waits for ShareLock on transaction 12509447; blocked by process 120436.                                                             
Process 120436 waits for ShareLock on transaction 12509444; blocked by process 120449.                                                                      
HINT:  See server log for query details.                                                                                                                    
CONTEXT:  while locking tuple (17699,101) in relation "oc_filecache"                                                                                        
#0 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1814): Doctrine\DBAL\Driver\API\PostgreSQL\ExceptionConverter->convert()
#1 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1749): Doctrine\DBAL\Connection->handleDriverException()                                    
#2 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1163): Doctrine\DBAL\Connection->convertExceptionDuringQuery()                              
#3 /var/www/nextcloud/lib/private/DB/Connection.php(295): Doctrine\DBAL\Connection->executeStatement()                                                      
#4 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(354): OC\DB\Connection->executeStatement()                                          
#5 /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\DBAL\Query\QueryBuilder->execute()                                        
#6 /var/www/nextcloud/lib/private/Files/Cache/Cache.php(386): OC\DB\QueryBuilder\QueryBuilder->execute()                                                    
#7 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(418): OC\Files\Cache\Cache->update()                                                              
#8 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(403): OC\Files\Cache\Scanner->scanChildren()                                                      
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(347): OC\Files\Cache\Scanner->scanChildren()                                                      
#10 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(269): OC\Files\Cache\Scanner->scan()                                                             
#11 /var/www/nextcloud/apps/files/lib/Command/Scan.php(164): OC\Files\Utils\Scanner->scan()                                                                 
#12 /var/www/nextcloud/apps/files/lib/Command/Scan.php(219): OCA\Files\Command\Scan->scanFiles()                                                            
#13 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()                                                 
#14 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()                                                         
#15 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()                                                          
#16 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()                                 
#17 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()                                        
#18 /var/www/nextcloud/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run()                                               
#19 /var/www/nextcloud/console.php(100): OC\Console\Application->run()                                                                                      
#20 /var/www/nextcloud/occ(11): require_once('...')                                                                                                         
#21 {main}                                                                                                                                                  
In ExceptionConverter.php line 91:
  An exception occurred while executing a query: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored u  
  ntil end of transaction block                                                                                                                            
In Exception.php line 30:
  SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block  
In Statement.php line 101:
  SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block

Tried to scan one user with:

sudo -u www-data php occ files:scan -p=/mnt/ncdata/userA userA

Ended with:

Unknown user 1 =

Eventually some, or maybe all files, were rescanned, but what is the proper way to act in such situations?

I’ve seen related [solved] 'Path not found' after migration to new Server and Files:scan --path=xxx sometimes could not find directorys

Nextcloud version (eg, 20.0.5): 27.0.0
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04.2 LTS
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.52
PHP version (eg, 7.4): PHP 8.1.2

CZLOWIEK · July 20, 2023, 10:23am

Yeah, this

was temporary. It didn’t show up after restart.

The question “What is the proper way to act in such situations?” remains.

ernolf · July 20, 2023, 11:05am

it’s either:
-p /userA/files
or:
-p userA/files
or:
--path=/userA/files
or:
--path=userA/files
but not:
-p=/userA/files
and certainly not canonical:
-p /mnt/ncdata/userA/files
or:
--path=/mnt/ncdata/userA/files

You should always read the help thoroughly:

occ files:scan --help
Description:
  rescan filesystem

Usage:
  files:scan [options] [--] [<user_id>...]

Arguments:
  user_id                  will rescan all files of the given user(s)

Options:
      --output[=OUTPUT]    Output format (plain, json or json_pretty, default is plain) [default: "plain"]
  -p, --path=PATH          limit rescan to this path, eg. --path="/alice/files/Music", the user_id is determined by the path and the user_id parameter and --all are ignored
      --generate-metadata  Generate metadata for all scanned files
      --all                will rescan all files of all known users
      --unscanned          only scan files which are marked as not fully scanned
      --shallow            do not scan folders recursively
      --home-only          only scan the home storage, ignoring any mounted external storage or share
  -h, --help               Display help for the given command. When no command is given display help for the list command
  -q, --quiet              Do not output any message
  -V, --version            Display this application version
      --ansi|--no-ansi     Force (or disable --no-ansi) ANSI output
  -n, --no-interaction     Do not ask any interactive question
      --no-warnings        Skip global warnings, show command output only
  -v|vv|vvv, --verbose     Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Much luck!

CZLOWIEK · July 20, 2023, 1:22pm

Thanks.
Results:

sudo -u www-data php occ files:scan -p=/userA/files userA

Unknown user 1 =
Files: 0 (as expected)

sudo -u www-data php occ files:scan -p /userA/files userA

Starting scan for user 1 out of 1 (userA)
Path not found: /userA/files
Files: many

sudo -u www-data php occ files:scan -p userA/files userA

Starting scan for user 1 out of 1 (userA)
Path not found: /userA/files
Files: many

sudo -u www-data php occ files:scan --path=userA/files userA

Starting scan for user 1 out of 1 (userA)
Path not found: /userA/files
Files: many

sudo -u www-data php occ files:scan --path=/userA/files userA

Starting scan for user 1 out of 1 (userA)
Path not found: /userA/files
Files: many

sudo -u www-data php occ files:scan --path=/userA/files --home-only userA

Starting scan for user 1 out of 1 (userA)
Files: many

sudo -u www-data php occ files:scan --path=/userA/files --home-only

Starting scan for user 1 out of 1 (userA)
Files: many

So, not only you were right, I also found that “–home-only” option is the only way to bypass “Path not found:” error, even though it’s correct path. Also, it’s wrong to specify user, when using --path (or -p), but it works with no error.

ernolf · July 20, 2023, 2:16pm

But you still haven’t read the help carefully:

the user_id is determined by the path and the user_id parameter and --all are ignored.

So it has to be this way:

sudo -u www-data php occ files:scan -p /userA/files

CZLOWIEK · July 21, 2023, 7:22am

That is why I added only at the bottom