Occ files:scan --unscanned does not work

ℹ️ Support
1

Nextcloud version (15.4):
Operating system and version (eg, Debian 8):
Apache or nginx version (eg, Apache 2.4.1):
PHP version (eg, 7.2):

Im uploading lots of files manually into a nextclouds user dir. They are from several cam which feed about 100 files per minute via ftp but dont show up in Files despite executing the occ service to scan them.

Steps to replicate it:

Execute: sudo -u www-data php occ files:scan --verbose vvv --all
Sucessfully scans all files including the new ones since last upload.

Execute: sudo -u www-data php occ files:scan --verbose vvv --unscanned --all
Finds no new files :frowning: Only outputs this:

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0       | 0     | 00:00:00     |
+---------+-------+--------------+

But thats impossible, there are several 100 new files in numerous folders since the last scan.
All files have ownership the nextcloud user -www-data.
I also used --path /path/to/userdir with no effect just as with --all.
Data Dir is local not a external mount.
But without --unscanned all is found. Which i cant use because there are so many files it takes to much time as to many new ones arrive during it scan.

The Log shows no errors.

config.php

<?php
$CONFIG = array (
  'instanceid' => 'REMOVED',
  'passwordsalt' => 'REMOVED',
  'secret' => 'REMOVED',
  'trusted_domains' => 
  array (
    0 => 'REMOVED',
    1 => 'REMOVED',
    2 => 'REMOVED',
  ),
  'datadirectory' => '/var/data',
  'skeletondirectory' => '/var/data/mydir/skeleton',
  'dbtype' => 'mysql',
  'version' => '15.0.5.3',
  'overwrite.cli.url' => 'REMOVED',
  'dbname' => 'REMOVED',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'REMOVED',
  'dbpassword' => 'REMOVED',
  'installed' => true,
  'user_backends' => 
  array (
    0 => 
    array (
      'class' => 'OC_User_IMAP',
      'arguments' => 
      array (
        0 => '{REMOVED:993/imap/ssl}',
      ),
    ),
  ),
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.secret' => 'REMOVED',
);

Thanks for any help.

2

Is the result the same if you force a scan for a specific user?

3

Yes same result, unscanned finds nothing, without unscanned all files are recanned.

4

exactly the same problem here. Nextcloud 15.0.0.1
on Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-43-generic x86_64)

5

There might be a misunderstanding in which function this switch provides. I’ve found this information:

So the question is, are the files you want to scan on an external share or not?

6

what do you class as “external” I run nextcloud docker with a PATH= to a host folder, does that count as external? unscanned is not working.

7

Nope, this is not “external storage”. For further information on external storage see https://docs.nextcloud.com/server/15/admin_manual/configuration_files/external_storage_configuration_gui.html

8

ok, so --unscanned is still not working

9

Same problem here. I’m trying to scan a mounted storage with Rclone (but am I’m not using the external app). The --unscanned scans 0 files always. Did anybody solved it?

10

See this: Document `--unscanned` option of occ files:scan · Issue #2435 · owncloud-archive/documentation · GitHub

This means the “files:scan” command only rescans all entries that have a size of “-1” (could also be called “marked as dirty” or “marked as unscanned”).

11

This is somehow logic. How should the command know which files are in the database without checking for each if it’s in the database. So necessarily always every file is scanned, there is not really a different way.

And since the oc_filecache table is usually huge, it’s extra slow to query it.

What I’m not sure about is if the scan also updates database entries for files which are already inside or only adds new entries. Would likely make a significant difference as well.

Finally Nextcloud is not really made to handle manually added files. For this an inotify watcher or similar makes sense, like Syncthing makes user of it. Otherwise, use webdav to transfer the files through Nextcloud, which is the more intended way.