Reported Files amount is growing with time after upgrade from 18 to 19

Nextcloud version (eg, 18.0.2): 19.03
Operating system and version (eg, Ubuntu 20.04): 18.04
Apache or nginx version (eg, Apache 2.4.25): 2.4.46
PHP version (eg, 7.1): 7.3.22

The issue you are facing: After upgrade from NC 18.09 to 19.03 API reported files amount is continuously growing with no reason…

I suppose this is an external share problem. But till NC 19, rescan did not harm my system.

When I run rescan manually I see in console:

PHP Fatal error:  Allowed memory size of 2671771648 bytes exhausted (tried to allocate 266240 bytes) in /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 158

Is this the first time you’ve seen this error? (Y/N): Yes, but similar was seen when DATA folder was moved Files amount after moving of data directory is wrong (much bigger)

Steps to replicate it:

  1. Have an installation with external shares
  2. Upgrade from 18.09 to 19.03
  3. Monitor files amount
    in a NC UI
    grafik
find . -type f | wc -l
647348

The output of your Nextcloud log in Admin > Logging:

Nothing related to the issue

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "19.0.3.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 1.5
        },
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "7be51475e95ea13b254cd880319d98ec",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1920,
        "preview_max_y": 1080,
        "jpeg_quality": 90,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "app_install_overwrite": [
            "keeweb",
            "radio"
        ],
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

The output of your Apache/nginx/system log in /var/log/____:

Nothing related to the issue

Do not know what happens, but I do export external share configuration:

sudo -u www-data php /var/www/nextcloud/occ files_external:export > /mnt/raid/nextcloud/externals.conf

then try to do files cleanup, but it meets timeout because of locks, in a log I saw, that files on deleted shares being locked, so I restart redis, perform cleanup and import configuration again

sudo -u www-data php /var/www/nextcloud/occ files_external:import /mnt/raid/nextcloud/externals.conf

After this extenal share rescan takes around 2 min, and files amount dropped to 853498.

{"reqId":"DVZzOiGTFYmma1qEykjC","app":"/var/www/nextcloud/occ files:scan","message":"+++ Cron Filescan Completed. Execution time: 1 minutes and 39 seconds +++","level":1,"time":"2020-09-24T08:32:42+02:00"}

and before it simply goes to infinity:

{"reqId":"K5Eo7xO6JhORjbdyKPy2","app":"/var/www/nextcloud/occ files:scan","message":"+++ Cron Filescan Completed. Execution time: 933 minutes and 52 seconds +++","level":1,"time":"2020-09-24T04:38:53+02:00"}