Occ files sync on 7 million+ imported files problem

ℹ️ Support
1

Nextcloud version (eg, 20.0.5): 18.0.12
Operating system and version (eg, Ubuntu 20.04): Ubuntu 18.04.4 LTS
Apache or nginx version (eg, Apache 2.4.25): Apache2 2.4.29
PHP version (eg, 7.4): 7.2.24

The issue you are facing: Some files imported to data folder does not get synced to nextcloud when running sudo -u www-data php occ -vvv files:scan --all

I have circa 7209601 files that weighs about 13Tb in my data dir.

They’re all owned by www-data and have correct rights. About half of the files show up in nextcloud after the occ sync (which takes about 8 hours to run).

Occ finishes with no errors what so ever.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Import a large amount of files to nextcloud data dir
  2. Run sudo -u www-data php occ -vvv files:scan --all

The output of your Nextcloud log in Admin > Logging:

My log is filled with these lines: 
[webdav] Fatal: OCA\DAV\Connector\Sabre\Exception\Forbidden: No read permissions at <<closure>>

 0. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 195
    OCA\DAV\Connector\Sabre\Directory->getChildren()
 1. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 899
    Sabre\DAV\Tree->getChildren("files/pom/Inför LR 15.04.2021 NEK")
 2. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 978
    Sabre\DAV\Server->generatePathNodes(Sabre\DAV\PropFind {}, [Sabre\DAV\PropF ... }])
 3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1666
    Sabre\DAV\Server->getPropertiesIteratorForPath("files/pom/Inför LR 15.04.2021 NEK", ["{DAV:}resource ... "], 1)
 4. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 355
    Sabre\DAV\Server->generateMultiStatus(Generator {}, false)
 5. <<closure>>
    Sabre\DAV\CorePlugin->httpPropFind(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
 6. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    call_user_func_array([Sabre\DAV\CorePlugin {},"httpPropFind"], [Sabre\HTTP\Requ ... }])
 7. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:PROPFIND", [Sabre\HTTP\Requ ... }])
 8. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
 9. /var/www/nextcloud/apps/dav/lib/Server.php line 319
    Sabre\DAV\Server->exec()
10. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
11. /var/www/nextcloud/remote.php line 165
    require_once("/var/www/nextcl ... p")

PROPFIND /remote.php/dav/files/pom/Inf%C3%B6r%20LR%2015.04.2021%20NEK
from 10.3.7.227 by pom at 2021-04-14T06:31:44+00:00

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '*',
  'passwordsalt' => '*',
  'secret' => '*',
  'trusted_domains' => 
  array (
    0 => '*',
    1 => '*',
  ),
  'datadirectory' => '*',
  'overwrite.cli.url' => '*',
  'htaccess.RewriteBase' => '/',
  'dbtype' => 'mysql',
  'version' => '18.0.12.2',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '*',
  'dbpassword' => '*',
  'logtimezone' => 'UTC',
  'installed' => true,
  'ldapIgnoreNamingRules' => false,
  'ldapProviderFactory' => '\\OCA\\User_LDAP\\LDAPProviderFactory',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'mail_from_address' => '*',
  'mail_smtpmode' => 'smtp',
  'mail_domain' => '*',
  'mail_smtphost' => '*',
  'mail_smtpport' => '*',
  'loglevel' => '1',
  'ldapUserCleanupInterval' => '5',
  'maintenance' => false,
  'theme' => '',
  'remember_login_cookie_lifetime' => 3600,
  'session_lifetime' => 3600,
  'app_install_overwrite' => 
  array (
    0 => 'deck',
    1 => 'impersonate',
    2 => 'files_downloadactivity',
  ),
);

The output of your Apache/nginx/system log in /var/log/apache2/error.log:

[Wed Apr 14 08:27:33.529476 2021] [access_compat:error] [pid 13788] [client 81.226.145.19:61637] AH01797: client denied by server configuration: /var/www/nextcloud/config
[Wed Apr 14 08:31:43.146019 2021] [access_compat:error] [pid 30878] [client 10.3.7.227:55583] AH01797: client denied by server configuration: /var/www/nextcloud/config
[Wed Apr 14 08:36:59.154412 2021] [access_compat:error] [pid 10058] [client 130.243.101.84:62659] AH01797: client denied by server configuration: /var/www/nextcloud/config
[Wed Apr 14 08:42:14.612384 2021] [access_compat:error] [pid 18902] [client 77.218.41.180:49730] AH01797: client denied by server configuration: /var/www/nextcloud/config
2

Make sure that all your files can be read by the www-data user under which Nextcloud is running. The easiest way to double-check this, is by switching to that user, navigating through the folder structure and test if a folder or file can be accessed.

3

Thanks but they’re all owned by www-data unfortunately…

4

“No read permissions” is usually a clear statement which no room for discussions. Have you tried to restrict a scan to a specific path instead of using the “–all” switch to find out if the problem is related to whitespaces or special characters in the file names, like e.g.

sudo -u www-data php occ -vvv files:scan --path "/<user>/files/pom"
5

Yes I’ve tried scanning the user in questions files and also a specific folder that exists but not showing up in the cloud interface

6

… and with which result? Can you please be AS PRECISE AS POSSIBLE what has exactely happened, about what folder you’re speaking which doesn’t show-up in the web interface and which related errors are shown in the nextcloud log file.

7

Yeah sorry this is what I did:

sudo -u www-data php occ -vvv files:scan --path username/folder_2016/

Scans a bit then completes with no errors in log or terminal:

±--------±------±-------------+
| Folders | Files | Elapsed time |
±--------±------±-------------+
| 10 | 28337 | 00:02:37 |
±--------±------±-------------+

Allthough the folder is not visible from the users cloud interface

Thanks

8

I think you should double-check and fix your directory/file structure first. The user specific directory structure is ususally stored in “<NC-data-directory>/<users>/files/...” and not “<NC-data-directory>/<users>/...”. As soon as you move the directory to the right location, it should be visible in the gui. It might be possible that this will fix the other problem too.

9

thanks a million years later :pleading_face: