Manually moving around files in data directory

Nextcloud version: nextcloud:fmt
Operating system and version: docker
Apache or nginx version: nginx:alpine
PHP version: 8.2.16

The issue you are facing:
I need to move very large directorys and doing it over internet with my hosted nextcloud system is not very fast. It would take like weeks and weeks.
So I thought; I connect the drive directly to my server and just move the files to the user files data directory.
It works great after rescanning the files with the occ command. Only problem is that most of the metadata seems gone, like when the file is edited or created. That’s just set to now. Is there any way to fix that?

Is this the first time you’ve seen this error?: No

Steps to replicate it:

  1. Move files to data/user/files/… folder and rescan files with occ
  2. metadata removed:(

The output of your config.php:
its a big file probably not very relevant. but if you see anything I can improve please let me know

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'password' => 'redis',
    'port' => 6379,
  ),
  'htaccess.RewriteBase' => '/',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'redacted',
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'maintenance_window_start' => 2,
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'redacted',
    2 => 'redacted',
    3 => '127.0.0.1',
    4 => '172.17.0.0/16',
  ),
  'trusted_proxies' =>
  array (
    0 => 'redacted',
    1 => 'redacted',
    2 => 'redacted',
    3 => '127.0.0.1',
    4 => '172.17.0.0/16',,
  ),
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\Imaginary',
    1 => 'OC\\Preview\\Image',
    2 => 'OC\\Preview\\Movie',
    3 => 'OC\\Preview\\TXT',
    4 => 'OC\\Preview\\MP3',
    5 => 'OC\\Preview\\MKV',
    6 => 'OC\\Preview\\MP4',
    7 => 'OC\\Preview\\AVI',
    8 => 'OC\\Preview\\MarkDown',
    9 => 'OC\\Preview\\OpenDocument',
    10 => 'OC\\Preview\\Krita',
  ),
  'preview_imaginary_url' => 'http://imaginary:3047',
  'datadirectory' => '/var/www/html/data',
  'check_data_directory_permissions' => false,
  'dbtype' => 'mysql',
  'version' => '28.0.2.5',
  'overwrite.cli.url' => 'https://redacted',
  'overwritehost' => 'redacted',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'nextcloud',
  'installed' => true,
  'app_install_overwrite' =>
  array (
    0 => 'occweb',
    1 => 'phonetrack',
    2 => 'quota_warning',
    3 => 'impersonate',
    4 => 'ojsxc',
    5 => 'ehr',
    6 => 'news',
    7 => 'facerecognition',
  ),
  'mail_from_address' => 'noreply',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'redacted',
  'mail_smtphost' => 'redacted',
  'mail_smtpport' => '587',
  'maintenance' => false,
  'log_type' => 'file',
  'logfile' => 'nextcloud.log',
  'loglevel' => 3,
  'logdateformat' => 'F d, Y H:i:s',
  'default_phone_region' => '1974',
  'theme' => '',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'redacted',
  'mail_smtppassword' => 'redacted',
  'encryption_skip_signature_check' => true,
  'csrf.optout' =>
  array (
    0 => '/^WebDAVFS/',
    1 => '/^Microsoft-WebDAV-MiniRedir/',
    2 => '/^\\bElectron\\b\\/',
    3 => '/YoMama 1.2/',
    4 => '/.*/',
  ),
  'defaultapp' => 'dashboard,files',
  'data-fingerprint' => 'redacted',
);

What was the EXACT command you used to move-/ copy the files?

If you used cp, did you set the flag

  • --preserve=all

or one that includes it, like -a or --archive (which is an alias for -dR --preserve=all)

An easy way to copy files while preserving ALL metadata from filesystem level (like a-, m- and ctime), is:

cp -a $source_dir/. $target_dir

That will recursively copy the content (including hidden files) of “$source_dir” inside of “$target_dir” while preserving ALL metadata like mode, ownership, timestamps, context, (hard-)links and extended attributes (as far as supported by target filesystem and possible with the rights of the invoking user)

Finaly you should run occ files:scan with the '--generate-metadata' option:

occ files:scan --generate-metadata -- <user_id>

I have done this multiple times. It always worked.
If you get problems with the files in the first directory, then copy the folder containing the files into the target folder:

cp -at $target_dir/ $source_dir

EDIT: You should/can of course

  • chown -R www-data.www-data $target_dir

… after copying the files. That will not change other meta-data.


Much and good luck,
ernolf