Moving from S3 bucket mounted as linux mountpoint to S3 as primary storage

[details=“Support intro”]

Nextcloud version (eg, 20.0.5): 27.0.2
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04.3 LTS
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.57 (Debian)
PHP version (eg, 7.4): 8.2.9

More info:
Running as docker container behind Traefikv2

The issue you are facing:
I am currently mounting a Wasabi S3 bucket as a Linux mountpoint using rclone. I have recently tried to move tens of GB of data, and Nextcloud will either return back to the state before the moves, or my Windows client and the Nextcloud server web interface are different.

If I run files:scan, it is worse, because it returns it back to an even older version. So, I’m not quite sure which log to paste as the logs contain filenames and therefore, respectfully, I hope you understand why I haven’t pasted it here.

I’m wondering if I make the S3 bucket the Primary Storage, whether I will avoid these types of issues.

If so, how would I do this? I’ve searched the forum here and Google’d to no avail.

Config.php:

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'trusted_proxies' =>
  array (
    0 => 'traefik',
  ),
  'instanceid' => 'ocxxxxxxxxxxx',
  'passwordsalt' => 'JKDFJOKSLKDFJKDFJKLJDFJKSLDFJSKLDFSJKL',
  'secret' => 'asdfjhJKLDKJSLDKGJLKJlasdkfjlaksdjfasdflakdsjf',
  'trusted_domains' =>
  array (
    0 => 'nc.domain.com',
  ),
  'datadirectory' => '/var/www/html/data',
  'default_phone_region' => 'FR',
  'dbtype' => 'mysql',
  'version' => '27.0.2.1',
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'https://nc.domain.com',
  'cache_chunk_gc_ttl' => 259200,
  'check_for_working_htaccess' => true,
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '3306',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nc_admin',
  'dbpassword' => 'adlksfjalskdjflakdjflkadjf',
  'tempdirectory' => '/var/www/html/tmp',
  'installed' => true,
  'theme' => '',
  'log_type' => 'file',
  'logfile' => 'nextcloud.log',
  'loglevel' => 3,
  'logdateformat' => 'F d, Y H:i:s',
  'maintenance' => false,
  'versions_retention_obligation' => 'auto, 1095',
  'trashbin_retention_obligation' => 'auto, 1095',
  'app_install_overwrite' =>
  array (
    0 => 'sharepermissions',
    1 => 'customproperties',
    2 => 'groupfolders',
    3 => 'loganalyzer',
    4 => 'activitylog',
  ),
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'domain.com',
  'mail_smtphost' => 'smtp.domain.net',
  'mail_smtpport' => '465',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'user@domain.com',
  'mail_smtppassword' => 'aksdfncalskdfjacn;lsdkjn;aljkscn;lafkdj',
  'twofactor_enforced' => 'false',
  'twofactor_enforced_groups' =>
  array (
    0 => 'admin',
  ),
  'filelocking.enabled' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'port' => 6379,
  ),
);

/var/log/apache2/access.log is pointed to /dev/stdout
/var/log/apache2/error.log is pointed to /dev/stderr

Thank you all!

I’m sorry it’s hard to understand the issue you have. Especially this statement makes me :thinking:

Maybe you can elaborate more about what you do - and what results do you see e.g. files stored on s3 is not visible in web UI. what happens to new added files? so they exist in the storage? if not can you track when/why/how it was deleted? I would focus on web UI first as it reflects the server view and move to the desktop client later (and definitely wait some time to allow the client to detect and sync added files, especially if there are many).

Maybe it’s good idea to move only few files first - review if this files are accessible on the Nextcloud host and appear in the system after running occ files:scan. Once you found a stable way to inject the files you can scale (maybe there are some delays when you add many files?). and maybe it’s a good idea to disconnect/stop the client and put the server in maintenance mode when you are going to scan all the new files to prevent issues when client operates on outdated data…

Thanks for the reply. I actually tried the UI first, but it gave me errors, so I tried it on the Windows desktop client, and it moved the files, so I just let Nextcloud catch up. But overnight, the moved files would return to its previous folders.

I did try runing occ files:scan. That was worse. It immediately moved the files back to the previous folders. Right now, my Windows files seems like it is staying put but for one folder, where (again, I’m moving GBs of files). Perhaps the Nextcloud server just needs to catch up, but I’m not sure where to find what the outstanding operations are for Nextcloud server.

I’m reluctant so far to just change my primary storage to S3 (actually Wasabi). The documentation is clear that I would not see the files.

The thing I would try is:

  1. Delete the folder sync
  2. Move the files to the way I want it on my Windows client
  3. Create an external storage using S3/Wasabi
  4. Upload (or sync) ALL my files again (now 100s of GBs) to the new external S3 storage – here I’m not sure if I have to upload it using something like rclone/rsync or whether after establishing a new Nextcloud external storage sync connection that my files on my Windows file system will simplly be uploaded to the new external storage
  5. Then (and here I have absolutely no idea, and I can’t find anything in these forums, nor the documentation), is there a way to “promote” the S3 external storage to primary storage and have my user (single user - me!) now use the external storage as the primary.

As you can see, a lot of variables, a lot of unknowns, and while I have to keep moving my files around, at least they’re still there, just not where I want them … at least not yet.

That depends.

How are you currently integrating this storage into Nextcloud?

Is your current S3 bucket, that is mounted via rclone, mounted as /var/www/html/data or are you including it in Nextcloud as External Storage as a Local type mount?

I will say that the approach you’re taking, utilizing an intermediate underlying mounting mechanism, is not ideal unless you really have a good reason for doing it that way. There are numerous caveats to mounting an S3 object store as a native filesystem.

1 Like