How can I move lots of data (100s of GB) to external storage, reliably?

Nextcloud version: 22.2
Operating system and version: not sure, it’s shared hosting
Apache or nginx version: same
PHP version: 7.4.23

I’ve been using Nextcloud for years, mainly for sharing and archiving photos to share with my family. Unfortunately this means I have absolutely obliterated my storage quota and need to sort that out. It’s a good problem to have, but, I’m having some difficulties…

I’ve recently added a Backblaze B2 instance as external storage. It was easy enough to set up via the admin settings. I was hoping to move a whole bunch of old photos to it from “regular” Nextcloud storage, about 100-200GB worth.

In the web UI I select a directory containing about 16GB worth of files (ie. a few thousand). I select “Move or copy” from the actions menu, and then select a folder under my new Backblaze storage. I see a spinner over the folder and a progress bar appear at the top (or half of one, anyway).

And that’s it for an hour or so. It goes and goes, and eventually I get an error saying the files could not be moved. There doesn’t seem to be a consistent amount of time or number of bytes it stops at. If I look in the Backblaze storage, I can see that some of the files were moved, but not all.

Nothing I can find in the docs tells me what state my files might be in: partly moved? Duplicated but not yet deleted from the source? Can I delete the destination files without losing data?

I try to move the same origin folder to the same destination again, but this time Nextcloud refuses because it already exists. What I really want is for it to merge them, but that doesn’t seem to be an option. There are way to many subfolders and files to merge them manually.

Inspecting it, it seems like Nextcloud copies the files over, and doesn’t delete them from the origin until the entire lot are copied. So I delete the destination folder, because I’m a renegade who likes to live dangerously. This works, but again, takes almost an hour.

My questions are:

  • The main questions: is there a way, a tool, an app, anything that lets me move large folders to external storage? Preferably without me having to babysit it for hours? I don’t need to do it on a schedule or anything, it’s just a one-off move.

  • How does Nextcloud do a move, does it copy the files and then only delete the origin files when the whole thing is done? Does it delete as it goes?

  • When I start a long operations like a move or delete in the web UI or Android app, does it continue after I exit the browser/app? Log out?

As always, thanks to the Nextcloud team and community for their work!


Other requested info:

The output of your Nextcloud log in Admin > Logging: I’m trying to get something useful out of this, unfortunately the files_sharing app spams it with deprecation notices several times a second so it’s quite a challenge.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!) (is this meant to be /path/to/nextcloud/config?):

# Removed a lot of secret stuff, here's what's left.
<?php
$CONFIG = array (
  'dbtype' => 'mysql',
  'version' => '22.2.0.2',
  'default_phone_region' => 'AU',
  'installed' => true,
  'theme' => '',
  'loglevel' => 0,
  'maintenance' => false,
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '127.0.0.1',
    'port' => 6379,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'trashbin_retention_obligation' => 'auto,60',
  'activity_expire_days' => 90,
  'mysql.utf8mb4' => true,
  'app_install_overwrite' => 
  array (
    0 => 'privatedata',
  ),
  'enable_previews' => false,
);

I have filed an issue for this. Meanwhile, if anyone can think of a workaround, please let me know.

I have discovered rclone which can do remote-to-remote copy via WebDAV. It’s a brilliant tool for this.

1 Like