Mass migration of photos from VPS to Amazon S3, via Nextcloud

What is the best way to move around 30 GB of files “out” of Nextcloud and in to an Amazon S3 bucket, while keeping them all available to Nextcloud?

I’ve set up an S3 account, connected Nextcloud to a bucket there using the External Storage plugin, and verified that files added to the folder in Nextcloud appear in the bucket via the S3 console. So far, so good!

Can I just sign in to Nextcloud via the browser, move all files from one folder to the other, and leave my VPS and S3 to get on with it for however long it takes?

I’d say yes. Be prepared to watch your Apache processes spike though :slight_smile:

Second opinion? @tflidd @LukasReschke @MorrisJobke

Quite! This is why I ask, really. Is it better to let my VPS churn away for hours moving 30 GB of files and then (presumably) have my laptop’s Nextcloud client redownload it all again in to the new folder the next time I boot it up, or is it perhaps better to manually upload the lot to S3 to begin with I wonder?

If it were my install, I’d rsync the data from the /data/ folder to the remote S3 bucket. With External storage NC will scan it for changes on access so there’s no worry about database issues or such like as there would be doing this the other way around.

That’d be a little friendlier on the server, assuming S3 can accept SSH (I’d assume so).

On the client side I’m not so sure. Moving things about on the local laptop may cause conflicts.

Not sure about the webinterface if this can run into a timeout. I would rather try to copy directly via webdav. Or you just move the whole data-folder like Jason proposed.

Note that there were some reports that NC can create a large number of requests:

1 Like

That’s some alarming reading - millions of requests to S3 and huge Amazon
bills from just a few files synced? Surely that’s not right.

I’ve done several successfully migrations with rsync

I believe it’s the most reliable choice.

1 Like

What about s3cmd - How to Sync Files to Amazon S3 on Linux ?

1 Like