Operating system and version (e.g., Ubuntu 24.04):
Ubuntu 22.04
Web server and version (e.g, Apache 2.4.25):
Apache 2.4
Reverse proxy and version _(e.g. nginx 1.27.2)
Traefik 2.11
PHP version (e.g, 8.3):
8.3
Is this the first time you’ve seen this error? (Yes / No):
N/A
When did this problem seem to first start?
N/A
Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
Docker image from crazy-max
Are you using CloudfIare, mod_security, or similar? (Yes / No)
No
Summary of the issue you are facing:
I want to move my primary S3 storage from provider A to provider B. Since they are different providers, there is no way to use the same endpoint url or bucket name. Nextcloud, as far as I can tell, takes the new config, and acts like I just added another storage location, not that I moved data. I see this in a bunch of the tables where files are listed with the id number of the old storage instead of Nextcloud just pointing at the new storage.
I think I once saw somewhere that changing providers like this is not supported, but I haven’t found that info again.
Before I go post a feature request on GitHub, I just want to confirm that what I’m doing is not supported. Is that true?
Thanks!
Steps to replicate it (hint: details matter!):
Configure Nextcloud to use S3 from any S3 provider as the primary storage location.
Specifically I switched from a self-hosted Minio instance to a Wasabi bucket.
Copy the data from the initial storage provider to a new storage provider.
Reconfigure Nextcloud to point at the new storage provider.
Discover that some files are not accessible anymore.
If the id for the storage of the file in oc_storages starts with object::user, the files seem to be available.
If the id starts with object::store:amazon the files are not available.
Specifically things like Team Folders and preview images.
I don’t have a solution for this, but thank you for letting me know. I’m using Wasabi, and part of my backup strategy assumes that I could simply switch to a cloned bucket. I hope there’s a solution, otherwise I’m going to have to rework things.
Yes, I know, test your backups, but this particular part seemed like such a minor thing that would obviously work, and my time is limited.
My setup was small enough that rebuilding it from scratch wasn’t hard.
One of the options I briefly looked at was finding all the places in the database using the old storage as a foreign key and updating that to the new storage. But I’d need to really dig into the db schema a lot farther than I want in order to make sure I didn’t miss anything.
Overall, I think that we need the maintainers to implement a migration tool, or make it so that primary storage has the same id no matter what you configure for the backend.
Oh, and if you haven’t noticed already, when using object storage for the primary, your files and folders are not stored in S3 using paths. It’s a flat structure with files named “urn:000001” (or something like that…). That means you can’t just download the files to a normal filesystem and use that as your data folder.
I haven’t ever developed anything for object storage, so I do wonder about the reasoning behind how everything is architect-ed. I’m sure there are good reasons, but I can’t say it makes sense at first look.
Thanks, yeah I am aware of this. I do keep three backups, and two of them are encrypted copies of the actual files, synced with restic. However, the bucket backup was supposed to be the easy restore option.
I haven’t ever developed anything for object storage, so I do wonder about the reasoning behind how everything is architect-ed.
My understanding is that the first reason is that you can’t rename things in object storage. So a pic.jpg renamed to pic.old.jpg would mean you have to copy and then delete the original. Slow.
Second, it’s because object storage doesn’t actually have folders. It can simulate folders using the file name (for example, a / could be interpreted as the file being in a folder), but this means if you have a folder with 1000 items in it, then you rename the folder, in an S3 bucket this is 1000 copy operations, then 1000 delete operations. If any of them fail, you’re left with a big mess.
So Nextcloud completely separates the file content (object storage) and metadata (database). Then renaming a folder is a single database operation, it either fails or succeeds. On a normal filesystem, renaming a folder is also a single operation.
So basically, error prevention and performance are at least two good reasons.
To be honest, I didn’t expect using object storage would work as well as it does. But I’ve been using it for about 3 months now, and doing a lot of work that involves small files like photo organization. Everything is pretty quick.