Migrate from local storage to S3

Hello all,

I have a relatively large NC installation with a few hundred users, about two dozen of whom are very active. At this point, I want to migrate to Kubernetes / S3 for a few reasons, including potential for scaling and general reliability.

I tried to manually migrate and discovered that local storage and object storage behave quite differently. For example, /nextcloud/myuser/herfile might be represented as urn:oid:1566488.

There are several issues discussing this, but the closest to a solution I found is this repository which does the opposite of what’s needed:

Has anybody already reversed this script? Or would be willing to do it for a bounty? I suspect I could pull it off, albeit with a tremendous amount of pain.

Thank you!

tarek : )

Have you tried consulting documentation and the associated object storage repos on github?

Curious, what are specs on your current server(s)?

I find it hard to answer the first part of this question. I’ve searched around about the documentation and seen the pages on object storage such as this one. I don’t know where I’d find documents describing the details of the object storage as it pertains to nextcloud, if that’s what you mean. I also am not sure how to find “associated object storage repos”.

As for the current servers, I’m migrating from an installation on a server using MySQL / local storage on a digitalocean droplet to a Kubernetes installation using MySQL/S3. I’ve similarly converted other installations such as Mattermost, but they basically used an S3’ized version of the same directory structure for local storage, so it was much less difficult to do, as i didn’t have to do any conversion of the tables and filenames.

You are correct, the documentation is lacking in details because an unknown amount of it is for paid support only. I think you’ll be on your own in testing and setting it up. Of course, people here might be able to help you.

Considering all of the resources already at your disposal, it sounds like you are super capable of figuring this out. Let us know if you discover more that you cannot find already discussed on the forum or wider net.

Hello @tarek, I have (partially) rewritten the script you have mentioned (there is an ‘issue’ on github where I have donated my extended version.

I also have (in the same manner) built the reverse. Big point with that is that on a “live version” it would take a heap of time migrating… I created a “test option” where one can perform a “pre/test migration”… and then with “maintenance mode on” do a real one, where ONLY the changed data needs to be synced… making the downtime for a live setup a matter of seconds (when timed nicely :wink: ).

And you were right… this did take a bit of time to pull it off…

1 Like

Hi @Eesger : This is pretty amazing. I looked at the repository and don’t see your work. Do you mind posting a direct link to it? I’ve been looking at this problem again, as our storage needs have just increased again…

Thank you!