Migrate from local storage to S3

ℹ️ Support
, ,
1

Hello all,

I have a relatively large NC installation with a few hundred users, about two dozen of whom are very active. At this point, I want to migrate to Kubernetes / S3 for a few reasons, including potential for scaling and general reliability.

I tried to manually migrate and discovered that local storage and object storage behave quite differently. For example, /nextcloud/myuser/herfile might be represented as urn:oid:1566488.

There are several issues discussing this, but the closest to a solution I found is this repository which does the opposite of what’s needed:

Has anybody already reversed this script? Or would be willing to do it for a bounty? I suspect I could pull it off, albeit with a tremendous amount of pain.

Thank you!

tarek : )

2

Have you tried consulting documentation and the associated object storage repos on github?

Curious, what are specs on your current server(s)?

3

I find it hard to answer the first part of this question. I’ve searched around about the documentation and seen the pages on object storage such as this one. I don’t know where I’d find documents describing the details of the object storage as it pertains to nextcloud, if that’s what you mean. I also am not sure how to find “associated object storage repos”.

As for the current servers, I’m migrating from an installation on a server using MySQL / local storage on a digitalocean droplet to a Kubernetes installation using MySQL/S3. I’ve similarly converted other installations such as Mattermost, but they basically used an S3’ized version of the same directory structure for local storage, so it was much less difficult to do, as i didn’t have to do any conversion of the tables and filenames.

4

You are correct, the documentation is lacking in details because an unknown amount of it is for paid support only. I think you’ll be on your own in testing and setting it up. Of course, people here might be able to help you.

Considering all of the resources already at your disposal, it sounds like you are super capable of figuring this out. Let us know if you discover more that you cannot find already discussed on the forum or wider net.

5

Hello @tarek, I have (partially) rewritten the script you have mentioned (there is an ‘issue’ on github where I have donated my extended version.

I also have (in the same manner) built the reverse. Big point with that is that on a “live version” it would take a heap of time migrating… I created a “test option” where one can perform a “pre/test migration”… and then with “maintenance mode on” do a real one, where ONLY the changed data needs to be synced… making the downtime for a live setup a matter of seconds (when timed nicely :wink: ).

And you were right… this did take a bit of time to pull it off…

1 Like
6

Hi @Eesger : This is pretty amazing. I looked at the repository and don’t see your work. Do you mind posting a direct link to it? I’ve been looking at this problem again, as our storage needs have just increased again…

Thank you!

7

Sorry for the late response, I don’t often check thing here…

Odd, thought I posted a link somewhere, in any case, here it is:

and later on I created a mores stand alone version:

Here is a very rudimentary way of migrating:

This does work, but later on I discovered several “crappy” situations/settings that I needed to fix… I put all those “hard learned things” in a “local → S3” script based upon the “S3 → local” script posted earlier.

I haven’t published the “local → S3” script anywhere… as you said it took quite a bit of time “and pain” to get this working properly…

I couldn’t resist, so I searched the internet for your name, are you this person? https://en.wikipedia.org/wiki/Tarek_Loubani

1 Like
8

Hi @Eesger!

Thank you so much for this! If I’m understanding you correctly, you took information from the last link you posted (local → S3) and reworked it with the learnings from the previous links (S3 → local), but haven’t published this for various reasons. Even though my needs have changed, I’m still happy to pay for a bounty, since it’s obvious that others are / will be interested in the same. If you’re willing to publish it under an open license, send me a DM and we can figure out what makes sense!

tarek : )

1 Like
9

Because of your request I’ve decided to clean up the code, put in the time for a decent instruction and put it on GitHub:

I’dd like to hear if it works out as good for you as it did for me :wink:

1 Like