Migrate data from S3 single bucket to a multi-bucket system

ℹ️ Support
1

Hello all,

i would like to migrate my data to a S3 multi bucket system, but i don’t/can’t find any docs about this.

Migrate from a single to another single bucket is easy, as data is put into that bucket(even in another server/provider) and then Nextcloud configured to access that bucket, but what about to migrate from single to multi, as those multi buckets are created on the fly by Nextcloud?

Thanks :slight_smile:

2

Nobody has done this before? :thinking:

3

A handful of people use S3 as their primary and fewer use multibucket.

I would start with comparing the database of primary vs multibucket and then figure out what changes you would have to make for the transfer. It’s going to involve editing the database no matter what.

Cheers!

4

I tried do that and it was working on my instance but I afraid to do that on my production environment.

First some basic info about my environment:

  1. docker container - Database - PostgreSQL 12.5
  2. docker container - Nextcloud 22.2.0
  3. Storage backend - S3 - not Amazon, Ceph, single bucket

Now procedure I went through:

  1. Enable Nextcloud maintenance mode
  2. Replace bucket name in Nextcoud configuration (objectstore.arguments.bucket) from {{ current name }} to {{ current name }}-0
  3. Restart Nextcloud
  4. Rename bucket in Ceph s3 storage (my Ceph administrator do that because user can not, I need to rename bucket because simple sync does not respect copying of versions, and when I copy all objects and their version then timestamps of versions where not preserved. Simple sync will also work but all versions will be lost in that case).
  5. Modify bucket name in database
    docker exec postgresql psql --username=admin --dbname=nextcloud --command "update oc_storages set id='object::store:amazon::{{ bucket_name }}-0' where id='object::store:amazon::{{ bucket_name }}'"
  6. Disable maintenance mode on Nextcloud
  7. Ensure data is still present on Nextcloud
  8. Again enable Nextcloud maintenance mode
  9. Remove objectstore from Nextcloud configuration
  10. Configure objectstore_multibucket on Nextcloud with:
    • bucket as {{ bucket_name }}-,
    • num_buckets as 4,
    • autocreate as true,
    • other arguments where the same as previous one.
  11. Disable maintenance mode on Nextcloud

This procedure works but as I noticed new entries in Database table oc_storages:

 numeric_id |                    id                    | available | last_checked 
------------+------------------------------------------+-----------+--------------
          6 | ef5ee33519d1a81ba22868a12902e595         |         1 |             
          1 | object::store:amazon::{{ bucket_name }}-1 |         1 |             
          9 | 14288ccb24faf9ce62d1e05dda7451cf         |         1 |

so configuring objectstore_multibucket means that each bucket is saved in database as new bucket and it is not recognize as object::store:amazon::{{ bucket_name }} but as some kind of hash. This raises my concerns if such state will work in future. I imagine that on some future update some update script will clear old entry in database and then all files which where assigned to old storage in database will probably disappear.