Is this possible? Migrate existing data storage to S3

lesaff · December 16, 2017, 2:51pm

Hi, I currently have a Nextcloud 12.0.4 using regular filesystem as my primary storage. I would like to migrate all files on existing primary storage to AWS S3.

Is there a way to do this on an existing installation?

Thanks
Rudy

Dan_Smith · January 7, 2018, 3:47am

no, been there and tried that. you can only use AWS S3 by:

install nextcloud
go to setup page but don’t enter anything (even admin user email)
edit config/config.php to put the S3 details in

 array (
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => '<BUCKET NAME>',
      'autocreate' => true,
      'key' => '<KEY>',
      'region' => '<REGION - IF REQUIRED>',
      'secret' => '<SECRET>',
      'use_ssl' => true,
      'use_path_style' => false,
    ),

now refresh the setup page and go from there.

ibr · January 16, 2018, 8:01am

Hi,

go to setup page but don’t enter anything (even admin user email)

you mean by that not even access the NextCloud server from the browser, or I can access the server from the browser but not to do anything there? because I want to try your solution on a different server and I don’t want to mess up anything and re install, and I shouldn’t also create admin user as well?

another question, this part I intend to add to the config.php, is it OK?

'objectstore' =>
  array(
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => array(
            'bucket' => 'nextcloud',
	    'autocreate' => true,
	    'key'    => 'XXXXXX',
	    'secret' => 'xxxxxxxxxxxxxxxxx',
	    'hostname' => 's3-us-east-1.amazonaws.com',
	    'use_ssl' => false,
	    'region' => 'us-east-1',
	  ),
  ),

keep in mind I made the argument 'use_ssl' to false since I’m don’t have certificate and I’m using http for testing, is that OK too?

Thanks

Dan_Smith · January 16, 2018, 10:03am

Think your config looks ok.

When you install nextcloud it installs the files to the correct path but the first time you launch the setup screen it creates the initial config file with the instance id. That’s the point when you need to edit it to enter the bucket details.

If you enter any information on the setup screen, even admin user id, I’ve found it goes a little further in the setup process and then it won’t work with S3. So in essence, open a browser window to the setup screen and leave it there until you’ve entered the bucket details and then go back to it to continue.

ibr · January 16, 2018, 10:13am

thanks,

one question before i try it, you wrote class’ => ‘OC\Files\ObjectStore\S3

yet in the link

its 'class' => 'OC\\Files\\ObjectStore\\S3' ,here its double “\”, and its the same in the config.php, classes names are with “\” so which one to use?
thanks

Dan_Smith · January 16, 2018, 10:17am

sorry, missed that. I just rechecked my config and you’re right it does have the double slashes - not too sure why when I copied it over it came out

EDIT - my fault, when I put my original post I forgot to put the code tags around it so it took the second slash to be an escape character and just shows one - edited that original one now and it shows correctly!

ibr · January 16, 2018, 11:16am

sorry for the bother,

I added the exact:

  'objectstore' => array(
        'class' => 'OC\\Files\\ObjectStore\\S3',
        'arguments' => array(
                'bucket' => 'nextcloud',
                'autocreate' => true,
                'key'    => 'XXXXXXXXXXXX',
                'secret' => 'xxxxxxxxxxxxxxxxxx',               
                'use_ssl' => true,
                'region' => 'us-east-1',
		    ‘use_path_style’ => false,
        ),
  ),

before even creating the admin user, then after I did that, reset the apache sudo service apache2 restart then refreshed the page, and continued.

now it was obvious server didn’t take the S3 bucket since the bucket is empty, but the weird thing, when I opened the config.php again, I didn’t find S3 configuration anywhere (the part that I have added). could it be overwritten, or I just forgot to save? have you faced something like this?

thanks

ibr · January 16, 2018, 3:34pm

the Amazon S3 part was deleted maybe because I restarted the apache, or maybe I just forgot to save, anyways I tried again with the same Amazon S3 configuration and refreshed the page, the configuration is there and wasn’t deleted , and I can create users and sync files but when I look what inside the S3 bucket through cloudberry I found file like urn:oid:1149

is that how NextCloud stores files in S3?

Dan_Smith · January 16, 2018, 3:47pm

yep, that’s right. if you look in the nextcloud ui you should be able to upload/download docs correctly.

couling · June 7, 2018, 9:11pm

I just wanted to drop a note to say that I have managed to migrate my instance to S3 from local storage, but it took some fairly mega changes to the file structure to achieve.

If your feeling brave (or stupid) I’ve written up my method here:

https://pedal.me.uk/migrating-a-nextcloud-instance-to-amazon-s3/

Reiner_Nippes · June 8, 2018, 2:36pm

Another way: -> https://bitbucket.org/nikratio/s3ql/
Mount S3 as a local fs before the setup of NextCloud.

lesaff · June 8, 2018, 3:47pm

I am currently away on vacation from June 1 to June 8, back on June 11. If you require immediate attention, please contact techsupport@adnetinc.com or updates@adnetinc.com

Regards

Rudy

flyinghuman · September 26, 2018, 8:04am

Hi couling, your Server seems to be offline. Can you please check? I found your method earlier and now i can need it thanks!

couling · October 4, 2018, 1:50pm

Yeah. Mysql crashed the day I left for vacation. It should be back up now but may go down again until I can access AWS.