[NC19] How to move user data, without hickups

,

Perfect post about how to move data folder is here HowTo: Change / Move data directory after installation

Only if you have a new location, you can also stop the server, move data to the new drive and mount it under old location. In this case no configuration change needed.

If you do not follow instructions from the link from above you could meet this nice bug Files amount after moving of data directory is wrong (much bigger)

@gas85,

Thanks again for all the help.
With regards to the bug, i already have a sort of similair problem. Like i told you, i cloned the nextcloud setup and removed all the users with much data. Ieft a couple accounts in total about 55GB. But when i look at the Nextxloud GUI it tells me 500GB is used.

I assumed that the nextcloud config wouldn’t take about 450+ GB. So do you have any idea why my Nextcloud is so big?

With regards to the moving, yes i would like to mount the new drive onder the old location. That was how i thought about it in my head :slight_smile:

But what do you mean by “stop the server off”?

Try to rescan data folder:

@gas85,
When i use ```
sudo -u www-data php occ files:files:scan --all
it tells me that:

There are no commands defined in the files:files namespace.
Did you mean one of these?
files
files:recommendations

should i just remove :files?

You right, i change this typo.
Docu about scan you can find here: https://docs.nextcloud.com/server/19/admin_manual/configuration_server/occ_command.html#file-operations

@gas85,

Just so you know, i tried to remove the 1.5TB disc on an extra cloned installation and this indeed broke the pool.

So i deleted this clone and will go back to trying to move the data to disc4 from the zfs pool on the first clone.

The problem with not showing the correct amount of data used, was solved by the commands you gave me. So on to the next problem :wink:

@gas85,

Sorry i am really a linux n00b. I have a new disc (sdd). But before i can move i will have to create a partition “sdd1” and format the partition as lvm right? I have tried this:

From this website:
https://docs.fedoraproject.org/en-US/quick-docs/creating-a-disk-partition-in-linux/

  • sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on
  • sudo parted /dev/sdd
  • mklabel gpt
  • mkpart primary 0 214GB
    When i did this i got “the resulting partition is not properly aligned for best performance ignore/cancel” so is this correct?
    quit

Is what i did correct?

And i am not sure what to do next and how to do it. I have created a partition now right? But how do i format it as lvm? See picture for my new disc + partition.

2020-09-06 01_53_54-root@cloud_ ~

@gas85, are you on holiday or maybe not receiving the email updates? Of course i am not in a big hurry and i am very greatful for you help so far and certainly not wanna rush you :slight_smile:

Somebody else maybe a suggestion? @eehmke maybe? You helped me out before in a great way!

Sorry, have a lot to do at work and wedding anniversary also taking a lot of time :slight_smile:

First of all you need to decide what you want to achieve: are U going to expand size of your hdds? Or you want to have something like RAID 1 where data is mirrored in case one of hdds is getting broken.

Why U need sdd? To move data folder from it? If yes, then it looks a bit too small, according to your screenshot from above you have around 1.4 Tb of data, it will not fit into 200 GB sdd.

Could you please check how you are using drives now?

df -h

Then check how big is your data directory. Please replace /var/www/nextcloud with your path to the Nextcloud:

du -sh $(grep datadirectory /var/www/nextcloud/config/config.php | cut -d "'" -f4)

Basically LVM looks like this lvm_scheme_full1

How LVM works in a simple 3 steps:

  1. you initialize any drive or partition to LVM by pvcreate. You even do not need to create any partition, simply added whole drive.
  2. you will create new virtual group on selected partition vgcreate
  3. you will create new logical volume on selected virtual Group lvcreate
  4. you will create FS on a logical volume that you set with e.g. mkfs.ext4 or zfs if you need.

Now you can move data to the new volume and added new drives to expand space. This is short story… Long for example here but it is in Russian or here is in English.

@gas85,

first of all happy anniversary! I have removed a lot of user data from my test Nextcloud so it was easier to try/test.

I simply want to add a new (virtual) hard disc (which i think i did, the “sdd”). So i would like to move the data FROM the zfs pool to SDD. After that delete the zfs pool, and mount the data on SDD exactly the same as it was (i think that is /mnt/ncdata). According to df -h, the sdd drive should be big enough right? SDD is 200GB and Ncdata is 60GB right?

2020-09-16 12_31_56-root@cloud_ ~

so now i just type in “pvcreate”? I do not have to change my console to “sdd” first? (Like in windows from C to D partition)?

There are a lot of LVMs howto in internet. Just to not live you without anything above I post already example of reference.

You can ether reuse your existing Volume Group: nextcloud-vg (in this case skip step 2 and replace vg-next-data with nextcloud-vg), or added new one. This example will added new one.

  1. Please MAKE A BACKUP OF DB AND DATA and other relevant folders BEFORE TO START:

  2. You need to create a LVM physical volume on the partition

pvcreate /dev/sdd1
  1. Then you need to create volume Group (e.g. name “vg-next-data”)
vgcreate vg-next-data /dev/sdd1
  1. create the logical volume that LVM will use (e.g. name “lv-next-data”):
lvcreate -L 200G -n lv-next-data vg-next-data

The -L command designates the size of the logical volume, in this case 200 GB, and the -n command names the volume. Vgpool is referenced so that the lvcreate command knows what volume to get the space from.

  1. Format and Mount the Logical Volume
mkfs.ext4 /dev/vg-next-data/lv-next-data
mkdir /mnt/ncdata_new
mount /dev/vg-next-data/lv-next-data /mnt/ncdata_new
  1. Stop your nextcloud/webserver, stop your DB
  2. Move data from /mnt/ncdata to /mnt/ncdata_new
  3. Dismount /mnt/ncdata and /mnt/ncdata_new
  4. Mount your LVM to the old location
mount /dev/vg-next-data/lv-next-data /mnt/ncdata
  1. Ensure, that zfs is not mounted anymore:
lsblk
  1. Start your server and check if everything ok and working.
  2. Play with your ZFS.

P.S. after you finished you can added your sdb to the LVM as physical disk:

vgextend vg-next-data /dev/sdb1

and then simply move data from sdd to the sdb:

pvmove /dev/sdd1 /dev/sdb1

And remove sdd from the LVM:

vgreduce vg-next-data /dev/sdd1

No mounts, no issues…

@gas85,

I have been away for a while (some personal business, sick family members :frowning: ), but i am back :slight_smile:
I hope you are still happily married :stuck_out_tongue_closed_eyes:

This is what i did:

pvcreate /dev/sdd1
vgcreate vg-next-data /dev/sdd1
lvcreate -L 190G -n lv-next-data vg-next-data

mkfs.ext4 /dev/vg-next-data/lv-next-data
mkdir /mnt/ncdata_new
mount /dev/vg-next-data/lv-next-data /mnt/ncdata_new

  1. sudo systemctl stop apache2
  2. sudo systemctl stop postgresql
  3. sudo rsync -avP /mnt/ncdata /mnt/ncdata_new/
  4. umount /mnt/ncdata
  5. umount /mnt/ncdata_new
  6. mount /dev/vg-next-data/lv-next-data /mnt/ncdata

This is what lsblk shows now:
2020-12-17 21_47_01-root@cloud_ ~

so i think i did it :slight_smile: I do not have to do some sort of SAVE ALL right? Not that after a reboot the mounting point, points to the previous partition again :stuck_out_tongue:

Is there a way to rename the "vg–next–data-lv–next… ? Just because it looks a bit ugly :stuck_out_tongue:

Do i still have to do the “P.S.” part? I am not sure why i should add sdb to LVM as a physical disk. And why should i move the data back from sdd to sdb? Isn’t the next thing to do just delete/remove sdb en sdc? And after that remove the virtual drives?

I am talking about this part:

P.S. after you finished you can added your sdb to the LVM as physical disk:

vgextend vg-next-data /dev/sdb1

and then simply move data from sdd to the sdb:

pvmove /dev/sdd1 /dev/sdb1

And remove sdd from the LVM:

vgreduce vg-next-data /dev/sdd1

No mounts, no issues…

To me it seems a bit like it is done after step 11, of course not saying you are wrong just curious :slight_smile:

Again many thanks so far! after these final steps i am gonna clone my Nextcloud again and do this proces again in one evening. If that goes well, i will do this to my “production” Nextcloud installation. (Of course not without a proper extra backup :slight_smile: )

1 Like

@gas85,

I have tried to skip the “P.S. part” and reboot the VM, but then i am not able to login my user account an during the reboot i see all kind of ZFS errors. So apparently i still have to remove and links to the ZFS pool?

“P.S.” Part is only possibility to move data between the HDDs without even unmount it. E.g. when older disk dies and you can still move data to the new one.

Regarding ZFS, I belive you need to cleanup your /etc/fstab and live only needed parts, e.g. boot and with LVM.

@gas85,

I have looked in fstab file:

But i don’t know what do do here, as far as i know the nextcloud–vg-root and swap are necessary. And i am not seeing and ZFS references.

I have started my sql en apache and i got the error “make sure .ocdata is present in main directory”.

So i compared with my production NC, and the user data is not in the /mnt/ncdata directory on my test nextcloud. The files are in the /mnt/ncdata/ncdata.

So somewhere down the line i made a mistake. So i need to move the files from /mnt/ncdata/ncdata to /mnt/ncdata. i moved the files with this command:

mv -v /mnt/ncdata/ncdata/* /mnt/ncdata

@gas85,

So i moved the files, but i am still getting an error:

datafolder is invalid
make sure .ocdata is present in main directory
Your data directory is not writable

Rights can be restored by giving root write access to the folder on the webserver
https://docs.nextcloud.com/server/20/go.php?to=admin-dir_permissions.

So i think i need to give write access to root on the new /mnt/ncdata folder. Do you agree?

Is this the right thing to do?

sudo mount -a
sudo chmod -Rf 777 /mnt/data

Hey first of all check where you data folder configured, just find config.php in your NC folder, e.g.:

grep "datadirectory" /var/www/nextcloud/config/config.php
#Outup example
'datadirectory' => '/var/nextcloud/data',

you will see where it should be, then check if .ocdata is there:

ls -la /var/nextcloud/data/.ocdata
-rw-r----- 1 www-data www-data 0 Dec 14 09:31 /var/nextcloud/data/.ocdata

Usually you have like 660 for files and 750 for folders. Owner is your web user.


I’m not familiar with ZFS, but documentation says you have to use something like zpool remove to remove drives from the pool and then destroy it.

@gas85,

This gives me the correct result:

(COMMAND) grep “datadirectory” /var/www/nextcloud/config/config.php
(RESULT) ‘datadirectory’ => ‘/mnt/ncdata’,

But i cannot seem to locate the .ocdata file:
ls -la /var/nextcloud/data/.ocdata
ls -la /mnt/ncdata/.ocdata

Both gives me “No such file or directory”.
But when i look at my LIVE Nextcloud in the /mnt/ncdata directory, there is also no .ocdata file…

When i search for the file .ocdata:

(COMMAND) find/ -name .ocdata
(RESULT) /mnt/ncdata/ncdata/.ocdata

So that would mean the file didn’t move when i moved all the files (from /mnt/ncdata/ncdata to /mnt/ncdata). But when i “ls” in the folder /mnt/ncdata/ncdata it doesn’t show any files. Isn’t that weird?

So i used “mv -v /mnt/ncdata/ncdata/.ocdata /mnt/ncdata” . Now when i search for the .ocdata file, it shows me that the file is located in /mnt/ncdata. So that should be ok now.

When i try to open the nextcloud by browser, it only says now that the data directory is not writeable. So we went from 2 to 1 error :stuck_out_tongue:

@gas85,

With regards to the rights, i saw this website:

They say:

chown -R www-data:www-data /mnt/ncdata/

So you can set the premisson like “grouchysysadmin” written.

chmod 0750 /mnt/ncdata/ → ( 7|5|0 mean => user: rwx | group: rx | other: - )
w = write / r=read / x=execute / - = no premissions

In your case: chmod 0750 /mnt/ncdata/
maybe you have to set the same premission to every subdirectory.

find /mnt/ncdata/ -type d -print0 | xargs -0 chmod 0750

And every file in the data folder
find /mnt/ncdata/ -type f -print0 | xargs -0 chmod 0640
→ exept: .htaccess
chown root:www-data .htaccess
chmod 0644 .htaccess

Are you sure i should use “sudo chmod -Rf 777 /mnt/data”?

Noo, better