This easy to follow tutorial will show the best practice to migrate mass data, say from an existing NAS (network drive) to a new NextCloud server, effectively and efficiently without wasting upload transfer time which may be a very lengthy process, save power and resources and get up to speed with client synchronization.
This tutorial is intended for everyone (beginner to advanced user)
You have an existing drive with over 1gb of data that you wish to migrate to your new NextCloud Install. You may also have an existing NextCloud install and wish to add new mass data to your server without recurring to use the NextCloud client sync app and or browser upload features.
This tutorial assumes that you have the following:
- A NextCloud Server properly installed and configured with enough storage to host the new data you are about to migrate. Preferably you have a high capacity drive already setup as a local under “external storage” in NextCloud server settings. Note on NextCloud 11 you need to enable the External Storage App from the Apps before.
- An external NAS or drive that stores your data that you wish to migrate to NextCloud. Ideally the data within is already sorted and orginised in a folder structure that you are happy with.
- If you intend to store NTFS files (for Windows Clients) you have already completed the tutorial Temporary NTFS tutorial. It will be replaced with a proper tutorial . Skip this if you will use Linux default filesystem.
Physically connect the external drive (generally via USB) containing the data that will be migrated to the host that has NextCloud server installed on it and then powerup the system.
SSH into your server as a superuser.
Let us find the dev that Linux gave to the external drive so that we could mount it. To do this, from terminal run
You should be presented with a list of devices and their mount point. You will be presented with a list similar to this
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.8T 0 disk ├─sda1 8:1 0 250G 0 part /media/crypto └─sda2 8:2 0 1.6T 0 part /media/win sdb 8:16 0 1.8T 0 disk └─sdb1 8:17 0 1.8T 0 part sdc 8:32 1 29.8G 0 disk ├─sdc1 8:33 1 487M 0 part /boot ├─sdc2 8:34 1 1K 0 part └─sdc5 8:37 1 29.3G 0 part ├─hostname--vg-root 252:0 0 21.3G 0 lvm / └─hostname--vg-swap_1 252:1 0 8G 0 lvm └─cryptswap1 252:2 0 8G 0 crypt [SWAP] sr0 11:0 1 1024M 0 rom
In the output above, I have sda set as my local external storage for NextCloud with two partitions named win (sda2) and crypto (sda1). Crypto is used for server side encryption and for the purpose of this tutorial I will omit it.
Notice that the external drive has been recongnised as sdb with one partition containing the data being sdb1. Remember to replace sdb1 with your dev number that the command lsblk has given you.
Let as create a directory so that we mount the external drive to. From terminal type
sudo mkdir /media/extdrive
You may wish to change “extdrive” to any name you wish as long as its not already mounted and used.
Now let us mount the drive to the folder we just created
sudo mount /dev/sdb1 /media/extdrive
Let us cross check to make sure the mount was correct by running again
This should output
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.8T 0 disk ├─sda1 8:1 0 250G 0 part /media/crypto └─sda2 8:2 0 1.6T 0 part /media/win sdb 8:16 0 1.8T 0 disk └─sdb1 8:17 0 1.8T 0 part /media/extdrive sdc 8:32 1 29.8G 0 disk ├─sdc1 8:33 1 487M 0 part /boot ├─sdc2 8:34 1 1K 0 part └─sdc5 8:37 1 29.3G 0 part ├─hostname--vg-root 252:0 0 21.3G 0 lvm / └─hostname--vg-swap_1 252:1 0 8G 0 lvm └─cryptswap1 252:2 0 8G 0 crypt [SWAP] sr0 11:0 1 1024M 0 rom
Notice that dev/sdb1 is now mounted at /media/extdrive.
Ok so far so good, let’s head to the data migration. For the intents of this tutorial, we will keep to /media/win as the local external drive set in our NextCloud install and we will be using a very useful bash command “rsync” to merge the data from our external drive to the NextCloud data folder.
We will be changing to root for this to bypass any restrictions, from terminal type
Notice the prompt changed to
Now we will typing the following command to merge the data. Replace “foldercontainingdata” with the name of the folder you used to store all the data on the external drive.
rsync -a /media/extdrive/foldercontainingdata/ /media/win
The data on your external drive is now being copied and or merged onto the storage of NextCloud Server local external storage.
This will take some time, depending on your server’s processors, ram and the amount of data you are transferring. It took me about 10 minutes to transfer 1.6TB of data on a dual xeon processor. If you are using an SoC like a Pi it would be longer.
When data syncing is ready you will be presented with the root prompt once again
I have the habit to cross check all processes, and would like to ascertain that the syncing happened. To do this I simply cd to the NextCloud data folder and run ls. As we are root I simply do
cd /media/win ls
This will present you with the folder structure and files of the data that has been synced.
Now its time to force NextCloud to scan for the new files. To do this from terminal type the following:
cd /var/www/nextcloud sudo -u www-data php console.php files:scan --all
This will start the scanning, needed to populate the NextCloud database and will output something like this
Scanning files for 2 users Starting scan for user 1 out of 2 (username) +---------+--------+--------------+ | Folders | Files | Elapsed time | +---------+--------+--------------+ | 28585 | 107801 | 02:11:30 | +---------+--------+--------------+
This process will be done for all the users.
Lastly we safely unmount our external drive sdb1 is the dev used in this tutorial, make sure to use yours instead.
sudo umount /dev/sdb1 sudo reboot
That’s it, the round trip time for the transfer to this data including NextCloud Scanning was a couple of hours, compared to over a month to do such with the conventional Desktop App or browser.
If you liked or found this tutorial helpful give it a like “heart”. Post any relevant questions hereunder, I will do my best to answer all.