Maximum amount of data/files that can be used reasonably with Nextcloud on a Pi

🏷️ General
1

I’ve been looking around in the Forum a bit, but didn’t find an answer to my specific questions.

I’ve been using Nextcloud at work and liked it so much, that I also wanted to use it privately. I’m mainly interested in the file sharing and not the modules like calendar, contacts, gallery etc…

I have two computers, a Mac and a PC, and want to share my data between them. So my entire data would be on both computers and synchronized between them via Nextcloud.

My setup is:

  • Raspberrypi 4 with 8GB and RaspberryOS (using USB Boot – no SD-Card involved)
  • A SATA SSD connected via USB3. Benchmark read speeds are about 314 MB/s; write speeds 183 MB/s
  • The Pi is connected to the network via Gigabit ethernet.
  • Nextcloud v1.18.0 installation via https://raw.githubusercontent.com/nextcloud/nextcloudpi/master/install.sh (i.e. on top of Raspbian… no Docker or NC Image)

My experience so far has been disappointing.

At first I’ve been testing it with about only one fifth of the data, which I want to keep on Nextcloud, which is about 300GB consisting of 190.000 files. (In total I would like to store about 1.7TB consisting of 300.000 files).

First problem I see:
The initial load has been very very slow. I did not expect wonders by the Pi and by transferring files over the network, but I also didn’t expect it to be that slow.

What would be the best way to perform the initial load? Lets say, if I have all my data on the Mac, would it work if I first copy it over manually to the Pi and to the PC (e.g. with a USB HD), and then add the Nextcloud Synchronization? Would nextcloud recognize, that this are the same files and nothing needs to be synchronized?

I’d be interested to understand, how the synchronization works. Is it the client, that checks for differences between the client and the server and performs the necessary changes? What are the criteria used, to determine differences?

Second Problem:
Since the number of files is so big, it seems to take ages for the Nextcloud client to find differences and start synchronizing.
I’m a bit afraid, that if the time to find differences is so long (I haven’t measured it yet, but for the sake of argument lets say it is one hour), it will never synchronize any changes, if you switch on your computer for less than an hour.

So my questions are:

  • What is the best way to perform the initial load?
  • Is there a way to improve/speedup the process, that looks for changes?
  • Is there maybe a limit to the amount of data/number of files, that can be used reasonably with Nexcloud running on a PI?
2

First perhaps is Nextcloud not the best solution for you.
A Pi is not the fasted and best system.
Perhaps you can use other hardware.

You can e.g. copy the files with sftp (Filezilla?) to the Pi and there in the data-dir of the user. After upload you must re-scan with occ. Link

3

If it is mainly to sync the devices and you don’t need the other functions and don’t want to share, perhaps take a look at syncthing. git-annex has a few nice options (like keeping at least 3 independent copies). The clients can sync with each other without a server (I hope to remember correctly).

You won’t get the same performance with Nextcloud, and I’d only go the extra step with a better setup and tuning the performance if you use other features (sharing, apps, etc).

4

Thanks for the replies.

Then I wonder, what other functions would justify the use of nextcloud?

What kind of data volumes/number of files do other people have running on nextcloudpi?

5

I just want o follow up on this.

I’ve now been using it for a while, and am actully quite happy with it.

Once everything is synched, it works reasonably well. Now I only have small changes, and they are synched reasonably fast. (Also the check for changes, is actually quite fast. I think the check appears only slow, as long as not everything is synched up yet during the “inital load”.)

So I’m quite happy to stick with this solution.

6

Well, people are very different, so some might not be happy at all when you are or the other way round. I have done a few tests on the 3 model where you could really feel an important difference to a “normal” computer. The 4 model is much better, and with 8 GB of RAM you can use different caches that speed the interface up.
And you can always start with such a setup and migrate the setup to a more powerful setup if you need it.