The program Nextcloud crahes wenn I sync many files

Nextcloudpi 13.0.2
raspberry pi 3 B

Hallo,
I have installed Nextcloudpi one a Rasbarry Pi.
I have installed the Windows Client on my Windows 10 and connected it with Nextcloudpi.
The Sync runs, when I save one file in C:\Users\siegfried\Nextcloud.
Wenn I put all my files (50 GB) in C:\Users\siegfried\Nextcloud\ the program Nextcloud crahes.
The Linux keeps running.
What is the problem?

Thanks in advance for the help
Siegfried

What is your load average looking like at the time? (Hint, watch it on the comand line with “htop,” which you should install if it’s not already installed). I’ll bet the load average is through the roof, and your Raspbery Pi just doesn’t have the horsepower, nor Disk I/O speediness to handle such an arduous task. Considerable metadata in the database must get created for every file you copy into Nextcloud.

The answer is to not dump all 50GB in at once, but rather try to put it in a few GB at a time, and watch to see how the Raspberry Pi can handle it (again, watch htop). Any time the load average goes above, say 4ish, the Raspberry Pi will probably act very bogged down. It’s not that Nextcloud has crashed, but rather it’s just really, really bogged down, waiting for the painfully slow Disk I/O of the Raspberry Pi to process the workload placed upon it.

If you want a cheap ARM board for running Nextcloud, comparable to the Raspberry Pi, consider getting a NanoPi Neo Core 2, with the “NAS kit”, aluminum enclosure, and a 2.5" SATA disk (preferably an SSD). The disk I/O will be quite a bit better for a bit more cost. I have one of these kits myself.

It could be that the Pi has insufficient memory.

I had recurring problems on my VPS with the nextcloud process crashing. Upgrading the system from 1GB to 4GB RAM solved the issue.

Thank you for your help

My system is a Rasbarry Pi 3+ whis an 8 GB micri ssd an a usb Stick 512 GB.

Here is one Screenshot. I hope its help.
Wenn I put 1,5 GB files in C:\Users\siegfried\Nextcloud\ the Nextcloud I have the same problem. Nextcloud crahes
Ther is a Mistace in the log. The file is in the ignoratelist (Die Datei ist in der Ignorierungsliste geführt.) and File was ignored because it is canned.
Where ist this ignoratelist and how can I edit it.
I dont want that Nextcloud ignored JPG and odt … file

Siegfried
(Sorry for my english)

The Raspberry Pi shoud be understood as being useful to prototype Nextcloud. You know, just get a little first taste of what’s it’s like to install and use (which is a great experience in and of itself, even if only to learn how it all works). But don’t expect it to actually haul any real workloads to speak of. The Raspberry Pi is just a starting point, for those wanting to approach Nextcloud using a cheap-as-possible initial investment.

Your Nextcloud server will almost certainly behave like a toy that breaks easily, if you don’t get something at least a little more performant, like say at least the NanoPi Core Neo 2 NAS kit I mentioned above (which was only like $80, plus shipping, which doesn’t include the price of a 2.5" hard drive). Any of the SBC’s which have a good current reputation on armbian.com should also be considered, as you can install Nextcloud now from Armbian’s “Softy” submenu (of the “armbian-config” text-based config utility).

The new LattePandas just out recently (such as the Lattepanda Delta) are quite sexy as an SBC, as well, and should work great with Nextcloud, as they are fully Linux-compatible. These are AMD64 architecture, not ARM.

Or any used PC (as in, not a Chromebook) laptop with SATA, USB 3.0, and gigabit ethernet, would also be a cheap but performant option.

Don’t expect miracles, however I’d expect the rapsberry to be slower but not to produce errors. To balance the resources, I’d check a bit how many connections you allow on the webserver and how many processes you start for the webserver, database server and php.

Or try to reduce the bandwidth of the client.

Check if the memory split is set to minimal for gpu, see docs here

The RPI’s 1 Gb ram is split between gpu and cpu.
You can set it in /boot/config.txt by editing that file or running raspi-config and selecting it from the menu.

gpu_mem=64

You dont need any gpu if your not running GUI

You can check available RAM for cpu with the command

free -h

You will need to reboot for changes to take effect.
That said, the max upload size in latest ncp/nc is set to 10Gb.so you are probably hitting php’s limitations too. So I d try uploading in batches not exceeding 10Gb.

I think you’ll find this comparison interesting, @Bacchus

I installed Nextcloud 14 on my NanoPi Neo Core 2 LTS, with the Nas kit, and a 2.5" SATA Samsung SSD (and I used “ncp-config” to move the Nextcloud database, and datadir onto that SSD, not the MicroSD card). I was curious as to how the performance would compare to your Raspberry Pi, on copying files into Nextcloud (over SSL-encrypted WebDAV, using my Linux Mint’s Nemo File Browser).

For testing, I had a 2GB .zip file, which uploaded in 3:03, or 10.9 MB/sec average (Note: If that same 2GB file were uploading through FTP into the very same hardware, it would instead get 29.9 MB/sec). The NanoPi’s 4 cores all worked about 15% average, each, and the load average stayed at about 1.3 the whole transfer. The CPU was about 45 deg C max, in a room which had an ambient temp. of about 30 deg C.

I also had a folder containing 2GB of many very small files (2,785 files of varying sizes). Copying the folder in took 34:11, or 1.0 MB/sec average (Note: If that same 2GB folder were uploading through FTP into the very same hardware, it would instead get 18.7 MB/sec). The NanoPi’s 4 cores all worked about 25% average, each, and the load average peaked out at a max of about 1.5, but was usually like 1.2 the whole transfer. The CPU slowly climbed up to 62 deg C, in a room which had an ambient temp. of about 30 deg C.

Pictured just below: on the left is Nemo file Browser copying in the 2GB folder over WebDAV, and on the right, you see an htop running on the NanoPi, running Nextcloud 14:

This performance might sound really poor (especially on many tiny files), but at least the NanoPi Neo Core 2 LTS didn’t fall on its face like the Raspberry Pi. I would suggest that the NanoPi Neo Core 2 LTS, with the NAS kit and an SSD is sort of like a bare minimum for an office with like 1 or 2 people in it, who are just storing documents, etc.

IMHO, You are just sort of torturing yourself far too much, just to save a very small amount of money, if you go any lower than the NanoPi Neo Core 2 LTS (with an SSD).

I don’t work for FriendlyElec in any way (the makers of the NanoPi Neo Core 2 LTS), BTW.

Edit: when I used an ethernet cable to connect my laptop to my wifi router, then re-ran the tests (such that I had an all-Ethernet connection between my laptop and the Nextcloud server), my average upload speeds improved:

  • When uploading the 2GB .zip file, the speed jumped from 10.9 MB/sec (with wifi-from-laptop) up to 17.2 MB/sec (all-ethernet), average. That’s considerably closer to my FTP test (which I’m almost certain I did as all-Ethernet), which got me 29.9 MB/sec .
  • When uploading the 2GB folder of many tiny files, the speed very slightly improved from 1.0 MB/sec (with wifi-from-laptop) up to 1.1 MB/sec (all-ethernet), average. That’s still a far cry from the FTP test, which got me 18.7 MB/sec
1 Like

that’s cool @esbeeb thanks for sharing

I have been wanting to do some standarized benchmarking with different boards for a while, hopefully soon I will have some time to do that!

1 Like

@nachoparker , I would like to suggest and propose that you might more prominently warn your Nextcloud Pi downloaders (for the Raspberry Pi) that Nextcloud Pi is only suitable for prototyping and/or testing Nextcloud, but it’s not really performant enough for production use. The bare minimum hardware for actual production use could perhaps be suggested as any Armbian-supported ARM board with at least SATA (and/or USB 3.0) and Gigabit Ethernet.

I would further suggest that the NanoPi Neo Core 2 LTS seems to be the minimally cheap (even with the NAS kit expansion board) and currently supported board over at armbian.com.

Well, I don’t particularly like the pi, but it is so popular that it makes sense for our quest to spread self-hosting and Nextcloud to support it.

It performs quite decently as well, even though it’s not the best choice

the nanopi neo looks good!

I wouldn’t go that far. Sure, there performance is limited, however you can do a lot of stuff, running a media-center and stuff like that why not use it as a cloud storage. For a single user or a few users it can be enough. And it is a good exercise for developers to optimize their code and configuration.

Agreed. Hopefully webdav file upload can be made more efficient in a future version of Nextcloud, such that the Raspberry Pi doesn’t get totally swamped like @Bacchus is experiencing on his Raspberry Pi. If it could even handle 1MB/sec upload (on many tiny files) like the NanoPi Core Neo 2 LTS currently can, that would be great. Having said that, I’m no developer.

Until such time, I’ll certainly not install Nextcloud Pi on either of my own 2 Raspbery Pi 3B’s, as I would far rather use different, speedier hardware instead (for relatively cheap, but not quite dirt cheap like the Raspberry Pi). There’s a “dotted line of painfulness” that I just won’t go below (which ends up costing a little more), which makes my life free of such vexation.

BTW: @Bacchus, you might want to consider getting a speedier USB stick, like a Sandisk Ultra Flair 64GB (which I’ve used with good success in a Raspberry Pi before), then once formatted with, say EXT4, and hard mounted (in /etc/fstab), use the ncp-config tool to move the database, and data dir of Nextcloud onto the USB stick. Don’t run Nextcloud’s db and data dir on the microSD card, where Raspbian runs!

Your MicroSD card will only be writable at about 10 MB/sec, sustained, maximum, but the Sandisk Ultra Flair, formatted with EXT4, should be able to handle ~20MB/sec, sustained, for it’s write speed (and I’m talking about raw filesystem performance here, not the performance that Nextcloud actually yields through WebDAV).

@nachoparker, you might want to pass this tip (i.e. put the db and data dir onto a fast USB stick) on to your Raspberry Pi downloaders for Nextcloud Pi as well. This is a bit different from your current advice. My tip splits the load a little differently: all OS stuff, and binaries still run from the MicroSD, it’s just the Nextcloud data (database and user files in Nextcloud) which gets read and written on the USB stick. This sort of splits the load into two. I’ve never done benchmarks to see which actually runs the fastest (compared to your existing instructions about using some external drive instead of the MicroSD card), having said this.

you can do than in NCP with nc-datadir and nc-database, that is the way I have it set up :wink:

Aargh. Sandisk changed that stick now, you’ll be lucky to get 16 MB/sec out of it. I have 2 of them, and they look superficially identical. One bought about 7 months ago, and one bought one month ago. The 7-month-old one will write at a nice 24 MB/sec, formatted with EXT4. The 1-month old one, also formatted with EXT4, copying in the same files (2GB of many tiny files), only gets 16 MB/sec. The total possible space available on the two sticks is slightly different as well.