[NextCloudPi 12.0.3] Upload crashes my Raspberry Pi

Hi,

I recently started NextCloudPi and I suffer from crashes when uploading files (pictures). These already occur when uploading 10-20 files of 5MB each. I use a Raspberry Pi 3 Model B with the official power supply. I ran some stresstests (cpu, mem, io) and the board was stable. Looking for your help…

I had a terminal open for 2 of those crashes.

  1. Crash resulted in stalled my raspberry pi, had to powercycle.
    Terminal logging: https://pastebin.com/UFJn3En7
    Most notable entries:

Message from syslogd@raspberrypi at Sep 27 19:21:49 …
kernel:[ 6137.025877] Internal error: Oops: 37 [#1] SMP ARM

Message from syslogd@raspberrypi at Sep 27 19:21:49 …
kernel:[ 6137.037443] Process ksoftirqd/0 (pid: 3, stack limit = 0xbcce6210)

  1. Crash resulted in unresponsive apache2 and php-fm ghosts, but with functional terminal/sshd.
    Terminal logging: https://pastebin.com/X69QF4xY
    dmesg output: https://pastebin.com/bp7ygLYE
    Notable entries:

[ 458.420875] kernel BUG at lib/radix-tree.c:769!
[ 458.421476] Internal error: Oops - BUG: 0 [#1] SMP ARM
[ 458.422081] Modules linked in: cmac bnep hci_uart btbcm bluetooth brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem uio_pdrv_genirq uio fixed ip_tables x_tables ipv6
[ 458.424175] CPU: 2 PID: 873 Comm: php-fpm7.0 Not tainted 4.9.48-v7+ #1034
[ 458.424888] Hardware name: BCM2835
[ 458.432685] Process php-fpm7.0 (pid: 873, stack limit = 0xb91a6210)


edit: 3rd occurence (pi completely stalled)
Message from syslogd@raspberrypi at Sep 27 20:14:39 …
kernel:[ 1482.011679] Internal error: Oops: 5 [#1] SMP ARM

Message from syslogd@raspberrypi at Sep 27 20:14:39 …
kernel:[ 1482.022784] Process swapper/0 (pid: 0, stack limit = 0x80c00210)

I monitored system RAM usage and it seemed to vary between 40M and 200M free for a long time before the crash, with two php-fm instances running for a long time:

861 www-data 20 0 567732 115408 25264 R 65.6 11.5 5:13.27 php-fpm7.0
1132 www-data 20 0 485452 33224 25712 S 45.3 3.3 5:14.96 php-fpm7.0

thanks for the info. This is new.

That kernel BUG looks bad. Never seen it before. Some people have reported that it takes a long time to upload many pictures but that’s reasonable considering the power of the pi and the amount of data.

Maybe you can try playing with PHP max children (pm.max_children)

mp.max_children is already set to 3 in this release (updated to NextCloudPi v0.31.1 also). Even setting it to 2 does not reduce crashes.

I set enable_preview to false, but this also prevents opening any file via the web interface… clicking on a photo gives me a download popup but not the actual image. Can I disable previews/thumbnails and still have the image viewer functional?

I disabled the previews for test with uploading… no longer crashes. Even after the upload, if I then enable previews the thumbnail building starts, and I have kernel errors again (twice) and shortly after the pi stalls:

[22107.765008] Unable to handle kernel paging request at virtual address e3530029
[22107.765669] pgd = b4748000
[22107.766276] [e3530029] *pgd=00000000
[22107.766898] Internal error: Oops: 5 [#1] SMP ARM
…
[22107.769611] CPU: 3 PID: 6567 Comm: top Not tainted 4.9.48-v7+ #1034
[22107.770327] Hardware name: BCM2835
…
[22107.776653] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user

and:

[22242.118012] Internal error: Oops - undefined instruction: 0 [#2] SMP ARM
[22242.120540] Modules linked in: cmac bnep hci_uart btbcm bluetooth brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem fixed uio_ pdrv_genirq uio ip_tables x_tables ipv6
[22242.128700] CPU: 1 PID: 8142 Comm: top Tainted: G D 4.9.48-v7+ #1034
[22242.134486] Hardware name: BCM2835

Maybe you could try again after reinstalling the kernel to upstream. It is a 4.9 now anyway

Seems stable after installing kernel “4.9.52-v7+ #1038 SMP”… will report in a few days if truely stable!

Thank you so far.

edit: not fully stable…but improved…?

[ 2846.232399] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM

Happy to discover that i’m not the only one. It happened for me after couple of giga d’upload.
Any update ?

If anyone can consistently reproduce this, it would be nice to test with different kernels, as it seems like some kernel instability under high load

After months of failure, i could upload 200 giga of data without issue.

The fix : reinstall of my server with a NEW sd card.
Hit : Even with a watchdog activated and tested the Pi was staying stuck.

weird errors almost always come from faulty SD cards or cheap power supplies

also upload performance has been improved now that we use redis.

good to know!

Hi,

I am sorry to have to report the same issue is repeatable on the PI distribution NextCloudPi_02-06-18.img with kernel 4.9.79-v7+

It crashes on images like tif och psd, but seems to handle jpg or any other docs (not 100% sure, synch aborts and crashes too frequently)

Message from syslogd@nextcloudpi at Feb 16 23:38:35 …
kernel:[19633.293175] Internal error: Oops: 17 [#1] SMP ARM

Message from syslogd@nextcloudpi at Feb 16 23:38:35 …
kernel:[19634.047928] Process swapper/3 (pid: 0, stack limit = 0xb9d4e210)

Message from syslogd@nextcloudpi at Feb 16 23:38:35 …
kernel:[19634.099783] Stack: (0xb9d4fcf8 to 0xb9d50000)

Message from syslogd@nextcloudpi at Feb 16 23:38:35 …
kernel:[19634.151654] fce0: af1e7280 baa4147c

Message from syslogd@nextcloudpi at Feb 16 23:38:35 …
kernel:[19634.203808] fd00: b9d4fd54 b9d4fd10 8025f1b8 8071bea4 b9d4fdd4 b9d4fd20 80159f98 80159468
… etc

From the admin log:

Fatal webdav Sabre\DAV\Exception\ServiceUnavailable: Could not open file 2018-02-17T00:58:52+0100
Error PHP fopen(/media/USBdrive/ncdata//files/Scannat/Tele Mobil Internet/Old/Tre 2007-07-31.tif): failed to open stream: No such file or directory at /var/www/nextcloud/lib/private/Files/Storage/Local.php#282 2018-02-17T00:58:52+0100

(surrounding PDF files seems to be copied)

After this. the PI is simply dead!

Best regards
Walth

can confirm the crashes and can reproduce them easily on the 2018 based NexcloudPi img. Have now built a nextcloud 13 environment on the same Pi with the same SD card manually and see no crashes. Missing the advantages of the NextcloudPi package though.
What could I check to find out the potential cause for the crashes? Could run a packed and a manual setup in parallel and test.

Regards
Wilf

Interesting. So, what image are you based on? Plain Raspbian? what kernel are you using in the image that doesn’t crash?

that would be great

yes it’s a plain Raspbian, actually the latest one (Nov 2017) you get from the raspberry.org download page.
Here is the kernel info:
Linux walabncp 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux

and here is the Kernel info of the NextcloudPi one:
Linux nextcloudpi 4.9.79-v7+ #1086 SMP Wed Jan 31 18:03:34 GMT 2018 armv7l GNU/Linux

can you try copying the kernel from the raspbian version to the ncp version?

do you know what the process is for copying a kernel from sys A to sys B ?

mmm I don’t remember exactly.

I think it is just a matter of copying /boot/kernel7.img ( I guess the whole folder can be copied ), and /lib/modules as well.

But better google that first

any luck with this? are you positive that the issue is 100% reproducible with ncp image and 0% reproducible with plain raspbian?

if that is the case, I can generate a NCP image that doesn’t update the kernel to rule that out, but that would take some time, so ideally just replacing the kernel files should be good enough for quick verification.

@remy, also it would be great if you could find the kernel crashes by running dmesg, to see if they are the same as in

didn’t find any time but will restart next week

Any news about a fix here? Having the same problem when uploading a lot of files. Crashes out my PI 3