Photo viewing cripples server

[spyro]

Hi,

Is it an intended feature of NC28 that scrolling through the photos app in the web interface is an effective DoS against every other simultaneous operatoin on a 4 core 2GHz x86 with 8GB of RAM and RAID5 that can hit 600MB/s?

I can’t believe that a machine with that much horsepower (even though you can spend far more on hardware nowadays) is really incapable of serving up a few thumbnails over the LAN without completely crippling itself.

I honestly thought that nextclouds utterly tragic upload speed whilst uploading photos (~1MB/sec over LAN) was because it was (behind the scenes) generating a few sizes of thumbnail and writing some indexes, but if it is, then I can confidently say that it’s totally wasting its time, as it obviously isnt working.

So if the upload is slow for some other reason, WHY? it’s got gigabit ethernet, it works FAST for wget / iperf. so it’s not the network. DNS is correct. certs are correct.

the RAID array can read the photos to /dev/null at ~400MB/s, with the CPU sat twiddling its thumbs on all 4 cores.

WHY is nextcloud such a slowpoke? I could use this machine to emulate a Win 95 PC and access it via SAMBA with more performance than this.

Has the whole world forgotten how to write anything that remotely resembles high performance software?

Or is it just the usual case of the documentation being too sh*t for anyone but the developers to actually know how to set it up?

Which is it?

Prediction: I’ll get one, maybe two “enthusiastic looking replies”,followed by people telling me to wind up the PHP memory usage to the point where PHP fpm tries to allocate 160GiB, followed by radio silence, just like everyone else with a performance problem in NC, AFAICT.

[bb77]

@spyro

Prediction: I’ll get one, maybe two “enthusiastic looking replies”,followed by people telling me to wind up the PHP memory usage to the point where PHP fpm tries to allocate 160GiB,

Well it looks like you defentitly should tune your PHP, preferably use PHP-FPM, and enable and tune OPcache and use an aditional memory cache like APCu, and Redis for File locking. All that will massively increase performance.

Prediction: You will now either tell me something along the lines that you have already done a lot of these things, of course without providing any relevant details, or that it shouldn’t be necessary in the first place because it requires effort and you don’t feel like it. Probably a mixture of both.

@spyro

followed by radio silence, just like everyone else with a performance problem in NC, AFAICT.

I mean what do you expect from your post? You are providing zero information combined with snarky comments, while expecting the community to fix your Nextcloud for you.

I can only say that the upload speed on my instance is up to 80 MB/s and the navigation is pretty snappy. Not Google Photos snappy, but snappy enough.

By the way, the previews are generated when you browse or open the files, not when they are uploaded. Something the preview generator can help with: Preview Generator - Apps - App Store - Nextcloud

[spyro]

I don’t think it’s snarky to complain that following the installation instructions in the documentation (incl. the optional performance improving bits about redis and the jit) results in a near non-functional setup.

I’ve already enabled the PHP opcache and redis. NC stopped reporting issues with those as soon as I did, so I have to assume that it is set up well enough for NC to work.

iperf3 is reporting 280Mbis/s over my WiFi to the server, but NC manages an abs… max of ~1.2MB/s.

Having writen an SMTP server that could saturate 100Mbit ethernet on a P120 running FreeBSD and mysql, 20 years ago, I strongly take issue with the idea that even without a jit, PHP on a machine 100x as powerful can’t manage over 12Mbit/s.

I’m aware that the previews are generated on-the-fly, that much is obvious from the servers CPU maxing out all 4 cores when I try to scroll the list.

I’m aware of “preview generator” (its mentioned frequently as a “solution” to this problem, and I plan to install it, possibly, but even generating thumbnails on the fly should work - and certainly shouldn’t DoS the any other function on the server whilst it chews through the thumbnail generation.

I don’t think it’s snarky to complain that the error log messages aren’t helpful. (the backtraces are at least somewhat useful, but I’m not so familiar with the NC codebase that I can do much with them.)

I’ve googled the various errors that have cropped up in the overview and in the logs extensively, and squasjed them one by one, leaving only two issues - one being the total lack of performance - which ultimately, I can live with if I really have to - and the other being a cURL timeout when attempting to play/download a video from a federated share. It seems to be a REALLY REALLY common problem, mentioned dozens of times in this wiki, but not once with a solution.

The error message gives no clue how to fix it.
The documentation gives no clue how to fix it.
This Wiki has never addressed the issue AFAICT.

It’s 2024 - syncing a few GB of photos shouldn’t be this hard / slow.

I see people claiming NC transferring data at ~30MBit/s.

If the bottleneck is allegedly the CPU (it isnt, its idle most of the time), this would imply that these people have 20GHz quad core CPUS.

Ergo, Its not the CPU.

It’s not my network.
It’s not my DNS (it resolves correctly to a local IP.)
It’s not my router and it’s traffic rules (or iperf would be in the toilet)
It’s not the disk array - I can rsync the files at >800Mbit/s over my LAN.

[spyro]

Btw - My dashboard currently shows that I’ve ploughed through 43 topics and 285 posts in the process of simply trying to set up something that should require “unzip” as more or less its only installation step.

and thats since I created an account on here, which covers maybe 10% of the time I’ve spent trying to make this work…

[bb77]

@spyro

If the bottleneck is allegedly the CPU (it isnt, its idle most of the time), this would imply that these people have 20GHz quad core CPUS.

How old is your CPU it and what’s the exact model?

And just for reference, I’m testing it right now with a folder containing 1581 photos, with a total size of 8.6 GB and it shows about 28MB/s in the WebUI. The upload of the entire folder took about 7-8 minutes.

Mine is running in a VM on Proxmox VE (4 Cores/8GB of RAM) on a Xeon E3-1240 v5 (4 Core/8 Threads, Skylake Generation). So nothing new or too fancy.

Software Stack: Apache, PHP8.2-FPM, Redis, MariaDB10.11.

My PHP, database and Nextcloud configuration is more or less a mix of the following guides: (German)

https://www.c-rieger.de/nextcloud-installationsanleitung/

https://www.c-rieger.de/nextcloud-installationsanleitung-apache2-fast-track/

[bb77]

Btw. The previous test was drag an drop to the browser from an SMB share mounted to my computer.

I then copied the folder from the NAS to my desktop and tested it again directly from the local filesystem of my computer:

[spyro]

Its an x86, although I’ll have to check the exact model when I get home, but why is it relevant? PHP is an interpretted language (ok, theres the jit, but the situation is identical with / without it).

My other NC box is on a 6 core 2GHz ARM64, and the performance is extremely similar.

Box 1: NC28, x86, 4 cores, 2GHz, 8GiB RAM, using debian packages for nginx, php8.2-fpm, redis, mariadb.

Box 2: NC28, ARM64, 4big 2 little, 2GHz, 4GiB RAM, debian packages as above.

using debian packages certbot and getting letsencrypt certs via dns-rfc2136 (something I do elsewhere on other machines and I know it working well.)

configuration is literally identical on each machine, bar hostnames, certs, trusted server list, etc.

Given the very different CPU architectures, RAM, etc. I dont think that the CPU is the issue here.

Besides, apart from when its actually generating previews, the CPU, network, and disk are near completely idle. The box is under practically 0 load, and this is the only thing its used for AT ALL - ie. its literally a box only for NC. there is no GUI no nothing, just a very minimal debian OS plus NC.

I’ve just read the German install instructions you posted above (hallo, wie gehts?), and they look extremely similar to what I’ve done. I’ll compare the config files when I get home, but nothing is jumping out at me.

Any further thoughts?

[bb77]

@spyro

Given the very different CPU architectures, RAM, etc. I dont think that the CPU is the issue here.

Based on what you are describing, I don’t think it’s the CPU either. Or maybe it is, but in a different way than you might think…

@spyro

Any further thoughts?

When I changed the CPU type in Proxmox to “host” instead of whatever the default generic KVM CPU type was at the time I installed Nextcloud, upload speeds became much faster. Probably because it couldn’t take advantage of some HW offloading features related to SSL before.

I’m not an expert on this, but very old CPUs that don’t support certain HW offloading features can be significantly slower when encryption comes into play. But that’s just a wild guess, and the upload speed should probably still be at least 10-20MBs, even without HW offloading, except maybe the CPU is much older than 10-15 years.

[spyro]

I’m not using a VPS, so the CPU “emulation” by the VPS provider is a non-issue for me.

I’ve checked the ARM box and it’s managing 250MBit/s via scp (encrypted and compressed) over the LAN, and (o this site) 60Mbit/s over the wifi.

It’s not the hardware - its nextcloud - either its misconfigured (which I dont believe, given how insistent it is in the maintainane page about checking everything), or there is a bug (which I strongly believe).

Thanks for trying to help though.

I will come back here when I figure out a solution, wether that be fixing nextcloud, or finding something else that actually works…

It’s a shame - many of nextclouds apps are quite nice, but its all utterly pointless when the basic, essential, completely fundamental, core parts of it just dont work at all (federation) or work too slowly to be any use at all (files).

The photos “app” might be useable, if if worked with any speed, but thus far, I havent even got part-way through my library scrolling, because every so often, it stucks and I have to wait for it to go non-responsive, 500 me, and recover, before I can scroll again… Except that even that’s pointless, because it never actually rendered most of the pictures (presumably because of the 500 errors).

All its doing now, is (slow, very slow) basic file autoupload /sync to my laptop from my phone. Its useless for anything else.

I could have that (faster and better) with SyncThing, but I wanted a personal cloud

[bb77]

@spyro

I’m not using a VPS, so the CPU “emulation” by the VPS provider is a non-issue for me.

So you’re running it on bare metal? Fine, but you still didn’t tell us what exact CPU you are using.

However, you’re probably right. If you only get 1MB/sec it’s not likely to be caused by the CPU, unless maybe that CPU is 20 years old…

[mdw]

@bb77

So you’re running it on bare metal? Fine, but you still didn’t tell us what exact CPU you are using.

It’s a “4 core 2GHz x86 with 8GB of RAM”. Or in other words likely a NAS with an Intel J/N-Series or Atom. Certainly barely-good-enough-class CPU, but also not the reason why NC would upload only with 1MB/s.

@spyro Stupid question, but are you using NC directly via the IP or are you using a domain? If so, you could be limited by your general internet upload speed.

Also do you use any kind of proxy, e.g. a HTTPS-proxy?

You should definitely install & configure the preview generator app, live previews are very CPU intensive, and the photos app does not care how many requests it sends.

[xCKx]

http: vs https: access can also vary speeds I found.

I do agree with the general complaint after using NC now for probably 5 years or longer.

After various tweaks and always staying up to date it just always has seemed laboured to me in performance, with barely any errors in the logs at all.

I’m forever hearing my (halfway decent) machine furiously working away to deal with photos in particular. When you use it long term and have 20 years+ of stuff it is slow to access meaning the joy in having all the memories in data is locked away not really usable in a more gratifyingly quick way.

[spyro]

Hi,

I’ve tried via its local IP, I’ve tried being the only other thing on the network (direct connection), and of course via its dns (which resolves correctly on WAN to the routers WAN IP, and to the NC boxes IP on the LAN). Any misguided packets (I guess when transitioning a mobile device from wan to lan), are handled via nat loopback, on an OpenWrt 22.03.3 router (old bt homehub 5a)

The PC is an Intel(R) Atom™ CPU C2550 @ 2.40GHz, according to /proc/cpuinfo.

Its a power hungry turd, and frankly, the RockPro64 with a PCIe Gen2 x2 SATA card makes it look really old…

But its not that bad

Im using port forwarding from my router to the NC box, which lives in a DMZ all by itself. (local traffic rules exit to contact it directly), so no proxy.

[mdw]

@spyro

Im using port forwarding from my router to the NC box, which lives in a DMZ all by itself. (local traffic rules exit to contact it directly), so no proxy.

Just to be clear here, i also meant proxy as in a proxy on the server itself. Often users also have other web services and use a HTTP-Proxy to handle HTTPS and route requests to the actual service. (so e.g. Internet → Nginx Proxy → Nextcloud AIO Docker Image)
That combination can be a performance limitation if the proxy server isn’t configured accordingly.

Have you used the Nextcloud WebDav Interface only over the official clients, or also trough other clients such as the Windows Explorer, FileZilla etc.?

I looked at how NC is handling file uploads on my server an noticed that it usually transfers the file in 10MB chunks and then assembles the whole file once it’s uploaded. This works around timeouts and upload size limits, but also costs performance.
On my server, NC clients/the web ui could only get to 800MBit/s while Ubuntu’s own file manager could get 1GBit/s.

How did you set up your NC? Is that a docker container or is it Apache/mod-php or Nginx/apache and PHP FPM? Have you checked any of the limits which apply there?

Lastly again about the “Photos” app & Previews. The photos app first loads a 64x64px preview, then once you scroll closer to the image it loads a large preview which can by default be up to 4096x4096px. Anyone correct me if i’m wrong here but that is how i understand the docs. Consider checking your config and limit the size.

I personally use the memories app instead of the official photos app as it handles scrolling much more efficiently. IT does less requests and smaller preview sizes.
When scrolling fast in the photos app, it often runs into my servers request limiter while the memories app never does.

[Rinkana]

One thing you could try is to temporarily disable any http → https redirection and try uploading over lan and http. Just to rule out any ssl cipher and old cpu bottleneck.

Have you been getting any errors in the different logs (i mean database, php-fpm, apache)? Enable slow log in both mariadb and php-fpm to see if there is something specific that will appear.

[spyro]

There is no redirection. the server only listens on port 443;

However, I did try as you suggested a few days ago ( for the same reason) and nothing improved.

There are no relevant errors (I’m trying to resolve another issue re: federation not working properly, but that’s (I expect) unrelated.)

[Rinkana]

I just recently found a mistake I had in my ZFS setup where the database was on a disk where the ZIL (read sync writes) was being written to the same hdds that the thumbnails where being saved, which crippled both the database and the hdds, maybe you have something similar? I’m just shooting into the wild.

I just remembered a thing I discovered during benchmarking an old instance a while ago (I think nc23?): h2 on the server gave notably slower upload than http1. However if you did try over lan without https then you probably tried that already indirectly. Nevertheless you can try disabling h2 for the sake of it. I don’t think this is a thing now, or at least in my case where a reverse proxy is terminating both ssl and h2 (I cant test because up and download are maxing my ethernet)

There are not that many things that reasonably would affect up/down -load speed:

1. Disk. This is by far the most important. Try running a disk benchmark during upload to nc as a good starting point
2. php-timeouts/maxupload/chunking. If your php-fpm gets killed before finishing upload it will cause a bunch of retries. This whole ordeal gets quite complicated with chunking as the whole php-fpm-fcgid-apache config gets involved. Here I’d mostly look ar the error logs, maybe enable the slow log. If you have a well configured apache php-fpm stack then this is unlikely the issue, but if you really cant find any reason why its slow, it would be here as debugging that evil trio is kinda hard if you ask me and there’s a lot of small things…
3. Network speed. Self explanatory, does not seem like an issue in your case. Just double check that the networking is being handled my the network card and not by cpu (virtual nics and such). I guess you could try rsync while uploading to nc

[spyro]

My server is running … I guess its called bare metal by some - no dockerisation. just nginx acting as a webserver and php-fpm and thats it.

I’ve only been using the official clients, and web. Oh, and whatever is built into debian’s file browser/gnome (I’ve not checked as I’m not using it, but I tried it…)

Thanks for the tips re ditching photos, I’ll look into that…

…clickety

…

Blimey. Well that’s a huge improvement. Quite why Memories isn’t the default app, I cannot fathom. What a transformation - from (honestly) complete garbage (Photos) that DoS’d the machine, to a completely workable solution. Even without preview generation, its completely useable on my hardware (above), with maybe a 10 second wait if I scroll like a madman, before it settles again. Which I don’t have to do anymore, because there’s a timeline.

Thanks, that’s made the whole thing viable again.

(If I could just see why its SO slow at file xfer, and get it to federate fully, it’d be close to perfect now.)

I don’t suppose there’s a better phonetrack app / plugin or sth. that can auto-split tracks into individual journeys? If not, I might have a look at writing one…

[wojtek_alma]

One thing is NC being awfully slow even on cutting-edge physical servers, because of how NC has been designed, how its comnfigured per host, etc etc…

The other thing is how NC staff is reacting to people when they criticize NC.

===

My story with NC ended last year, when, after have been running problemless for close to 8 years, my NC suddenly stopped and will not “boot” back. No errors/warnings have been logged, no traces of hardware failure(s), just ordinary operation. Stopped and that was THE END.

[spyro]

Have you found a replacement?