Nextcloud jail on truenas - anormal upload speed issues

ℹ️ Support
1

[/details]
Hypervisor : VMWare 6.0U3
Truenas version : TrueNAS-12.0-U3.1
Nextcloud version (eg, 20.0.5): 21.0.2running on jail
Operating system and version (eg, Ubuntu 20.04): FreeBSD 12.2-RELEASE-p6
Apache or nginx version (eg, Apache 2.4.25): 2.4.46
PHP version (eg, 7.4): 7.4

The issue you are facing:

Hello,

I’m quite new with TrueNAS / Nextcloud but I’ve been reading and working fully on it for 1 week now.

I have an esxi running a truenas server on a VM. This server has one jail with nextcloud running on it (not pluggin, had the same issues though with the pluggin, which is the reason why I switched to dedicateed jail). I followed the following very good post for the installation (except for the reverse proxy/https part, I wasn’t not ready yet) : https://www.samueldowling.com/2020/07/24/install-nextcloud-on-freenas-iocage-jail-with-hardened-security/

Truenas as Nextcloud both work fine. But I have performance issues when uploading files in Nextcloud : I have a direct cable beween my computer and the ESX (both 1Gbe) and have the following performance copying for a 2GB file or a 4GB file :

  • Upload SMB to Truenas (same pool used by nextcloud) : 110MB/s
  • Download SMB from Truenas (same pool used by nextcloud) : 110MB/s
  • Upload http to Nextcloud : 50-60MB/s
  • Download http from Nextcloud : 110MB/s

(clearly no hardware issue)

To reach those speed and avoid any hardware limitation, I already ran a bunch of tweeks on my pool and dataset, disabling sync, compression, atime. And my RAM is not fully used.
On PHP.conf side, I increased the max_upload_size and max_memory.
I disabled zfs caching for db data to avoid double caching with the db

Aside that, I tried an installation on 3 other configurations with different versions of OS and PHP with the same results (download perfect, upload less than half the bandwidth)

  • Pluggin Nextcloud on TrueNAS instead of dedicated jail (with same versions)
  • one ubuntu 18 on a new VM with php 8.0
  • one ubuntu 20.4 running on windows 10 environment with php 7.3.

And I’m running out of ideas now, so I hope I missed something in tweeking PHP or database because I have the feeling the problem comes from an option here. I didn’t customize anything on db side. I didn’t know what to tweek inside.

Does anyone have an idea or recommendation please ?
I plan to buy a bigger server wih 10Gbe support, but if I don’t even manage to pass 500mb/s upload with my old, there is no point yet…

Thanks for your help.
Damien

2

DB caching can help a bit. You have to consider that Nextcloud is doing quite a few things in the background when you upload new files, so you won’t reach the speed of protocols that don’t do that. With proper settings, you should get half of the native speed (from previous topics on upload speed problems), beyond that it is harder. And for a single file, the db stuff should be much less than for a lot of small files.
Depends a bit where you do the upload (certainly the client, not sure about the web-browser), it can upload in chunks of a certain size (10 MB) which is not ideal for very fast connections.

1 Like
3

Thank you verty much for your answer.
I didn’t find the topics you’re refering regarding the limitation to half of the bandwidth, but I really started to wonder if it was the case when I had always the half of my bandwidth on 4 different environments… It’s a bit frustrating though.

To complete, I’m testing using a web browser for my uploads with windows, not the webdav client (because about packet size limitations).
With the Android webdav client it’s slower, but since I have wifi limitations at 600Mb/s, I wanted to be sure of the capacity of my server before digging more on webdav client improvment.

Questions :

  • does that mean that if I build a 10Gb/s server, I will be able to reach ~5Gb/s max writing speed ?
  • would you have any link to tweek the db caching or other parameters ?
  • Any chance to tweek the android webdav client ? I read that on previous version it wasn’t possible, who knows now ^^

I found this link, I’ll finish my https/http2 configuration to see if it improves the perfs https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html

Thanks

4

As complementary information :
I tried to upload 2 files at the same time and I manage to reach 100MB/s

So it’s lear now that the limitation comes from a parameter in nextcloud like max_speed_per_file and not the cache configuration. Any way to change that ?

5

Just out of curiosity: why not run a Nextcloud VM directly on ESXi…?

6

It’s a good question :slight_smile:
[long story]

My goal was to use my NAS with TrueNAS to aggregate my disks with ZFS + backup, and Nextcloud as a replacement for Google photos, having a SMB access to the Nextcloud folders.
So far, I’m evaluating NAS solutions to see if I do everything opensource or if I buy a syno or a QNAP, but they don’t have the hardware I want. Anyway.
I chose TrueNAS for his Nextcloud plugin at the beginning. Then read that it was better avoiding it for compatibility purpose and started using jail and even try it on a dedicated VM…

[short story]
I ended up seeing that I could map directly the datasets with my jail (theorically up to 6Gb/s for my SATA3 SSD) instead of using external storage through network (1Gb/s), saving network ressources. It should improve response time compared to a dedicated VM mapped through network.
And I’m not confident using TrueNas as iSCSI share for Netcloud on a virtual environment. Too much ties on VMWare layer. if I finally decide to run my TrueNas directly on the hardware instead of a VM, the migration will be easier with everything in the same machine

1 Like
7

That’s as good a reason as there can be, no problem with that…

A few notes…
You can get the Hansson NC VM (for ESXi) that uses ZFS for the (separate) data partition…

Running nested hypervisors - jail on ESXi in this case - is always a crutch.
You can never be sure future updates won’t kill/castrate the “duet”…

ESXi is very efficient with iSCSI of any shape or form I tried.
Including the enterprise QNAP boxes (that run ZFS).
I haven’t used myself, but think TrueNAS as iSCSI target for ESXi should be a good fit.
Or maybe NFS (I had problems with losing QNAP NFS maps during network issues)…

1 Like
8

Thanks for the notes, I didn’t know Hanson NC, I’ll read about it.

Dedicated Truenas as iSCSI target for ESXi works well (I did saw it) but in my case, Truenas is already a VM of the ESXi, and I don’t want to use a VM as iSCSI target for the host itself ^^

And I didn’t say much about it, but my goal is to build an ultra silent low consumption NAS with 10Gbe support. I will probably go a supermicro mb X10SDV-8C-TLN4F+ (35W-816vcpu-210Gbe). So I would like to avoid having to buy a second machine, cause passive low consumption 10Gbe machines are not cheap… But I’ll adjust when I migrate on it what architecture gets the best performances.

9

I doubt that since the network speed is not your current limitation either. I’d use iotop, top, … to checkout if you can identify the limiting process and the resource used.
For database you have tuning-primer and mysqltuner scripts that can help you to analyse the size of your caches.

2 Likes
10

Thanks. As I tested in my previous message, the limitation, end up to be per file, not with the server itself : I manage to reach top speed while uploading 2 files at the same time.

Here is top during one transfer :
image

and here is the one with 2 transfers :

image
image714×491 12.4 KB

it looks like my limitation is based on CPU, no ? one upload seems to use only one core instead of multicore. Not sure the cache will make a big difference. Any tweek I missed ? or it’s a normal mecanism ?

11

Or not, in fact we can see 3 cores used for one upload (if the column “C” is the core number)…
I have seriously no idea what’s wrong here…

12

The Linux top version also shows io-wait (check for freebsd how to show them), these processes are limited by io-operations.

If you have to go more into detail, there are a few topics on github where they debug parts of the code and they show with phpstorm how to find the problematic parts in code.