External Storage Speed with Memories

Hello, all. I’m struggling to get sufficient speed out of my network server when loading lots of files (over 100,000 photos, 400,000 files, 10+TB).

Background: this box is a TrueNas raidz2 system with two VMs on top: Nextcloud 28 (snap) on Ubuntu 22.04 LTS and Windows 2019 Essentials.

My metal specs are (8) 12TB 12gb/s SAS drives with a 1TB NVME m.2 l2arc, Xeon e5 2687W v4 and 128GB ECC DDR4 (2133 MT/s) and (2) 10GBe x540-T nics. Data write speeds seem pretty darn good on the TrueNas box with fio and dd giving me around 1.5GB/s on average.

What I’m struggling with is that Memories is not displaying photos very quickly. I’ve had to attach to my device with smb using the removable-media connector, as it appears to be the only way to connect. I’ve spent weeks working on making this as fast as possible and I’m at my wit’s end. Anyone have any idea how to speed up display of these files?

As far as I can think,

  1. Ensure the database is on the NVME
  2. If possible put the app data (basically the file previews) on the NVME as well.

Also generally the latency from DB to PHP should be as low as possible since Nextcloud makes hundreds of queries. Ideally they should run on the same machine.

Thanks, pulsejet.

So actually, I’ve made tremendous progress on this. I did a couple things that made marked improvement:

  1. Update php memory limit to 12GB, since I have the memory to spare.
  2. Expand the drive size, since Nextcloud was running out of space and couldn’t properly index all of the files. The database and files it’s found amount to over 560GB. Since Memories downloads the files from external links to speed itself up, it makes sense, I guess.

I’m working on your suggestion currently, but zvols are not kind in transferring unused data space for zfs send/receive (even space that is unused shows as used), so I’ve had to clone disks slowly inside the VM using dd.

Well yes and no. The original files would only be downloaded during indexing / when they are changed. During normal usage, only the previews are used (which exist in the app data folder) and the database needs to be accessed.

My optimized setup with docker looks something like this:

  app:
    ...
    volumes:
      - /faststorage/nextcloud:/var/www/html
      - /storage/ncdata:/var/www/html/data
      - /faststorage/nc_appdata_oc2tcfwr1cuh:/var/www/html/data/appdata_oc2q3fxr3cuh

The storage is a slow HDD that stores the actual data, faststorage is NVME that stores the database, PHP files and app data.

OK, returning back to this. After downloading even more files and letting this sucker continue to run, I found a few things:

  1. The NVME was useless. If anything, it actually slowed down the system. I tried it as zfs L2 cache, I tried it as another drive hosting the OS (still using snap, of course), and I tried using it as an SLOG. None of those helped one bit, and in fact the extra latency with the OS drive being independent of the storage drive (and because it has to be connected with external storage) made it slower.
  2. After increasing the drive size to at least 1TiB, the entire thing sped up dramatically. I mean, night and day difference. Apparently preview generation was hitting a ceiling, and from a caching standpoint has been very greedy spatially. However, my library grown even more since then, so I bumped the host drive to 2TiB and the removable-media stayed in the same location.

Oh, and one more thing I did: I properly aggregated the 10GBe cards for throughput instead of failover. That didn’t make that much of a difference because it exceeds the throughput of my HDDs, but worth mentioning if someone else struggles.