External Storage Speed with Memories

🍱 Features & apps Memories
, , , ,
1

Hello, all. I’m struggling to get sufficient speed out of my network server when loading lots of files (over 100,000 photos, 400,000 files, 10+TB).

Background: this box is a TrueNas raidz2 system with two VMs on top: Nextcloud 28 (snap) on Ubuntu 22.04 LTS and Windows 2019 Essentials.

My metal specs are (8) 12TB 12gb/s SAS drives with a 1TB NVME m.2 l2arc, Xeon e5 2687W v4 and 128GB ECC DDR4 (2133 MT/s) and (2) 10GBe x540-T nics. Data write speeds seem pretty darn good on the TrueNas box with fio and dd giving me around 1.5GB/s on average.

What I’m struggling with is that Memories is not displaying photos very quickly. I’ve had to attach to my device with smb using the removable-media connector, as it appears to be the only way to connect. I’ve spent weeks working on making this as fast as possible and I’m at my wit’s end. Anyone have any idea how to speed up display of these files?

2

As far as I can think,

  1. Ensure the database is on the NVME
  2. If possible put the app data (basically the file previews) on the NVME as well.

Also generally the latency from DB to PHP should be as low as possible since Nextcloud makes hundreds of queries. Ideally they should run on the same machine.

3

Thanks, pulsejet.

So actually, I’ve made tremendous progress on this. I did a couple things that made marked improvement:

  1. Update php memory limit to 12GB, since I have the memory to spare.
  2. Expand the drive size, since Nextcloud was running out of space and couldn’t properly index all of the files. The database and files it’s found amount to over 560GB. Since Memories downloads the files from external links to speed itself up, it makes sense, I guess.

I’m working on your suggestion currently, but zvols are not kind in transferring unused data space for zfs send/receive (even space that is unused shows as used), so I’ve had to clone disks slowly inside the VM using dd.

4

Well yes and no. The original files would only be downloaded during indexing / when they are changed. During normal usage, only the previews are used (which exist in the app data folder) and the database needs to be accessed.

My optimized setup with docker looks something like this:

  app:
    ...
    volumes:
      - /faststorage/nextcloud:/var/www/html
      - /storage/ncdata:/var/www/html/data
      - /faststorage/nc_appdata_oc2tcfwr1cuh:/var/www/html/data/appdata_oc2q3fxr3cuh

The storage is a slow HDD that stores the actual data, faststorage is NVME that stores the database, PHP files and app data.