External local storage not showing files, forgetting shares and previews

I have setup a running nextcloud using Docker / Compose, following the basic instructions on Docker Hub. Internal storage is working fine.

However, I have also configured an external storage of type Local (under admin settings as a global storage - users are not allowed to add their own external storage).

 {
    "mount_id": 1,
    "mount_point": "\/my-data",
    "storage": "\\OC\\Files\\Storage\\Local",
    "authentication_type": "null::null",
    "configuration": {
        "datadir": "\/mnt\/my-data"
    },
    "options": {
        "enable_sharing": true,
        "encoding_compatibility": false,
        "encrypt": true,
        "filesystem_check_changes": 1,
        "previews": true,
        "readonly": false
    },
    "applicable_users": [],
    "applicable_groups": []
}
More details

Server-side encryption is disabled, so encrypt: true has no effect.

The external storage is pre-populated with data from outside of nextcloud.
Obviously, I have also modified the docker-compose.yml file to mount my real folder into the container.

version: '3'

services:
…

  app:
    …
    volumes:
      - nextcloud:/var/www/html
      - /pools/data/my-data:/mnt/my-data

Every user can see and access the folder my-data in their Files in the Web UI. This works.

However, I have four issues:

  1. When I open my-data or any sub-folder in the Web UI, often the folder is shown as empty. When I reload the page, or simply click again on the folder name in the bread crumb, the content appears. This is pretty consistent: on first access the folder is empty, and from there the content appears.
  2. After 5 minutes (specifially, after the next cron run), the folders are first showing as empty as well.
  3. When the folder contents loads, file previews are generated. When I open the folder again, the previews load much faster (i.e. the cached previews are used). However, after the next cron run, the file previews are re-generated.
  4. I have created an external share link for some sub-folder inside the local storage. This share link is also gone after the next cron run. This makes sharing on external storage useless.

Except for the first point, all issues are related to the cron job.

I have tried to run php occ files:scan --all to see if it makes any difference (should it?), but it does not scan my-data. Only the files in the user’s homes internal storage. I have checked with files:scan -v --all, it only lists the internal files.

/var/www/html $ php occ files:scan -v --all | grep my-data | wc -l
0

The number of files in the files:scan output certainly does not include the files from my external storage.

I have also tried files:scan --path /mnt/my-data or files:scan --path /admin/files/my-data, but it just errors out.

output
/var/www/html $ php occ files:scan --path /mnt/my-data
Unknown user 1 mnt
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 0       | 0     | 0      | 00:00:00     |
+---------+-------+--------+--------------+
/var/www/html $ php occ files:scan --path /admin/files/my-data
Starting scan for user 1 out of 1 (admin)
Path not found: /admin/files/my-data
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 0       | 0     | 1      | 00:00:00     |
+---------+-------+--------+--------------+

Do you have any advice?
Since so much is going wrong, I have the feeling that I am missing some fundamental understanding.

Versions

Nextcloud version: 27.0.2
Operating system and version: Alpine Linux v3.18
Apache or nginx version: 1.25.1
PHP version: 8.2.8

You understand and do many things wrong. Mainly because you didn’t read the manuals thoroughly. Too many for me to answer everything alone.

READ THIS:

occ files:scan --help
Description:
  rescan filesystem

Usage:
  files:scan [options] [--] [<user_id>...]

Arguments:
  user_id                  will rescan all files of the given user(s)

Options:
      --output[=OUTPUT]    Output format (plain, json or json_pretty, default is plain) [default: "plain"]
  -p, --path=PATH          limit rescan to this path, eg. --path="/alice/files/Music", the user_id is determined by the path and the user_id parameter and --all are ignored
      --generate-metadata  Generate metadata for all scanned files
      --all                will rescan all files of all known users
      --unscanned          only scan files which are marked as not fully scanned
      --shallow            do not scan folders recursively
      --home-only          only scan the home storage, ignoring any mounted external storage or share
  -h, --help               Display help for the given command. When no command is given display help for the list command
  -q, --quiet              Do not output any message
  -V, --version            Display this application version
      --ansi|--no-ansi     Force (or disable --no-ansi) ANSI output
  -n, --no-interaction     Do not ask any interactive question
      --no-warnings        Skip global warnings, show command output only
  -v|vv|vvv, --verbose     Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

So you have to do “–path=/path” or “-p /path” but not “–path /path” and $path can not be an absolute path, only pathes begining from your data directory, the files scan will then scan from inside of the users view. the external storage for any user again.

The system will need some time to create all previews and so on.

Of course I read the help output. As is standard for many arg parsers, --long x, --long=x, -s x and -sx are all equivalent, as is for occ:

/var/www/html $ php occ files:scan --path /my-user/files/some-folder
Starting scan for user 1 out of 1 (my-user)
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 3       | 3     | 0      | 00:00:00     |
+---------+-------+--------+--------------+
/var/www/html $ php occ files:scan --path=/my-user/files/some-folder
Starting scan for user 1 out of 1 (my-user)
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 3       | 3     | 0      | 00:00:00     |
+---------+-------+--------+--------------+
/var/www/html $ php occ files:scan -p/my-user/files/some-folder
Starting scan for user 1 out of 1 (my-user)
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 3       | 3     | 0      | 00:00:00     |
+---------+-------+--------+--------------+
/var/www/html $ php occ files:scan -p /my-user/files/some-folder
Starting scan for user 1 out of 1 (my-user)
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 3       | 3     | 0      | 00:00:00     |
+---------+-------+--------+--------------+

I am also aware that the format of the --path string should be <user>/files/<path>. I was just highlighting the fact that it is unclear how you would pass in the path to a global external storage (not user-created).

occ uses Symfony (e.g. Scan.php), which supports this standard as well.

The system will need some time to create all previews and so on.

I am also aware of that. The folders are full of JPGs and HEICs, it takes some time to generate the previews. And that’s ok.
However, as I have noted in my first post, the previews are lost (the cache is no longer valid) after the next cron run. Then it has to generate the previews again, which is the issue I am reporting.
I am sorry if that was not clear from my post.

You are right, sory for that, it was sepcially this echo:

/var/www/html $ php occ files:scan --path /mnt/my-data
Unknown user 1 mnt

that caught my eye.

For the rest I have to say that Docker is absolutely not my strength. So sory for my insubstantial post :pray:

Much luck!

Ok, in the mean time, I tried another occ files:scan --all. This time, it actually also scanned the external storage.
I have no idea why.

Before (on every run I tried), the output was something like

$ php occ files:scan --path /admin/files/my-data | head
Starting scan for user 1 out of 2 (admin)
Error: path not found /admin
Starting scan for user 2 out of 2 (my-user)
Error: path not found /my-user
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 10     | 100 | 0      | 00:00:02     |
+---------+-------+--------+--------------+

Now, since its working:

/var/www/html $ php occ files:scan --path /admin/files/my-data
Starting scan for user 1 out of 1 (admin)
+---------+-------+--------+--------------+
| Folders | Files | Errors | Elapsed time |
+---------+-------+--------+--------------+
| 700     | 56831 | 0      | 00:00:06     |
+---------+-------+--------+--------------+

Here you also see that the way to reference the external storage for --path is <user>/files/<folder>. But before, it was not working. I do not know why. How can I initiate scanning of a folder, when NC does not know about the folder, because I have to scan it first?

Also note that files:scan --all will scan the external storage for every user, so it’s best to do it only for one user:

/var/www/html $ php occ files:scan --all -v | grep my-data | wc -l
115062
/var/www/html $ php occ files:scan --path /admin/files/my-data -v | grep my-data | wc -l
57531

Since this is now working, all my issues have been solved. At least for now, I could not reproduce it any longer, previews and shares are stable.

Could this be due to a full cron job queue? As far as I can tell, NextCloud maintains a list of jobs to execute for a cron run. The cron job (every 5 minutes), takes a bunch (all?) of the jobs from that queue and executes it.
Due to some misconfiguration, my cron job was not running for two days. I have fixed it since, which started the issues with my external storage.
Could it be that the cron job queue was just very full, and the cron job did not execute all pending jobs? And maybe there was some pending job that made the external storage available for files:scan? This is just wild guessing on my side …

1 Like

I can not answer that with full certainty but you can list the jobs with:

occ background-job:list --limit=100

(or with an even higher limit if you want, default is 10)

Then you could - if not yet done - install the previewgenerator app, to create the previews in advance:

   App-Id          previewgenerator
   App-Name        Preview Generator
   Summary         A preview pre-generation app
   Categories      multimedia
   App-Version     5.3.0
   Repository      https://github.com/nextcloud/previewgenerator
   Issue-Tracker   https://github.com/nextcloud/previewgenerator/issues
   PHP min/max     7.4 / 8.2
   NC min/max      25 / 27
                   Not-shipped (not included) App
   Appstore-link   https://apps.nextcloud.com/apps/previewgenerator
   Created         2016-12-07T10:01:33.940211Z
   Last modified   2023-05-17T07:51:15.130089Z

much luck!

ernolf

1 Like