So, this post below describes a similar problem, but doesn’t detail a fix. I remain confused by this as it is pretty reproducible.
The short version is that a number of UI icons and fields don’t render. This means that although the UI can generate a share link if you know which invisible area to click, there’s no means of an average user working out where to do this. Screenshots below.
The long version is that it appears some CSS variables aren’t being set. I’m not a frontend person so haven’t worked out how these are set, but working examples look like rather than being set as paths, they’re actually populated as data. This leads me to believe they come from the database, but my postgres and application logs are inconclusive.
As can be seen above, I can locate the correct element on screen, and see that the variable for the icon background (which I believe is how the icon is displayed visually) is not set. I think the fix for this is to work out why, but I haven’t had much luck.
Has anyone encountered anything similar to this? I’d love to work out a fix and document it in public for anyone else having the same problem.
Technical details:
Using the docker image (apache) for 18.0.4.2
Postgresql database
Redis and APCu enabled
Object storage as a primary backend
Slightly customised entrypoint.sh
More on request, I guess. It works fine aside from that, apart from general slowness I’m attributing to also using object storage as a primary backend.
I think I’ve narrowed this down somewhat. On a fresh install of nextcloud with all defaults in place, the variables above for the missing icons are present as base64 encoded svg data. I believe the source images are in place in my home deployment, but have for some reason haven’t been compiled into the css assets and it isn’t clear to me where the generated css lives if it has a permanent home at all. Judging by the issues on github, it is currently generated on the fly by the server compiling scss files.
I don’t have anything in my logs indicating files can’t be read, and it isn’t clear to me which svgs are being read within /var/www/html - I’m assuming things under core/css/ of my install, or apps/files/css/.
After more investigation I’ve found that something turns up in the logs - seemingly intermittently? - which looks pretty spot on for what I’m experiencing:
$ ./occ log:tail
--------- ------------------- ------------------------------------------------------------------ ---------------------------
Level App Message Time
--------- ------------------- ------------------------------------------------------------------ ---------------------------
Warning core Failed to compile and/or save /var/www/html/core/css/server.scss 2020-04-30T17:53:05+00:00
Error cssresourceloader Could not find resource css/server.css to load 2020-04-30T17:53:05+00:00
Warning core Failed to compile and/or save /var/www/html/core/css/server.scss 2020-04-30T17:53:07+00:00
Error cssresourceloader Could not find resource css/server.css to load 2020-04-30T17:53:07+00:00
Warning core Failed to compile and/or save /var/www/html/core/css/server.scss 2020-04-30T17:56:14+00:00
Error cssresourceloader Could not find resource css/server.css to load 2020-04-30T17:56:14+00:00
Warning core Failed to compile and/or save /var/www/html/core/css/server.scss 2020-04-30T17:56:23+00:00
Error cssresourceloader Could not find resource css/server.css to load 2020-04-30T17:56:23+00:00
Warning core Failed to compile and/or save /var/www/html/core/css/server.scss 2020-04-30T17:57:07+00:00
Error cssresourceloader Could not find resource css/server.css to load 2020-04-30T17:57:07+00:00
/var/www/html/core/css/server.scss definitely exists, but server.css only exists in one theme:
So, inspecting further and looking for icon-share-000 (the missing variable) on a fresh install has yielded a lot of generated assets in /var/www/html/data/appdata_<my_instance_id> which don’t exist in the corresponding directory on my own server. This stands to reason, as my backend is an object storage backend. I’ll probe whether or not these files exist on the object storage backend by recreating locally with minio then grepping for the right files, and investigate higher verbosity in the logs. It’d be nice to be able to log object storage backend requests of this kind and whether or not they’re successful.
So, a fresh install fixes this. Pretty mystifying. I’m going to fish out the file from the s3 backend (which sadly seems to involve pulling down the first thousand objects and grepping them) and remove it in the hope it’ll be regenerated. Failing that, I may just poke a new one in.
I didn’t fish the file out of S3: I pulled the first ten thousand objects out of the store and grepped them for the right string. Nada. Later on, I checked the oc_filecache table and was able to find the correct file, albeit definitely with the wrong contents.
Instead, I’ve installed https://github.com/mwalbeck/nextcloud-breeze-dark which fixed the problem briefly. I’m not sure what happened after that - possibly instantiating a new container - but the problem occurred again.
I’ve since compared icons-vars.css and confirmed that a very truncated version is being generated in the broken install, which doesn’t set most variables. I can regenerate this by clearing the items starting with the string appdata in the oc_filecache table, which re-inserts them into object storage and prompts a rebuild, including reinserting references into the databsae. The result is the same, so I’ll be returning to investigating the PHP which generates this.
I’ve also found errors in the nextcloud log not visible through occ log:tail, which refer to these files not being found, even with max verbosity turned on.
Trying random things and generally taking my own customisations out, I hit upon the problem. I had been using object storage, with minio acting as a gateway between the application and an S3 backend. I’d been using it for caching. Taking minio out and waiting a while led to the problem going away, seemingly permanently. I guess it added latency and there was a timeout or something, but who knows.
