Endless/unfinished automatic share expiration task: my weirdest Nextcloud outage ever, which took down the whole server

ℹ️ Support
,
1
Support intro

Sorry to hear you’re facing problems. :slightly_frowning_face:

The community help forum (help.nextcloud.com) is for home and non-enterprise users. Support is provided by other community members on a best effort / “as available” basis. All of those responding are volunteering their time to help you.

If you’re using Nextcloud in a business/critical setting, paid and SLA-based support services can be accessed via portal.nextcloud.com where Nextcloud engineers can help ensure your business keeps running smoothly.

Getting help

In order to help you as efficiently (and quickly!) as possible, please fill in as much of the below requested information as you can.

Before clicking submit: Please check if your query is already addressed via the following resources:

(Utilizing these existing resources is typically faster. It also helps reduce the load on our generous volunteers while elevating the signal to noise ratio of the forums otherwise arising from the same queries being posted repeatedly).

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 33.0.7.1
  • Operating system and version (e.g., Ubuntu 24.04):
    • Raspberry Pi OS 12
  • Web server and version (e.g, Apache 2.4.25):
    • nginx 1.22.1
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • nginx 1.22.1
  • PHP version (e.g, 8.3):
    • 8.4.23
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • When a public link share expired
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • Bare Metal
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

A public link share expired as planned right at midnight. Shortly after, php-fpm load increased massively which took down the whole server roughly 25 minutes later due to being rebooted by the system watchdog (load too high). A second reboot happened 10 to 15 minutes later for the same reason, which was right at the time I started investigating the incident. I also discovered:

  • activities app: created a ton of entries regarding the public link share expiration, dozens a second (in total according to the database: 254.429 :warning::warning::warning: - if the server reboots and my PHP-FPM service stop / NC maintenance mode activation would not have been there, this would for sure had increased even more, potentially endless)
  • when trying to access the share settings from the web Ui or the Windows desktop client or page /apps/files/sharinglinks, it times out (neverending load time)
  • nextcloud.log.1 increased to a massive size of 24 GB (instead of max. 100 MB)

grafik
grafik597×374 15.5 KB

I fixed this this way:

  1. Deleted affected shared folder physically from server and ran occ files:scan --all
  2. Then sudo -u www-data php /var/www/nextcloud/occ share:list --owner Username noted
In Share.php line 171:
Node for share not found, fileid: 1431041
  1. Therefore I identified and deleted this ID from nc_share table.
  2. Then I identified (SELECT COUNT(*) AS cnt FROM nc_activity WHERE subject LIKE '%link_expired%' AND file LIKE "/foldername";) and deleted all activity entries in nc_activity except the very first one with select activity_id from nc_activity where subject like "%link_expired%" and file like "/foldername" ORDER BY activity_id ASC LIMIT 1; and delete from nc_activity where subject like "%link_expired%" and file like "/foldername" and activity_id <> 340198;.
    This fixed occ share:list --owner Username and /apps/files/sharinglinks also was loading fine again. Still existing problems:
  • Re-creating the folder at the same place with the same name was impossible, no matter if via Web or desktop client (which indicated some kind of locked state)
  • One PHP-FPM process was still running at 100 % shortly after PHP-FPM got restarted, so something from NC side still was not alright
  1. Purged all entries from nc_file_locks table
  2. occ files:cleanup gave no deletion results
  3. Restarted php-fpm and nginx.
    Now no more cpu spikes anymore by PHP-FPM - finally silence on the server as regular. Also creating the folder at the same location with the very same name was possible again.
  4. Finally I took care of the massive nextcloud.log.1 by simply sudo truncate -s 0 /...nextcloud.log.1.

This was pretty much a nightmare, taking me 2 to 2,5 hours for investigation and fixing, only on the NC side. I let AI search for GitHub and forums issues related to what I experienced, but there were no non-halucinated results so I decided to precisely report what I experienced here.

Please note: everything is fixed, I need absolutely no support on this anymore. I’m more interested in understanding what was causing this and how to prevent this in future from happening again. For now I won’t use automatic expiration for any shares/shared links anymore.

In my theory, the share expiration background job went crazy and everything else like the activity app spam and the nextcloud.log spam as well as the massive server load were a result of PHP-FPM trying to work on the share expiration triggered by NC’s cron.

Steps to replicate it (hint: details matter!):

  1. Create a public link share and set expiration date (in my case 2026-07-30 00:00 so midnight)
  2. Shortly after 00:00 (I guess the cron started few minutes later, running every 5 minutes) I discovered the above mentioned symptoms.

For what it’s worth (complete picture): note I updated NC from 33.0.6.2 to 33.0.7.1 a few hours before the planned share expiration. No apps got updated during that process. A few more hours before, apps got updated manually - but the activities app wasn’t.

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

Had to be truncated, impossible to handle 24 GB files.

Web Browser

If the problem is related to the Web interface, open your browser inspector Console and Network tabs while refreshing (reloading) and reproducing the problem. Provide any relevant output/errors here that appear.

PASTE HERE

Web server / Reverse Proxy

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

PASTE HERE

Apps

The output of occ app:list (if possible).

Tips for increasing the likelihood of a response

  • Use the preformatted text formatting option in the editor for all log entries and configuration output.
  • If screenshots are useful, feel free to include them.
    • If possible, also include key error output in text form so it can be searched for.
  • Try to edit log output only minimally (if at all) so that it can be ran through analyzers / formatters by those trying to help you.
2

Thanks for the detailed write-up, that is more than most outage reports come with.

The one value your report is missing is probably the key one here: your log level. It is a suspicion rather than a firm diagnosis, but a nextcloud.log.1 that large really only comes from loglevel at 0 (DEBUG) with debug => true. And even the 100 MB you name as the normal ceiling is already well above what a healthy instance writes, which only reinforces that.

A guess on why one misbehaving share took the whole box down instead of just spinning in the background: writing tens of GB of debug output is heavy I/O, and on a Pi (SD card, or a USB-attached disk) that logging can itself become the bottleneck that pushed the load into a watchdog reboot. On fast NVMe the same issue might have passed as a hiccup. Where your log and data physically live matters here, which is another reason the config would help.

On the cleanup: there is no occ command to delete a single share by id, so editing nc_share directly was a fair option. The only caveat is that a raw delete can leave related rows behind (child shares via parent, nc_mounts) and skips the normal cache cleanup, so if something still looks off around that folder, start there. The supported route next time is the web UI, or the OCS Share API: OCS Share API — Nextcloud 35 Developer Manual

To pin the actual cause, the two outputs still missing would tell the most:

  • `occ config:list system (for loglevel, database)
  • occ app:list

Plus: How are background jobs run (system cron, webcron, AJAX)? Where do the log and data directory sit (SD card, USB disk, SSD)? And the share that expired: single file or a folder, and if a folder, how many files and how deep, and was any of it re-shared?

Meanwhile, on a production instance, set loglevel back to 3 (2 at most) and turn debug off if it was on.

ernolf

3

Loglevel is 1, for years. 100 MB is enough for a few regular weeks. Debug is false.
System runs on a NVMe, including log and data directory.
System cron is used.
Share was a single folder with roughly 20 files in it. None of them were re-shared.

You completely ignored the massive spamming of the activity app. Which is what I think stressed the system that much. Something’s wrong with scheduled share expirations and/or the activity app, that‘s still my assumption.

4

Thanks, that is genuinely helpful. It clears several suspicions in one go: loglevel 1, debug off, NVMe for both log and data, system cron, and a share that was just one folder with about 20 files. So this is neither a logging misconfiguration nor slow storage. Good to have that off the table.

And I fully get that you have already dealt with it and are back up, so none of what follows is about fixing your instance. It is the other way around: what you hit looks like it could catch other people the same way, and right now you are the only one holding the evidence that could pin down where exactly it goes wrong. A single small share should never spiral into 254k activities, and if that is a real weakness in the expiration path, the sooner it is located the fewer people run into it.

That is where loglevel 1 turns into a lucky break. Your 24 GB log is the single best artifact anyone could have for finding the actual cause. You cannot open it whole, but you do not need to. A targeted slice of the incident window is enough, and pulling that would be a real help to whoever hits this next.

The commands below assume you are in your log directory, otherwise just point them at your rotated nextcloud.log.1.

The log is newline-delimited JSON, one object per line, each with an integer level and an ISO 8601 time string. So a slice of the first minutes of the storm, filtered to warnings and errors only (level > 2), cuts the noise right down. From your screenshot the onset was around midnight going into Thursday the 30th, so:

grep '"2026-07-30T00:0' nextcloud.log.1 \
  | jq -Rc 'fromjson? // empty
            | select(.time >= "2026-07-30T00:00:00" and .time < "2026-07-30T00:05:00")
            | select(.level > 2)
            | {time, level, reqId, app, method, url, user, message}'

Nextcloud writes these timestamps in UTC by default (the +00:00 at the end), so if the time in your screenshot is local and you are not on UTC, just shift the window by your offset. Point it at whichever file holds that night. The fromjson? // empty is there so a single truncated line in a 24 GB file cannot abort the run.

Two more one-liners characterize the loop cheaply, without you having to paste much:

What actually repeats in that window:

grep '"2026-07-30T00:0' nextcloud.log.1 \
  | jq -Rc 'fromjson? // empty
            | select(.time >= "2026-07-30T00:00:00" and .time < "2026-07-30T00:05:00")
            | .message' \
  | sort | uniq -c | sort -rn | head -30

Whether it was one runaway process or a flood of separate requests (count of distinct request IDs):

grep '"2026-07-30T00:0' nextcloud.log.1 \
  | jq -Rc 'fromjson? // empty
            | select(.time >= "2026-07-30T00:00:00" and .time < "2026-07-30T00:05:00")
            | .reqId' \
  | sort -u | wc -l

If the level > 2 slice comes back thin, drop it to .level > 1 or remove the level filter entirely. And if any line in there carries an exception with a stack trace, one full example of that entry is worth more than a hundred plain ones.

A share on a single small folder should never generate 254k expiration activities, so something kept retrying the expiration in a tight loop. The warning or error line that repeats in that window, plus its stack trace, is what would turn this into a reproducible bug report worth filing.

ernolf