Issues after update to NC 20.0.3 / NCP 1.33.0

OS: Armbian Buster (Kernel 5.9)
Installation method: NCP curl method
Original NCP version: 1.32.0(?)
Original NC version: 20.0.2(?)
Filesystem: /boot as ext4 on SD, / as btrfs on SSD, datadir as btrfs on HDD

Issues:

  1. Windows sync client reports error 503 (service unavailable)
  2. Windows sync client reports error 403 (permission denied)
  3. Floccus / bookmark app reports rerror 503 during sync
  4. I found out that my /mnt/datadir filesystem changes to read-only after some time. Why on earth would it do that?!

The system is still running and I can use the web UIs of NC and NCP. To fix the issues I need to reboot the system.
What I don’t understand are the messages about lack of permissions and non-existance of the data dir (which is clearly there and which I moved using NCP).

Analysis and examples of log file messages:

  • NCP WebUI
    “Nextcloud check” reports as error (what does this mean exactly?)
  • Manual check of file permissions
    Seems as if every file and dir belongs to www-dir (and I have never transfered files directly, only via web UI or Android/Windows sync client)
  • ncp.log
ncp.log message

“message”:“Temporary directory /mnt/datadir/tmp is not present or writable”,“userAgent”:"–",“version”:“20.0.3.2”}
“message”:{“Exception”:“Error”,“Message”:“fopen(/mnt/datadir/data_dir_writability_test_5fd7513ddb73d.tmp): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/legacy/OC_Util.php#804”,“Code”:0,“Trace”:[{“function”:“onError”,“class”:“OC\Log\ErrorHandler”, […]

{“reqId”:“Ibi6NNeNQSNVESzpXfvH”,“level”:3,“time”:“2020-12-14T11:49:17+00:00”,“remoteAddr”:"",“user”:"–",“app”:“PHP”,“method”:"",“url”:"–",“message”:“fopen(/mnt/datadir/nextcloud.log): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/Log/File.php#85”,“userAgent”:"–",“version”:“20.0.3.2”}
data directory not found

  • nextcloud.log
nextcloud.log message

“message”:“Deprecated event type for \OC\Repair::step: Symfony\Component\EventDispatcher\GenericEvent is used”,“userAgent”:"–",“version”:“20.0.2.2”}

If I run the ncp permission fix command (to be precise this part: sudo -u www-data php occ config:system:get datadirectory), I get a lot of error output:

command output

{“reqId”:“hycaS8MzX0XWePKtkigq”,“level”:2,“time”:“2020-12-14T13:17:05+00:00”,“remoteAddr”:"",“user”:"–",“app”:“no app in context”,“method”:"",“url”:"–",“message”:“Temporary directory /mnt/datadir/tmp is not present or writable”,“userAgent”:"–",“version”:“20.0.3.2”}
{“reqId”:“hycaS8MzX0XWePKtkigq”,“level”:3,“time”:“2020-12-14T13:17:06+00:00”,“remoteAddr”:"",“user”:"–",“app”:“PHP”,“method”:"",“url”:"–",“message”:{“Exception”:“Error”,“Message”:“fopen(/mnt/datadir/data_dir_writability_test_5fd765d2e7b38.tmp): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/legacy/OC_Util.php#804”,“Code”:0,“Trace”:[{“function”:“onError”,“class”:“OC\Log\ErrorHandler”,“type”:"::",“args”:[2,“fopen(/mnt/datadir/data_dir_writability_test_5fd765d2e7b38.tmp): failed to open stream: Read-only file system”,"/var/www/nextcloud/lib/private/legacy/OC_Util.php",804,{“config”:{“class”:“OC\SystemConfig”},“l”:{“class”:“OC\L10N\LazyL10N”},“errors”:[],“CONFIG_DATADIRECTORY”:"/mnt/datadir",“webServerRestart”:false,“setup”:{“class”:“OC\Setup”},“urlGenerator”:{“class”:“OC\URLGenerator”},“availableDatabases”:{“mysql”:“MySQL/MariaDB”},“testFile”:"/mnt/datadir/data_dir_writability_test_5fd765d2e7b38.tmp"}]},{“file”:"/var/www/nextcloud/lib/private/legacy/OC_Util.php",“line”:804,“function”:“fopen”,“args”:["/mnt/datadir/data_dir_writability_test_5fd765d2e7b38.tmp",“w”]},{“file”:"/var/www/nextcloud/lib/private/Console/Application.php",“line”:161,“function”:“checkServer”,“class”:“OC_Util”,“type”:"::",“args”:[{“class”:“OC\SystemConfig”}]},{“file”:"/var/www/nextcloud/console.php",“line”:99,“function”:“loadCommands”,“class”:“OC\Console\Application”,“type”:"->",“args”:[{“class”:“Symfony\Component\Console\Input\ArgvInput”},{“class”:“Symfony\Component\Console\Output\ConsoleOutput”}]},{“file”:"/var/www/nextcloud/occ",“line”:11,“args”:["/var/www/nextcloud/console.php"],“function”:“require_once”}],“File”:"/var/www/nextcloud/lib/private/Log/ErrorHandler.php",“Line”:91,“CustomMessage”:"–"},“userAgent”:"–",“version”:“20.0.3.2”}
Your data directory is not writable
Permissions can usually be fixed by giving the webserver write access to the root directory. See https://docs.nextcloud.com/server/20/go.php?to=admin-dir_permissions.

An unhandled exception has been thrown:
Exception: Environment not properly prepared. in /var/www/nextcloud/lib/private/Console/Application.php:168
Stack trace:
#0 /var/www/nextcloud/console.php(99): OC\Console\Application->loadCommands(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#1 /var/www/nextcloud/occ(11): require_once(’/var/www/nextcl…’)
#2 {main}{“reqId”:“hycaS8MzX0XWePKtkigq”,“level”:3,“time”:“2020-12-14T13:17:06+00:00”,“remoteAddr”:"",“user”:"–",“app”:“PHP”,“method”:"",“url”:"–",“message”:“fopen(/mnt/datadir/nextcloud.log): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/Log/File.php#85”,“userAgent”:"–",“version”:“20.0.3.2”}

Any hints on where I could start debugging?

Indication of drive failure or corrupt data.
I’d look at end of dmesg and syslog when it occurs

Happened to me after a restore, tmp_dir in php.ini file not set correctly, that bug was fixed in one of the latest version updates (ncp v1.33.0)

Thanks Oliver :slight_smile:

The problems started after the update to NCP 1.33.0.

I just checked the php.ini files (/etc/php/7.3/cli/ and /etc/php/7.3/fpm/) and under file uploads the setting was
upload_tmp_dir = /mnt/datadir/tmp
I changed the filesize from 2M to 10G (like I did via the NCP WebUI already - not sure why it doesn’t reflect in the ini files), but that couldn’t have been the reason, since files larger than 2M were synced without issues.
Should there be another entry with a general tmp dir, not only for uploads?

The read-only issue seemed to be related to btrfs clean space cache errors and the syncing has been running for a few hours without the filesystem becoming ready-only again.
I was able to fix it with: btrfs check --clear-space-cache /dev/sda1