File upload Forbidden (not file permission, not selinux)

Nextcloud version (eg, 10.0.2): 11.0.3.2
Operating system and version (eg, Ubuntu 16.04): CentOS 6
Apache or nginx version (eg, Apache 2.4.25): httpd24
PHP version (eg, 5.6): rh-php70
Is this the first time you’ve seen this error?: yes

Can you reliably replicate it? (If so, please outline steps):

Yes, upload a file.

The issue you are facing:

Uploading file shows “Forbidden” error, log file is not updated.

Things I checked:

  • file permission are ok (see below)
  • selinux settings are ok, permissive mode does not change problem. (see below)

This is an owncloud installation upgraded to nextcloud 11.

# ps -ef | grep http
apache xxxx xxxx 0 17:03 ? 00:00:09 /opt/rh/httpd24/root/usr/sbin/httpd

# ls -lasZ data
total 84
drwxrwx—. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 .
drwxr-x–x. apache apache system_u:object_r:httpd_sys_content_t:s0 …
drwxr-x–x. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 admin
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 appdata_ocqhhjszijrv
drwxr-x–x. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 files_external
-rw-r-----. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 .htaccess
-rw-r-----. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 index.html
drwxr-x–x. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 jove
-rw-r-----. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 nextcloud.log
-rw-r-----. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 .ocdata
-rw-r–r--. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 updater.log
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_rw_content_t:s0 updater-ocqhhjszijrv

# getenforce
Permissive

# sudo -u apache -g apache /opt/rh/rh-php70/root/usr/bin/php /var/www/html/nextcloud/occ files:scan --all

Scanning files for 2 users
Starting scan for user 1 out of 2 (admin)
Home storage for user admin not writable
Make sure you’re running the scan command only as the user the web server runs as
Starting scan for user 2 out of 2 (jove)
Home storage for user jove not writable
Make sure you’re running the scan command only as the user the web server runs as

±--------±------±-------------+
| Folders | Files | Elapsed time |
±--------±------±-------------+
| 0 | 0 | 00:00:00 |
±--------±------±-------------+

The output of your Nextcloud log in Admin > Logging:

None, nextcloud can’t write to logfile either.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

“system”: {
“updatechecker”: false,
“instanceid”: “ocqhhjszijrv”,
“passwordsalt”: “REMOVED SENSITIVE VALUE”,
“secret”: “REMOVED SENSITIVE VALUE”,
“trusted_domains”: [
mydomain.eu
],
“datadirectory”: “/var/www/html/owncloud/data”,
“overwrite.cli.url”: “https://mydomain.eu/owncloud”,
“dbtype”: “mysql”,
“version”: “11.0.3.2”,
“dbname”: “owncloud”,
“dbhost”: “localhost”,
“dbtableprefix”: “oc_”,
“dbuser”: “REMOVED SENSITIVE VALUE”,
“dbpassword”: “REMOVED SENSITIVE VALUE”,
“logtimezone”: “UTC”,
“installed”: true,
“memcache.local”: “\OC\Memcache\APCu”,
“integrity.check.disabled”: true,
“loglevel”: 2,
“maintenance”: false,
“updater.release.channel”: “stable”,
“mail_smtpmode”: “php”,
“mail_from_address”: “nextcloud”,
“mail_domain”: “mydomain.eu
},

This might be relevant.

Sun Jun 11 16:34:13.411554 2017] [authz_core:error] [pid 12934] [client w.x.y.z:xxxxx] AH01630: client denied by server configuration: /var/www/html/owncloud/data/htaccesstest.txt
[Sun Jun 11 16:36:00.695978 2017] [:error] [pid 11939] [client w.x.y.z:xxxxx] PHP Warning: file_put_contents(): Only 0 of 24 bytes written, possibly out of free disk space in /var/www/html/owncloud/updater/index.php on line 900
[Sun Jun 11 16:41:01.157351 2017] [authz_core:error] [pid 12915] [client w.x.y.z:xxxx] AH01630: client denied by server configuration: /var/www/html/owncloud/data/htaccesstest.txt
[Sun Jun 11 16:42:29.299567 2017] [:error] [pid 13036] [client w.x.y.z:xxxxx] PHP Warning: file_put_contents(): Only 0 of 24 bytes written, possibly out of free disk space in /var/www/html/owncloud/updater/index.php on line 900
[Sun Jun 11 16:48:36.166436 2017] [:error] [pid 11890] [client w.x.y.z:xxxxx] PHP Warning: scandir(/var/www/html/owncloud/data/updater-ocqhhjszijrv/downloads/): failed to open dir: No such file or directory in /var/www/html/owncloud/updater/index.php on line 610
[Sun Jun 11 16:48:36.166565 2017] [:error] [pid 11890] [client w.x.y.z:xxxxx] PHP Warning: scandir(): (errno 2): No such file or directory in /var/www/html/owncloud/updater/index.php on line 610
[Sun Jun 11 16:49:42.078434 2017] [authz_core:error] [pid 11892] [client w.x.y.z:xxxxx] AH01630: client denied by server configuration: /var/www/html/owncloud/data/htaccesstest.txt
[Sun Jun 11 16:50:49.088172 2017] [:error] [pid 13053] [client w.x.y.z:xxxxx] PHP Warning: file_put_contents(): Only 0 of 24 bytes written, possibly out of free disk space in /var/www/html/owncloud/updater/index.php on line 900
[Sun Jun 11 16:57:09.547890 2017] [authz_core:error] [pid 11942] [client w.x.y.z:xxxxx] AH01630: client denied by server configuration: /var/www/html/owncloud/data/htaccesstest.txt
[Sun Jun 11 16:58:17.375031 2017] [:error] [pid 11942] [client w.x.y.z:xxxxx] PHP Warning: file_put_contents(): Only 0 of 24 bytes written, possibly out of free disk space in /var/www/html/owncloud/updater/index.php on line 900
[Sun Jun 11 17:05:20.293663 2017] [authz_core:error] [pid 12915] [client w.x.y.z:xxxxx] AH01630: client denied by server configuration: /var/www/html/owncloud/data/.ocdata

strace to the rescue:

stat("/var/www/html/owncloud/data/admin/", {st_mode=S_IFDIR|0751, st_size=10, …}) = 0
access("/var/www/html/owncloud/data/admin/", W_OK) = 0
lstat("/var/www/html/owncloud/data/admin/files", 0x7ffd0cb7ac70) = -1 ENOENT (No such file or directory)
lstat("/var/www/html/owncloud/data/admin", {st_mode=S_IFDIR|0751, st_size=10, …}) = 0
lstat("/var/www/html/owncloud/data", {st_mode=S_IFDIR|0770, st_size=226, …}) = 0
stat("/var/www/html/owncloud/data/admin/files", 0x7ffd0cb7dd50) = -1 ENOENT (No such file or directory)
access("/var/www/html/owncloud/data/admin/", F_OK) = 0
lstat("/var/www/html/owncloud/lib/composer/composer/…/…/…/lib/private/ForbiddenException.php", {st_mode=S_IFREG|0644, st_size=985, …}) = 0
open("/var/www/html/owncloud/lib/private/ForbiddenException.php", O_RDONLY) = 8

Creating the missing “files” directory fixes the problem.

I have already attempted “maintenance:repair”, should that not restore missing directories like this?

Created issue 5446