Custom CSS broken with php-fpm in Apache

Nextcloud version (eg, 18.0.2): 19.0.4
Operating system and version (eg, Ubuntu 20.04): CentOS 8
Apache or nginx version (eg, Apache 2.4.25): 2.4.37
PHP version (eg, 7.1): 7.4.11 (php-fpm)

The issue you are facing:
All CSS and other files that are in the “appdata” directory are ignored by Firefox because they return a 404 status code.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Run NextCloud on Apache using mod_php and htaccess config transposed into main vhost config
  2. Set a custom colour in Admin > Themeing
  3. Happily use it for years
  4. Switch the server to PHP-FPM
  5. Refresh the page and watch it break

Screenshot_2020-10-17 Settings - Nextcloud

  1. Open the Firefox console, go to Network and reload. See the “404” codes for the CSS files that are in /css/core/ (which gets loaded from the appdata dir in the data dir that’s outside the web root by the CssController). But, if you click on the file and check the response then it has CSS content (which Firefox presumably ignores because of the 404 code)

The output of your Nextcloud log in Admin > Logging:

... Nothing relevant - it's all CLI/cron entries...

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '...',
  'passwordsalt' => '...',
  'datadirectory' => '/srv/nextcloud-data',
  'dbtype' => 'mysql',
  'version' => '19.0.4.2',
  'dbname' => '...',
  'dbhost' => 'localhost',
  'dbtableprefix' => '...',
  'dbuser' => '...',
  'dbpassword' => '...',
  'installed' => true,
  'forcessl' => true,
  'maintenance' => false,
  'theme' => '',
  'trusted_domains' =>
  array (
    0 => 'pim.ibboard.co.uk',
  ),
  'mail_from_address' => 'owncloud-server',
  'mail_smtpmode' => 'sendmail',
  'mail_domain' => 'ibboard.co.uk',
  'secret' => '...',
  'forceSSLforSubdomains' => true,
  'loglevel' => 0,
  'trashbin_retention_obligation' => 'auto',
  'htaccess.RewriteBase' => '/',
  'updater.release.channel' => 'stable',
  'mysql.utf8mb4' => true,
  'overwrite.cli.url' => 'https://pim.ibboard.co.uk',
  'defaultapp' => 'apporder',
  'remember_login_cookie_lifetime' => 1296000,
  'session_lifetime' => 86400,
  'session_keepalive' => true,
  'token_auth_enforced' => false,
  'auth.bruteforce.protection.enabled' => true,
  'mail_smtpauthtype' => 'LOGIN',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'updater.secret' => '...',
);


The output of your Apache/nginx/system log in /var/log/____:

... Nothing logged in /var/log/httpd/error_<site>.log...

From various bits of debugging, what I believe is happening is that the “ends with svg or other known file type” RewriteCond gets hit and so the file doesn’t go through the normal rewrite, the file then falls through to the ErrorDocument 404 handling, that then gets rewritten to index.php as a subrequest, Apache passes that to PHP-FPM over the FCGI proxy, and Nextcloud finds the file and returns the file content with a “200 OK” status line*.

Normally that would set the correct status, but I think Apache Apache doesn’t replace the 404 status code because it can’t tell whether 200 means “the error handler found the file” or “the proxy successfully found the file to show for error 404”. The Apache docs say “If the ErrorDocument specifies a local redirect to a CGI script, the script should include a “Status:” header field in its output in order to ensure the propagation all the way back to the client of the error condition that caused it to be invoked”, which I’m taking to be this situation.

The only way I’ve found to fix it is to add the following at the start of lib/public/AppFramework/Http/FileDisplayResponse.php's callback function:

header("Status: " . $output->getHttpResponseCode());

I don’t know whether this is a bug that has somehow remained undiscovered with PHP-FPM and FCGI setups or whether there’s just something odd with my setup.

* Technically it returns a “HTTP/1.1 200 OK” despite getting a HTTP/2 request, but that’s because Apache seems to be sending it “HTTP/2.0” and Nextcloud doesn’t recognise that as a valid protocol and defaults to HTTP/1.1