Custom CSS broken with php-fpm in Apache

Nextcloud version (eg, 18.0.2): 19.0.4
Operating system and version (eg, Ubuntu 20.04): CentOS 8
Apache or nginx version (eg, Apache 2.4.25): 2.4.37
PHP version (eg, 7.1): 7.4.11 (php-fpm)

The issue you are facing:
All CSS and other files that are in the “appdata” directory are ignored by Firefox because they return a 404 status code.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Run NextCloud on Apache using mod_php and htaccess config transposed into main vhost config
  2. Set a custom colour in Admin > Themeing
  3. Happily use it for years
  4. Switch the server to PHP-FPM
  5. Refresh the page and watch it break

Screenshot_2020-10-17 Settings - Nextcloud

  1. Open the Firefox console, go to Network and reload. See the “404” codes for the CSS files that are in /css/core/ (which gets loaded from the appdata dir in the data dir that’s outside the web root by the CssController). But, if you click on the file and check the response then it has CSS content (which Firefox presumably ignores because of the 404 code)

The output of your Nextcloud log in Admin > Logging:

... Nothing relevant - it's all CLI/cron entries...

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '...',
  'passwordsalt' => '...',
  'datadirectory' => '/srv/nextcloud-data',
  'dbtype' => 'mysql',
  'version' => '19.0.4.2',
  'dbname' => '...',
  'dbhost' => 'localhost',
  'dbtableprefix' => '...',
  'dbuser' => '...',
  'dbpassword' => '...',
  'installed' => true,
  'forcessl' => true,
  'maintenance' => false,
  'theme' => '',
  'trusted_domains' =>
  array (
    0 => 'pim.ibboard.co.uk',
  ),
  'mail_from_address' => 'owncloud-server',
  'mail_smtpmode' => 'sendmail',
  'mail_domain' => 'ibboard.co.uk',
  'secret' => '...',
  'forceSSLforSubdomains' => true,
  'loglevel' => 0,
  'trashbin_retention_obligation' => 'auto',
  'htaccess.RewriteBase' => '/',
  'updater.release.channel' => 'stable',
  'mysql.utf8mb4' => true,
  'overwrite.cli.url' => 'https://pim.ibboard.co.uk',
  'defaultapp' => 'apporder',
  'remember_login_cookie_lifetime' => 1296000,
  'session_lifetime' => 86400,
  'session_keepalive' => true,
  'token_auth_enforced' => false,
  'auth.bruteforce.protection.enabled' => true,
  'mail_smtpauthtype' => 'LOGIN',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'updater.secret' => '...',
);


The output of your Apache/nginx/system log in /var/log/____:

... Nothing logged in /var/log/httpd/error_<site>.log...

From various bits of debugging, what I believe is happening is that the “ends with svg or other known file type” RewriteCond gets hit and so the file doesn’t go through the normal rewrite, the file then falls through to the ErrorDocument 404 handling, that then gets rewritten to index.php as a subrequest, Apache passes that to PHP-FPM over the FCGI proxy, and Nextcloud finds the file and returns the file content with a “200 OK” status line*.

Normally that would set the correct status, but I think Apache Apache doesn’t replace the 404 status code because it can’t tell whether 200 means “the error handler found the file” or “the proxy successfully found the file to show for error 404”. The Apache docs say “If the ErrorDocument specifies a local redirect to a CGI script, the script should include a “Status:” header field in its output in order to ensure the propagation all the way back to the client of the error condition that caused it to be invoked”, which I’m taking to be this situation.

The only way I’ve found to fix it is to add the following at the start of lib/public/AppFramework/Http/FileDisplayResponse.php's callback function:

header("Status: " . $output->getHttpResponseCode());

I don’t know whether this is a bug that has somehow remained undiscovered with PHP-FPM and FCGI setups or whether there’s just something odd with my setup.

* Technically it returns a “HTTP/1.1 200 OK” despite getting a HTTP/2 request, but that’s because Apache seems to be sending it “HTTP/2.0” and Nextcloud doesn’t recognise that as a valid protocol and defaults to HTTP/1.1

Just to confirm that this still happens in 20.0.3.

Also, the additional line should probably be:

$output->setHeader("Status: " . $output->getHttpResponseCode());

(inside the callback() function but outside the if block)

First time I’ve ever installed Nextcloud. Just installed version 23.
Same issue.

I’m still tracking down other issues, but here’s how I fixed this one.

--- a/lib/private/AppFramework/Http/Output.php	2022-04-16 05:20:21.947184262 +0000
+++ b/lib/private/AppFramework/Http/Output.php	2022-04-16 05:19:59.498717020 +0000
@@ -75,6 +75,9 @@
 	 */
 	public function setHttpResponseCode($code) {
 		http_response_code($code);
+		if (substr(PHP_SAPI,-4) == 'fcgi') {
+			$this->setHeader("Status: $code");
+		}
 	}
 
 	/**

I’ll give that a go. It’s a much cleaner approach that’s probably more likely to be accepted as a pull request :slightly_smiling_face:

Testing this out with v25 (because I seem to have the problem with every update since ~v19), the suggestion from @h0tw1r3 doesn’t work for me because that function isn’t called. If I put a debugging print in setOutput() then that gets printed, but the one in setHttpResponseCode() never gets printed. And my change no longer seems to work either.

Are there so few people running through php-fpm on Apache? Or is there something missing from my config? Because all of my other scripts (custom scripts, custom CMS, and Wordpress) don’t have issues with the HTTP header codes.

This feels uglier and uglier, but what seems to fix it with the latest update of v25 is replacing:

        public function setOutput($out) {
                print($out);
        }

with

        public function setOutput($out) {
                if (substr(PHP_SAPI,-4) == 'fcgi') {
                        $this->setHeader("Status: " . $this->getHttpResponseCode());
                        http_response_code($this->getHttpResponseCode());
                }
                print($out);
        }

It was the quickest and easiest way that I could find to get a function that was actually called. The previous locations don’t seem to change anything and still give HTTP/2 404 responses for all of the CSS.

Hi
I’ve got nextcloud version 28 and sadly your solution doesn’t work
I have the sccs files in 404 error …

did anyone fix it ?

(Un)fortunately I’m still on v27 because News isn’t compatible with v28 yet. But the delay is because of UI changes, so I’m not entirely surprised that this broke. Looks like I need to delay updating and possibly investigate it again.

If the page looks broken in NC 28. check if you have to update your server configuration.

Nextcloud added .mjs files. If these aren’t delivered correctyl by the server, the browser may reject them. Here is the updated nginx example.

Hi
Fortunatly my pages are not broken :slightly_smiling_face:. I just see error in firefox inspector because i was looking for “how t change OPenDyslexic font size” (see my question here How to change size of dyslexic font)
I think may be my problem could be solved with a right access to scss (?)

I’ m using apache … thank you for your answer anyway